2024-03-28T05:01:37.040 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-28T05:01:37.040 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} --archive /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625303 --verbose --name mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi --owner mchangir -- /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625303/orig.config.yaml 2024-03-28T05:01:37.094 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T05:01:37.216 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625303 branch: wip-mchangir-testing-main-20240318.032620 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} email: mchangir@redhat.com first_in_suite: false job_id: '7625303' kernel: client: sha1: distro flavor: default kdb: true sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-testing-main-20240318.032620 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 1 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mchangir-testing-main-20240318.032620 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 owner: mchangir priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3645 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 sleep_before_teardown: 0 subset: 132/256 suite: fs suite_branch: wip-mchangir-testing-main-20240318.032620 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi023.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGcXHqnEegKTQ9ebyHpfZMjiU4KptQZ9YMSDrpQMmlrzfi7WmArglGEiTJELPEulThAjk2p8G1sPG4rUfrCnpmE= smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIdTI17QUixmhmqfRSyea4E8FISJaOKfNIg5xLG7WF57n71QL7PhRe7qz04si31/T45JHr8AUTNalQX/xsSc6Yo= smithi113.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEzPeFrAgrCKNnDBti4oU+l/qiuSUt8/4o8ID2LRmpn+ISFgq7e38/AvoGNy+z4d6Y85wfrkpH4N67g8ebfQ8ZY= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - check-counter: {} - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/blogbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-27_03:45:23 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-28T05:01:37.216 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa; will attempt to use it 2024-03-28T05:01:37.217 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks 2024-03-28T05:01:37.217 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-28T05:01:37.218 INFO:teuthology.task.internal:Checking packages... 2024-03-28T05:01:37.238 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3be9a5c9bc793e11e2800a8c0c696e8b46742033' 2024-03-28T05:01:37.239 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-28T05:01:37.239 INFO:teuthology.packaging:ref: None 2024-03-28T05:01:37.239 INFO:teuthology.packaging:tag: None 2024-03-28T05:01:37.239 INFO:teuthology.packaging:branch: wip-mchangir-testing-main-20240318.032620 2024-03-28T05:01:37.239 INFO:teuthology.packaging:sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T05:01:37.239 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-testing-main-20240318.032620 2024-03-28T05:01:37.460 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2145.g3be9a5c9 2024-03-28T05:01:37.461 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-28T05:01:37.467 INFO:teuthology.task.internal:no buildpackages task found 2024-03-28T05:01:37.467 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-28T05:01:37.480 INFO:teuthology.task.internal:Saving configuration 2024-03-28T05:01:37.499 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-28T05:01:37.505 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T05:01:37.529 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625303', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 04:55:40.385159', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGcXHqnEegKTQ9ebyHpfZMjiU4KptQZ9YMSDrpQMmlrzfi7WmArglGEiTJELPEulThAjk2p8G1sPG4rUfrCnpmE='} 2024-03-28T05:01:37.554 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625303', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 04:55:40.382945', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIdTI17QUixmhmqfRSyea4E8FISJaOKfNIg5xLG7WF57n71QL7PhRe7qz04si31/T45JHr8AUTNalQX/xsSc6Yo='} 2024-03-28T05:01:37.575 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi113.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625303', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 04:55:40.386874', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEzPeFrAgrCKNnDBti4oU+l/qiuSUt8/4o8ID2LRmpn+ISFgq7e38/AvoGNy+z4d6Y85wfrkpH4N67g8ebfQ8ZY='} 2024-03-28T05:01:37.575 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-28T05:01:37.581 INFO:teuthology.task.internal:roles: ubuntu@smithi023.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-28T05:01:37.581 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-28T05:01:37.582 INFO:teuthology.task.internal:roles: ubuntu@smithi113.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-28T05:01:37.582 INFO:teuthology.run_tasks:Running task console_log... 2024-03-28T05:01:37.681 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff9ef0340d0>, signals=[15]) 2024-03-28T05:01:37.681 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-28T05:01:37.687 INFO:teuthology.task.internal:Opening connections... 2024-03-28T05:01:37.687 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi023.front.sepia.ceph.com 2024-03-28T05:01:37.689 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T05:01:37.791 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-03-28T05:01:37.792 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T05:01:37.879 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi113.front.sepia.ceph.com 2024-03-28T05:01:37.880 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T05:01:37.966 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-28T05:01:37.972 DEBUG:teuthology.orchestra.run.smithi023:> uname -m 2024-03-28T05:01:37.991 INFO:teuthology.orchestra.run.smithi023.stdout:x86_64 2024-03-28T05:01:37.991 DEBUG:teuthology.orchestra.run.smithi023:> cat /etc/os-release 2024-03-28T05:01:38.049 INFO:teuthology.orchestra.run.smithi023.stdout:NAME="CentOS Stream" 2024-03-28T05:01:38.049 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION="9" 2024-03-28T05:01:38.049 INFO:teuthology.orchestra.run.smithi023.stdout:ID="centos" 2024-03-28T05:01:38.049 INFO:teuthology.orchestra.run.smithi023.stdout:ID_LIKE="rhel fedora" 2024-03-28T05:01:38.049 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION_ID="9" 2024-03-28T05:01:38.049 INFO:teuthology.orchestra.run.smithi023.stdout:PLATFORM_ID="platform:el9" 2024-03-28T05:01:38.049 INFO:teuthology.orchestra.run.smithi023.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T05:01:38.049 INFO:teuthology.orchestra.run.smithi023.stdout:ANSI_COLOR="0;31" 2024-03-28T05:01:38.049 INFO:teuthology.orchestra.run.smithi023.stdout:LOGO="fedora-logo-icon" 2024-03-28T05:01:38.049 INFO:teuthology.orchestra.run.smithi023.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T05:01:38.049 INFO:teuthology.orchestra.run.smithi023.stdout:HOME_URL="https://centos.org/" 2024-03-28T05:01:38.049 INFO:teuthology.orchestra.run.smithi023.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T05:01:38.049 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T05:01:38.049 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T05:01:38.050 INFO:teuthology.lock.ops:Updating smithi023.front.sepia.ceph.com on lock server 2024-03-28T05:01:38.070 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-03-28T05:01:38.088 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-03-28T05:01:38.088 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-03-28T05:01:38.145 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2024-03-28T05:01:38.145 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2024-03-28T05:01:38.145 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2024-03-28T05:01:38.145 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2024-03-28T05:01:38.145 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2024-03-28T05:01:38.145 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2024-03-28T05:01:38.145 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T05:01:38.145 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-03-28T05:01:38.145 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2024-03-28T05:01:38.145 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T05:01:38.145 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2024-03-28T05:01:38.145 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T05:01:38.146 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T05:01:38.146 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T05:01:38.146 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-03-28T05:01:38.170 DEBUG:teuthology.orchestra.run.smithi113:> uname -m 2024-03-28T05:01:38.188 INFO:teuthology.orchestra.run.smithi113.stdout:x86_64 2024-03-28T05:01:38.188 DEBUG:teuthology.orchestra.run.smithi113:> cat /etc/os-release 2024-03-28T05:01:38.244 INFO:teuthology.orchestra.run.smithi113.stdout:NAME="CentOS Stream" 2024-03-28T05:01:38.245 INFO:teuthology.orchestra.run.smithi113.stdout:VERSION="9" 2024-03-28T05:01:38.245 INFO:teuthology.orchestra.run.smithi113.stdout:ID="centos" 2024-03-28T05:01:38.245 INFO:teuthology.orchestra.run.smithi113.stdout:ID_LIKE="rhel fedora" 2024-03-28T05:01:38.245 INFO:teuthology.orchestra.run.smithi113.stdout:VERSION_ID="9" 2024-03-28T05:01:38.245 INFO:teuthology.orchestra.run.smithi113.stdout:PLATFORM_ID="platform:el9" 2024-03-28T05:01:38.245 INFO:teuthology.orchestra.run.smithi113.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T05:01:38.245 INFO:teuthology.orchestra.run.smithi113.stdout:ANSI_COLOR="0;31" 2024-03-28T05:01:38.245 INFO:teuthology.orchestra.run.smithi113.stdout:LOGO="fedora-logo-icon" 2024-03-28T05:01:38.245 INFO:teuthology.orchestra.run.smithi113.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T05:01:38.245 INFO:teuthology.orchestra.run.smithi113.stdout:HOME_URL="https://centos.org/" 2024-03-28T05:01:38.245 INFO:teuthology.orchestra.run.smithi113.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T05:01:38.245 INFO:teuthology.orchestra.run.smithi113.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T05:01:38.245 INFO:teuthology.orchestra.run.smithi113.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T05:01:38.246 INFO:teuthology.lock.ops:Updating smithi113.front.sepia.ceph.com on lock server 2024-03-28T05:01:38.278 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-28T05:01:38.288 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-28T05:01:38.294 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-28T05:01:38.294 DEBUG:teuthology.orchestra.run.smithi023:> test '!' -e /home/ubuntu/cephtest 2024-03-28T05:01:38.297 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-03-28T05:01:38.299 DEBUG:teuthology.orchestra.run.smithi113:> test '!' -e /home/ubuntu/cephtest 2024-03-28T05:01:38.315 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-28T05:01:38.321 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-28T05:01:38.322 DEBUG:teuthology.orchestra.run.smithi023:> test -z $(ls -A /var/lib/ceph) 2024-03-28T05:01:38.355 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-03-28T05:01:38.357 DEBUG:teuthology.orchestra.run.smithi113:> test -z $(ls -A /var/lib/ceph) 2024-03-28T05:01:38.461 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-28T05:01:38.542 INFO:teuthology.run_tasks:Running task kernel... 2024-03-28T05:01:38.556 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'} 2024-03-28T05:01:38.556 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-03-28T05:01:38.556 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-03-28T05:01:38.557 DEBUG:teuthology.orchestra.run.smithi023:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T05:01:38.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T05:01:38.574 DEBUG:teuthology.orchestra.run.smithi023:> uname -r 2024-03-28T05:01:38.632 INFO:teuthology.orchestra.run.smithi023.stdout:5.14.0-361.el9.x86_64 2024-03-28T05:01:38.632 INFO:teuthology.task.kernel:Running kernel on smithi023: 5.14.0-361.el9.x86_64 2024-03-28T05:01:38.632 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum install -y kernel 2024-03-28T05:01:40.764 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-28T05:01:42.198 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-28T05:01:43.530 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - BaseOS 8.3 MB/s | 8.0 MB 00:00 2024-03-28T05:01:45.390 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-03-28T05:01:49.657 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - CRB 9.7 MB/s | 6.1 MB 00:00 2024-03-28T05:01:51.375 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-03-28T05:01:52.895 INFO:teuthology.orchestra.run.smithi023.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-28T05:01:57.992 INFO:teuthology.orchestra.run.smithi023.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-03-28T05:02:00.191 INFO:teuthology.orchestra.run.smithi023.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-28T05:02:00.191 INFO:teuthology.orchestra.run.smithi023.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-28T05:02:00.284 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2024-03-28T05:02:00.290 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2024-03-28T05:02:00.291 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2024-03-28T05:02:00.291 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2024-03-28T05:02:00.291 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2024-03-28T05:02:00.291 INFO:teuthology.orchestra.run.smithi023.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-28T05:02:00.291 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2024-03-28T05:02:00.291 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-28T05:02:00.291 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-28T05:02:00.291 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-28T05:02:00.291 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:02:00.291 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2024-03-28T05:02:00.291 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2024-03-28T05:02:00.291 INFO:teuthology.orchestra.run.smithi023.stdout:Install 4 Packages 2024-03-28T05:02:00.291 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:02:00.292 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 76 M 2024-03-28T05:02:00.292 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 123 M 2024-03-28T05:02:00.292 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2024-03-28T05:02:00.552 INFO:teuthology.orchestra.run.smithi023.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 297 kB/s | 66 kB 00:00 2024-03-28T05:02:02.285 INFO:teuthology.orchestra.run.smithi023.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 7.6 MB/s | 15 MB 00:01 2024-03-28T05:02:03.135 INFO:teuthology.orchestra.run.smithi023.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 12 MB/s | 33 MB 00:02 2024-03-28T05:02:03.936 INFO:teuthology.orchestra.run.smithi023.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 8.1 MB/s | 27 MB 00:03 2024-03-28T05:02:03.936 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2024-03-28T05:02:03.936 INFO:teuthology.orchestra.run.smithi023.stdout:Total 21 MB/s | 76 MB 00:03 2024-03-28T05:02:04.630 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2024-03-28T05:02:05.064 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2024-03-28T05:02:05.064 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2024-03-28T05:02:06.107 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2024-03-28T05:02:06.107 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2024-03-28T05:02:08.733 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2024-03-28T05:02:09.352 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-28T05:02:09.452 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T05:02:12.348 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T05:02:12.489 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T05:02:17.598 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T05:02:17.639 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-28T05:02:22.768 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T05:02:53.570 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T05:02:53.571 INFO:teuthology.orchestra.run.smithi023.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-28T05:02:53.571 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:02:53.620 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-28T05:02:53.999 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-28T05:02:53.999 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-28T05:02:54.000 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T05:02:54.000 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T05:02:54.451 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T05:02:54.451 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:02:54.451 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2024-03-28T05:02:54.451 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-28T05:02:54.451 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-28T05:02:54.451 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-28T05:02:54.452 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-28T05:02:54.452 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:02:54.452 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2024-03-28T05:02:54.773 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T05:02:54.879 INFO:teuthology.orchestra.run.smithi023.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-28T05:02:54.879 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-28T05:02:54.879 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-28T05:02:54.879 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-28T05:02:54.880 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-28T05:02:54.880 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-28T05:02:54.880 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi023.front.sepia.ceph.com, path=None, version=distro) 2024-03-28T05:02:54.880 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T05:02:54.963 INFO:teuthology.orchestra.run.smithi023.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-28T05:02:54.964 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-28T05:02:54.964 DEBUG:teuthology.orchestra.run.smithi023:> sudo rpm -qi grub2-tools 2024-03-28T05:02:55.009 INFO:teuthology.orchestra.run.smithi023.stdout:Name : grub2-tools 2024-03-28T05:02:55.009 INFO:teuthology.orchestra.run.smithi023.stdout:Epoch : 1 2024-03-28T05:02:55.010 INFO:teuthology.orchestra.run.smithi023.stdout:Version : 2.06 2024-03-28T05:02:55.010 INFO:teuthology.orchestra.run.smithi023.stdout:Release : 68.el9 2024-03-28T05:02:55.010 INFO:teuthology.orchestra.run.smithi023.stdout:Architecture: x86_64 2024-03-28T05:02:55.010 INFO:teuthology.orchestra.run.smithi023.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-28T05:02:55.010 INFO:teuthology.orchestra.run.smithi023.stdout:Group : Unspecified 2024-03-28T05:02:55.010 INFO:teuthology.orchestra.run.smithi023.stdout:Size : 8264254 2024-03-28T05:02:55.010 INFO:teuthology.orchestra.run.smithi023.stdout:License : GPLv3+ 2024-03-28T05:02:55.010 INFO:teuthology.orchestra.run.smithi023.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-28T05:02:55.010 INFO:teuthology.orchestra.run.smithi023.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-28T05:02:55.010 INFO:teuthology.orchestra.run.smithi023.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-28T05:02:55.010 INFO:teuthology.orchestra.run.smithi023.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-28T05:02:55.010 INFO:teuthology.orchestra.run.smithi023.stdout:Packager : builder@centos.org 2024-03-28T05:02:55.010 INFO:teuthology.orchestra.run.smithi023.stdout:Vendor : CentOS 2024-03-28T05:02:55.010 INFO:teuthology.orchestra.run.smithi023.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-28T05:02:55.011 INFO:teuthology.orchestra.run.smithi023.stdout:Summary : Support tools for GRUB. 2024-03-28T05:02:55.011 INFO:teuthology.orchestra.run.smithi023.stdout:Description : 2024-03-28T05:02:55.011 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:02:55.011 INFO:teuthology.orchestra.run.smithi023.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-28T05:02:55.011 INFO:teuthology.orchestra.run.smithi023.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-28T05:02:55.011 INFO:teuthology.orchestra.run.smithi023.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-28T05:02:55.011 INFO:teuthology.orchestra.run.smithi023.stdout:hardware devices. 2024-03-28T05:02:55.011 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:02:55.011 INFO:teuthology.orchestra.run.smithi023.stdout:This subpackage provides tools for support of all platforms. 2024-03-28T05:02:55.012 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-28T05:02:55.012 INFO:teuthology.task.kernel:Updating grub on smithi023 to boot 5.14.0-432.el9.x86_64 2024-03-28T05:02:55.012 DEBUG:teuthology.orchestra.run.smithi023:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-28T05:02:55.718 INFO:teuthology.orchestra.run.smithi023.stderr:Generating grub configuration file ... 2024-03-28T05:02:57.198 INFO:teuthology.orchestra.run.smithi023.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-28T05:02:57.226 INFO:teuthology.orchestra.run.smithi023.stderr:done 2024-03-28T05:02:57.228 DEBUG:teuthology.orchestra.run.smithi023:> mktemp 2024-03-28T05:02:57.243 INFO:teuthology.orchestra.run.smithi023.stdout:/tmp/tmp.SAZF4wSoGM 2024-03-28T05:02:57.244 DEBUG:teuthology.orchestra.run.smithi023:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.SAZF4wSoGM 2024-03-28T05:02:57.315 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0666 /tmp/tmp.SAZF4wSoGM 2024-03-28T05:02:57.457 DEBUG:teuthology.orchestra.remote:smithi023:/tmp/tmp.SAZF4wSoGM is 6KB 2024-03-28T05:02:57.505 DEBUG:teuthology.orchestra.run.smithi023:> rm -fr /tmp/tmp.SAZF4wSoGM 2024-03-28T05:02:57.519 DEBUG:teuthology.orchestra.run.smithi023:> sudo /bin/ls /boot/loader/entries || true 2024-03-28T05:02:57.587 INFO:teuthology.orchestra.run.smithi023.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-28T05:02:57.587 INFO:teuthology.orchestra.run.smithi023.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-28T05:02:57.587 INFO:teuthology.orchestra.run.smithi023.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-28T05:02:57.587 INFO:teuthology.orchestra.run.smithi023.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-28T05:02:57.588 DEBUG:teuthology.orchestra.run.smithi023:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-28T05:02:57.728 DEBUG:teuthology.orchestra.run.smithi023:> sudo shutdown -r now 2024-03-28T05:03:27.731 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-28T05:03:27.731 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-28T05:03:27.733 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T05:03:46.206 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-28T05:03:55.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-28T05:03:55.208 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T05:03:58.270 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-28T05:04:10.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-28T05:04:10.273 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T05:04:13.342 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-28T05:04:28.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-28T05:04:28.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T05:05:28.347 DEBUG:teuthology.orchestra.remote:timed out 2024-03-28T05:05:46.348 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-28T05:05:46.349 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T05:05:46.666 DEBUG:teuthology.orchestra.run.smithi023:> true 2024-03-28T05:05:47.051 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-28T05:05:47.051 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-432.el9.x86_64"... 2024-03-28T05:05:47.052 DEBUG:teuthology.orchestra.run.smithi023:> uname -r 2024-03-28T05:05:47.067 INFO:teuthology.orchestra.run.smithi023.stdout:5.14.0-432.el9.x86_64 2024-03-28T05:05:47.067 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-28T05:05:47.067 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T05:05:47.067 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-28T05:05:48.068 DEBUG:teuthology.parallel:result is None 2024-03-28T05:05:48.068 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-28T05:05:48.076 INFO:teuthology.task.internal:Creating test directory... 2024-03-28T05:05:48.076 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T05:05:48.079 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T05:05:48.081 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T05:05:48.099 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-28T05:05:48.106 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-28T05:05:48.114 INFO:teuthology.task.internal:Creating archive directory... 2024-03-28T05:05:48.114 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T05:05:48.136 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T05:05:48.138 DEBUG:teuthology.orchestra.run.smithi113:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T05:05:48.484 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-28T05:05:48.490 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-28T05:05:48.491 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T05:05:48.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T05:05:48.507 DEBUG:teuthology.orchestra.run.smithi113:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T05:05:48.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T05:05:48.523 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T05:05:48.526 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T05:05:48.550 DEBUG:teuthology.orchestra.run.smithi113:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T05:05:48.589 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T05:05:48.613 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T05:05:48.614 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T05:05:48.618 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T05:05:48.638 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T05:05:48.651 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T05:05:48.653 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-28T05:05:48.660 INFO:teuthology.task.internal:Configuring sudo... 2024-03-28T05:05:48.661 DEBUG:teuthology.orchestra.run.smithi023:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T05:05:48.663 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T05:05:48.665 DEBUG:teuthology.orchestra.run.smithi113:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T05:05:48.696 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-28T05:05:48.705 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-28T05:05:48.705 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T05:05:48.734 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T05:05:48.737 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T05:05:48.755 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T05:05:48.832 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T05:05:48.877 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T05:05:48.957 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T05:05:49.002 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T05:05:49.003 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T05:05:49.070 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T05:05:49.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T05:05:49.170 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T05:05:49.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T05:05:49.297 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-28T05:05:49.297 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T05:05:49.366 DEBUG:teuthology.orchestra.run.smithi113:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T05:05:49.409 DEBUG:teuthology.orchestra.run.smithi113:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T05:05:49.466 DEBUG:teuthology.orchestra.run.smithi113:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T05:05:49.550 DEBUG:teuthology.orchestra.run.smithi113:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T05:05:49.594 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-28T05:05:49.594 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T05:05:49.663 DEBUG:teuthology.orchestra.run.smithi023:> sudo service rsyslog restart 2024-03-28T05:05:49.666 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-03-28T05:05:49.668 DEBUG:teuthology.orchestra.run.smithi113:> sudo service rsyslog restart 2024-03-28T05:05:49.707 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T05:05:49.707 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T05:05:49.742 INFO:teuthology.orchestra.run.smithi113.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T05:05:50.142 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-28T05:05:50.150 INFO:teuthology.task.internal:Starting timer... 2024-03-28T05:05:50.150 INFO:teuthology.run_tasks:Running task pcp... 2024-03-28T05:05:50.161 INFO:teuthology.run_tasks:Running task selinux... 2024-03-28T05:05:50.169 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-28T05:05:50.169 DEBUG:teuthology.orchestra.run.smithi023:> sudo service auditd rotate 2024-03-28T05:05:50.227 INFO:teuthology.orchestra.run.smithi023.stdout:Rotating logs: 2024-03-28T05:05:50.229 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-03-28T05:05:50.292 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-03-28T05:05:50.294 DEBUG:teuthology.orchestra.run.smithi113:> sudo service auditd rotate 2024-03-28T05:05:50.353 INFO:teuthology.orchestra.run.smithi113.stdout:Rotating logs: 2024-03-28T05:05:50.355 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-28T05:05:50.355 DEBUG:teuthology.orchestra.run.smithi023:> /usr/sbin/getenforce 2024-03-28T05:05:50.380 INFO:teuthology.orchestra.run.smithi023.stdout:Permissive 2024-03-28T05:05:50.380 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-03-28T05:05:50.405 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-03-28T05:05:50.405 DEBUG:teuthology.orchestra.run.smithi113:> /usr/sbin/getenforce 2024-03-28T05:05:50.438 INFO:teuthology.orchestra.run.smithi113.stdout:Permissive 2024-03-28T05:05:50.438 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi023.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi098.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi113.front.sepia.ceph.com': 'permissive'} 2024-03-28T05:05:50.438 DEBUG:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T05:05:50.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T05:05:50.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T05:05:50.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T05:05:50.498 DEBUG:teuthology.orchestra.run.smithi113:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T05:05:50.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T05:05:50.527 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-28T05:05:50.527 DEBUG:teuthology.orchestra.run.smithi023:> sudo /usr/sbin/setenforce permissive 2024-03-28T05:05:50.573 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-03-28T05:05:50.607 DEBUG:teuthology.orchestra.run.smithi113:> sudo /usr/sbin/setenforce permissive 2024-03-28T05:05:50.640 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-28T05:05:50.649 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-28T05:05:50.664 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-28T05:05:50.665 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi023.front.sepia.ceph.com,smithi098.front.sepia.ceph.com,smithi113.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-28T05:10:28.476 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi023.front.sepia.ceph.com'), Remote(name='ubuntu@smithi098.front.sepia.ceph.com'), Remote(name='ubuntu@smithi113.front.sepia.ceph.com')] 2024-03-28T05:10:28.477 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-28T05:10:28.478 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T05:10:28.568 DEBUG:teuthology.orchestra.run.smithi023:> true 2024-03-28T05:10:28.641 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-28T05:10:28.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-28T05:10:28.642 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T05:10:28.731 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-03-28T05:10:28.796 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-28T05:10:28.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-03-28T05:10:28.797 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T05:10:28.884 DEBUG:teuthology.orchestra.run.smithi113:> true 2024-03-28T05:10:28.954 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-03-28T05:10:28.954 INFO:teuthology.run_tasks:Running task clock... 2024-03-28T05:10:28.965 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-28T05:10:28.966 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T05:10:28.966 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T05:10:28.968 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T05:10:28.968 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T05:10:28.970 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T05:10:28.970 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T05:10:28.998 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T05:10:28.999 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T05:10:29.014 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T05:10:29.016 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T05:10:29.027 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T05:10:29.044 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T05:10:29.044 INFO:teuthology.orchestra.run.smithi023.stderr:sudo: ntpd: command not found 2024-03-28T05:10:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-03-28T05:10:29.057 INFO:teuthology.orchestra.run.smithi023.stdout:506 Cannot talk to daemon 2024-03-28T05:10:29.059 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-03-28T05:10:29.074 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: ntpd: command not found 2024-03-28T05:10:29.074 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T05:10:29.075 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T05:10:29.087 INFO:teuthology.orchestra.run.smithi113.stdout:506 Cannot talk to daemon 2024-03-28T05:10:29.090 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T05:10:29.091 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T05:10:29.104 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T05:10:29.121 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T05:10:29.135 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-03-28T05:10:29.138 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T05:10:29.139 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-03-28T05:10:29.139 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T05:10:29.139 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T05:10:29.139 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T05:10:29.139 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T05:10:29.148 INFO:teuthology.orchestra.run.smithi023.stderr:bash: line 1: ntpq: command not found 2024-03-28T05:10:29.152 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T05:10:29.152 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2024-03-28T05:10:29.152 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T05:10:29.152 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T05:10:29.152 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T05:10:29.152 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T05:10:29.164 INFO:teuthology.orchestra.run.smithi113.stderr:bash: line 1: ntpq: command not found 2024-03-28T05:10:29.168 INFO:teuthology.orchestra.run.smithi113.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T05:10:29.168 INFO:teuthology.orchestra.run.smithi113.stdout:=============================================================================== 2024-03-28T05:10:29.168 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T05:10:29.168 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T05:10:29.168 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T05:10:29.168 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T05:10:29.169 INFO:teuthology.run_tasks:Running task install... 2024-03-28T05:10:29.177 DEBUG:teuthology.task.install:project ceph 2024-03-28T05:10:29.177 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}} 2024-03-28T05:10:29.177 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'} 2024-03-28T05:10:29.177 INFO:teuthology.task.install:Using flavor: default 2024-03-28T05:10:29.184 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-28T05:10:29.185 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-28T05:10:29.185 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T05:10:29.185 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T05:10:29.187 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T05:10:29.187 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T05:10:29.188 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T05:10:29.188 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T05:10:29.401 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T05:10:29.401 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T05:10:29.405 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T05:10:29.405 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T05:10:29.463 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T05:10:29.463 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T05:10:29.572 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T05:10:29.572 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T05:10:29.572 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T05:10:29.574 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T05:10:29.574 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-28T05:10:29.574 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T05:10:29.584 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T05:10:29.584 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-28T05:10:29.584 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T05:10:29.602 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T05:10:29.602 DEBUG:teuthology.orchestra.run.smithi023:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T05:10:29.603 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T05:10:29.603 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T05:10:29.612 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T05:10:29.612 DEBUG:teuthology.orchestra.run.smithi113:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T05:10:29.669 DEBUG:teuthology.orchestra.run.smithi023:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T05:10:29.671 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T05:10:29.681 DEBUG:teuthology.orchestra.run.smithi113:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T05:10:29.749 DEBUG:teuthology.orchestra.run.smithi023:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T05:10:29.750 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T05:10:29.762 DEBUG:teuthology.orchestra.run.smithi113:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T05:10:29.815 INFO:teuthology.orchestra.run.smithi023.stdout:check_obsoletes = 1 2024-03-28T05:10:29.817 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum clean all 2024-03-28T05:10:29.817 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2024-03-28T05:10:29.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2024-03-28T05:10:29.828 INFO:teuthology.orchestra.run.smithi113.stdout:check_obsoletes = 1 2024-03-28T05:10:29.829 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum clean all 2024-03-28T05:10:30.050 INFO:teuthology.orchestra.run.smithi098.stdout:74 files removed 2024-03-28T05:10:30.063 INFO:teuthology.orchestra.run.smithi113.stdout:74 files removed 2024-03-28T05:10:30.072 INFO:teuthology.orchestra.run.smithi023.stdout:74 files removed 2024-03-28T05:10:30.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T05:10:30.090 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T05:10:30.099 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T05:10:30.722 INFO:teuthology.orchestra.run.smithi023.stdout:ceph packages for x86_64 262 kB/s | 83 kB 00:00 2024-03-28T05:10:30.773 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 204 kB/s | 83 kB 00:00 2024-03-28T05:10:30.805 INFO:teuthology.orchestra.run.smithi113.stdout:ceph packages for x86_64 195 kB/s | 83 kB 00:00 2024-03-28T05:10:31.039 INFO:teuthology.orchestra.run.smithi023.stdout:ceph noarch packages 38 kB/s | 10 kB 00:00 2024-03-28T05:10:31.091 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 36 kB/s | 10 kB 00:00 2024-03-28T05:10:31.156 INFO:teuthology.orchestra.run.smithi113.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-28T05:10:31.442 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-03-28T05:10:31.464 INFO:teuthology.orchestra.run.smithi023.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-03-28T05:10:31.514 INFO:teuthology.orchestra.run.smithi113.stdout:ceph source packages 5.8 kB/s | 1.9 kB 00:00 2024-03-28T05:10:31.928 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-28T05:10:32.142 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-28T05:10:32.183 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-28T05:10:33.353 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-28T05:10:33.600 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-28T05:10:33.667 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.0 MB 00:00 2024-03-28T05:10:34.177 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-28T05:10:34.383 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-28T05:10:35.970 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-03-28T05:10:36.362 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-28T05:10:36.441 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - BaseOS 3.3 MB/s | 8.0 MB 00:02 2024-03-28T05:10:39.814 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-28T05:10:39.834 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - AppStream 8.1 MB/s | 19 MB 00:02 2024-03-28T05:10:40.377 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - CRB 17 MB/s | 6.1 MB 00:00 2024-03-28T05:10:41.247 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 16 kB 00:00 2024-03-28T05:10:41.786 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - Extras packages 85 kB/s | 16 kB 00:00 2024-03-28T05:10:42.688 INFO:teuthology.orchestra.run.smithi023.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-03-28T05:10:44.166 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 7.4 MB/s | 21 MB 00:02 2024-03-28T05:10:44.268 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - CRB 6.1 MB/s | 6.1 MB 00:00 2024-03-28T05:10:45.652 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 16 kB 00:00 2024-03-28T05:10:46.837 INFO:teuthology.orchestra.run.smithi113.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-28T05:10:47.813 INFO:teuthology.orchestra.run.smithi023.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-28T05:10:48.983 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-28T05:10:50.333 INFO:teuthology.orchestra.run.smithi023.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T05:10:50.334 INFO:teuthology.orchestra.run.smithi023.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T05:10:50.349 INFO:teuthology.orchestra.run.smithi023.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T05:10:50.350 INFO:teuthology.orchestra.run.smithi023.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T05:10:50.351 INFO:teuthology.orchestra.run.smithi023.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T05:10:50.352 INFO:teuthology.orchestra.run.smithi023.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T05:10:50.354 INFO:teuthology.orchestra.run.smithi023.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T05:10:50.357 INFO:teuthology.orchestra.run.smithi023.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T05:10:50.359 INFO:teuthology.orchestra.run.smithi023.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T05:10:50.445 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2024-03-28T05:10:50.458 INFO:teuthology.orchestra.run.smithi023.stdout:============================================================================================ 2024-03-28T05:10:50.458 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2024-03-28T05:10:50.458 INFO:teuthology.orchestra.run.smithi023.stdout:============================================================================================ 2024-03-28T05:10:50.459 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2024-03-28T05:10:50.459 INFO:teuthology.orchestra.run.smithi023.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T05:10:50.459 INFO:teuthology.orchestra.run.smithi023.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T05:10:50.459 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T05:10:50.459 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T05:10:50.459 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T05:10:50.459 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T05:10:50.459 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T05:10:50.459 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T05:10:50.460 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T05:10:50.460 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T05:10:50.460 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T05:10:50.460 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T05:10:50.460 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T05:10:50.460 INFO:teuthology.orchestra.run.smithi023.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T05:10:50.461 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T05:10:50.461 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T05:10:50.461 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T05:10:50.461 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T05:10:50.461 INFO:teuthology.orchestra.run.smithi023.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T05:10:50.461 INFO:teuthology.orchestra.run.smithi023.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T05:10:50.461 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T05:10:50.461 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T05:10:50.461 INFO:teuthology.orchestra.run.smithi023.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T05:10:50.461 INFO:teuthology.orchestra.run.smithi023.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T05:10:50.462 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T05:10:50.462 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T05:10:50.462 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T05:10:50.462 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T05:10:50.462 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T05:10:50.462 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T05:10:50.462 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T05:10:50.462 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T05:10:50.462 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T05:10:50.462 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T05:10:50.463 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T05:10:50.463 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T05:10:50.463 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T05:10:50.463 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrading: 2024-03-28T05:10:50.463 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T05:10:50.463 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T05:10:50.463 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T05:10:50.463 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T05:10:50.463 INFO:teuthology.orchestra.run.smithi023.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T05:10:50.463 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T05:10:50.464 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T05:10:50.464 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T05:10:50.464 INFO:teuthology.orchestra.run.smithi023.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T05:10:50.464 INFO:teuthology.orchestra.run.smithi023.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T05:10:50.464 INFO:teuthology.orchestra.run.smithi023.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T05:10:50.464 INFO:teuthology.orchestra.run.smithi023.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T05:10:50.464 INFO:teuthology.orchestra.run.smithi023.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T05:10:50.464 INFO:teuthology.orchestra.run.smithi023.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T05:10:50.464 INFO:teuthology.orchestra.run.smithi023.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T05:10:50.464 INFO:teuthology.orchestra.run.smithi023.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T05:10:50.465 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T05:10:50.465 INFO:teuthology.orchestra.run.smithi023.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T05:10:50.465 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T05:10:50.465 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T05:10:50.465 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T05:10:50.465 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T05:10:50.465 INFO:teuthology.orchestra.run.smithi023.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T05:10:50.465 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T05:10:50.465 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2024-03-28T05:10:50.466 INFO:teuthology.orchestra.run.smithi023.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T05:10:50.466 INFO:teuthology.orchestra.run.smithi023.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T05:10:50.466 INFO:teuthology.orchestra.run.smithi023.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T05:10:50.466 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T05:10:50.466 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T05:10:50.466 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T05:10:50.466 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T05:10:50.466 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T05:10:50.466 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T05:10:50.467 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T05:10:50.467 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T05:10:50.467 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T05:10:50.467 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T05:10:50.467 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T05:10:50.467 INFO:teuthology.orchestra.run.smithi023.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T05:10:50.467 INFO:teuthology.orchestra.run.smithi023.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T05:10:50.467 INFO:teuthology.orchestra.run.smithi023.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T05:10:50.467 INFO:teuthology.orchestra.run.smithi023.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T05:10:50.467 INFO:teuthology.orchestra.run.smithi023.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T05:10:50.467 INFO:teuthology.orchestra.run.smithi023.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T05:10:50.468 INFO:teuthology.orchestra.run.smithi023.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T05:10:50.468 INFO:teuthology.orchestra.run.smithi023.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T05:10:50.468 INFO:teuthology.orchestra.run.smithi023.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T05:10:50.468 INFO:teuthology.orchestra.run.smithi023.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T05:10:50.468 INFO:teuthology.orchestra.run.smithi023.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T05:10:50.468 INFO:teuthology.orchestra.run.smithi023.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T05:10:50.468 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T05:10:50.468 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T05:10:50.468 INFO:teuthology.orchestra.run.smithi023.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T05:10:50.468 INFO:teuthology.orchestra.run.smithi023.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T05:10:50.468 INFO:teuthology.orchestra.run.smithi023.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T05:10:50.469 INFO:teuthology.orchestra.run.smithi023.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T05:10:50.469 INFO:teuthology.orchestra.run.smithi023.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T05:10:50.469 INFO:teuthology.orchestra.run.smithi023.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T05:10:50.469 INFO:teuthology.orchestra.run.smithi023.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T05:10:50.469 INFO:teuthology.orchestra.run.smithi023.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T05:10:50.469 INFO:teuthology.orchestra.run.smithi023.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T05:10:50.469 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T05:10:50.469 INFO:teuthology.orchestra.run.smithi023.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T05:10:50.469 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T05:10:50.469 INFO:teuthology.orchestra.run.smithi023.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T05:10:50.469 INFO:teuthology.orchestra.run.smithi023.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T05:10:50.470 INFO:teuthology.orchestra.run.smithi023.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T05:10:50.470 INFO:teuthology.orchestra.run.smithi023.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T05:10:50.470 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T05:10:50.470 INFO:teuthology.orchestra.run.smithi023.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T05:10:50.470 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T05:10:50.470 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T05:10:50.470 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T05:10:50.470 INFO:teuthology.orchestra.run.smithi023.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T05:10:50.470 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T05:10:50.470 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T05:10:50.471 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T05:10:50.471 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T05:10:50.471 INFO:teuthology.orchestra.run.smithi023.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T05:10:50.471 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T05:10:50.471 INFO:teuthology.orchestra.run.smithi023.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T05:10:50.471 INFO:teuthology.orchestra.run.smithi023.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T05:10:50.471 INFO:teuthology.orchestra.run.smithi023.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T05:10:50.471 INFO:teuthology.orchestra.run.smithi023.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T05:10:50.471 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T05:10:50.471 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T05:10:50.471 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T05:10:50.471 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T05:10:50.472 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T05:10:50.472 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T05:10:50.472 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T05:10:50.472 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T05:10:50.472 INFO:teuthology.orchestra.run.smithi023.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T05:10:50.472 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T05:10:50.472 INFO:teuthology.orchestra.run.smithi023.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T05:10:50.472 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T05:10:50.472 INFO:teuthology.orchestra.run.smithi023.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T05:10:50.472 INFO:teuthology.orchestra.run.smithi023.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T05:10:50.472 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T05:10:50.473 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T05:10:50.473 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T05:10:50.473 INFO:teuthology.orchestra.run.smithi023.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T05:10:50.473 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T05:10:50.473 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T05:10:50.473 INFO:teuthology.orchestra.run.smithi023.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T05:10:50.473 INFO:teuthology.orchestra.run.smithi023.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T05:10:50.473 INFO:teuthology.orchestra.run.smithi023.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T05:10:50.473 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T05:10:50.473 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T05:10:50.473 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T05:10:50.474 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T05:10:50.474 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T05:10:50.474 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T05:10:50.474 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T05:10:50.474 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T05:10:50.474 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T05:10:50.474 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T05:10:50.474 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T05:10:50.474 INFO:teuthology.orchestra.run.smithi023.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T05:10:50.474 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T05:10:50.474 INFO:teuthology.orchestra.run.smithi023.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T05:10:50.475 INFO:teuthology.orchestra.run.smithi023.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T05:10:50.475 INFO:teuthology.orchestra.run.smithi023.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T05:10:50.475 INFO:teuthology.orchestra.run.smithi023.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T05:10:50.475 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T05:10:50.475 INFO:teuthology.orchestra.run.smithi023.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T05:10:50.475 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T05:10:50.475 INFO:teuthology.orchestra.run.smithi023.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T05:10:50.475 INFO:teuthology.orchestra.run.smithi023.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T05:10:50.475 INFO:teuthology.orchestra.run.smithi023.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T05:10:50.475 INFO:teuthology.orchestra.run.smithi023.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T05:10:50.476 INFO:teuthology.orchestra.run.smithi023.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T05:10:50.476 INFO:teuthology.orchestra.run.smithi023.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T05:10:50.476 INFO:teuthology.orchestra.run.smithi023.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T05:10:50.476 INFO:teuthology.orchestra.run.smithi023.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T05:10:50.476 INFO:teuthology.orchestra.run.smithi023.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T05:10:50.476 INFO:teuthology.orchestra.run.smithi023.stdout:Installing weak dependencies: 2024-03-28T05:10:50.476 INFO:teuthology.orchestra.run.smithi023.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T05:10:50.476 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:10:50.476 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2024-03-28T05:10:50.476 INFO:teuthology.orchestra.run.smithi023.stdout:============================================================================================ 2024-03-28T05:10:50.476 INFO:teuthology.orchestra.run.smithi023.stdout:Install 150 Packages 2024-03-28T05:10:50.477 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrade 24 Packages 2024-03-28T05:10:50.477 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:10:50.477 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 225 M 2024-03-28T05:10:50.477 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2024-03-28T05:10:51.339 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T05:10:51.340 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T05:10:51.347 INFO:teuthology.orchestra.run.smithi023.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 28 kB/s | 6.6 kB 00:00 2024-03-28T05:10:51.353 INFO:teuthology.orchestra.run.smithi098.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T05:10:51.354 INFO:teuthology.orchestra.run.smithi098.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T05:10:51.355 INFO:teuthology.orchestra.run.smithi098.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T05:10:51.357 INFO:teuthology.orchestra.run.smithi098.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T05:10:51.358 INFO:teuthology.orchestra.run.smithi098.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T05:10:51.362 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T05:10:51.363 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T05:10:51.441 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-28T05:10:51.468 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-28T05:10:51.468 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-28T05:10:51.468 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-28T05:10:51.468 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-28T05:10:51.469 INFO:teuthology.orchestra.run.smithi098.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T05:10:51.469 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T05:10:51.469 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T05:10:51.469 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T05:10:51.469 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T05:10:51.469 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T05:10:51.469 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T05:10:51.469 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T05:10:51.469 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T05:10:51.469 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T05:10:51.469 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T05:10:51.469 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T05:10:51.470 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T05:10:51.470 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T05:10:51.470 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T05:10:51.470 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T05:10:51.470 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T05:10:51.470 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T05:10:51.470 INFO:teuthology.orchestra.run.smithi098.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T05:10:51.470 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T05:10:51.470 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T05:10:51.470 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T05:10:51.470 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T05:10:51.470 INFO:teuthology.orchestra.run.smithi098.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T05:10:51.470 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T05:10:51.470 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T05:10:51.471 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T05:10:51.471 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T05:10:51.471 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T05:10:51.471 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T05:10:51.471 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T05:10:51.471 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T05:10:51.471 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T05:10:51.471 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T05:10:51.471 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T05:10:51.471 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T05:10:51.471 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T05:10:51.471 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2024-03-28T05:10:51.471 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T05:10:51.471 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T05:10:51.471 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T05:10:51.472 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T05:10:51.472 INFO:teuthology.orchestra.run.smithi098.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T05:10:51.472 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T05:10:51.472 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T05:10:51.472 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T05:10:51.472 INFO:teuthology.orchestra.run.smithi098.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T05:10:51.472 INFO:teuthology.orchestra.run.smithi098.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T05:10:51.472 INFO:teuthology.orchestra.run.smithi098.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T05:10:51.472 INFO:teuthology.orchestra.run.smithi098.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T05:10:51.472 INFO:teuthology.orchestra.run.smithi098.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T05:10:51.472 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T05:10:51.472 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T05:10:51.472 INFO:teuthology.orchestra.run.smithi098.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T05:10:51.472 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T05:10:51.472 INFO:teuthology.orchestra.run.smithi098.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T05:10:51.472 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T05:10:51.473 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T05:10:51.473 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T05:10:51.473 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T05:10:51.473 INFO:teuthology.orchestra.run.smithi098.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T05:10:51.473 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T05:10:51.473 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-28T05:10:51.473 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T05:10:51.473 INFO:teuthology.orchestra.run.smithi098.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T05:10:51.473 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T05:10:51.473 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T05:10:51.473 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T05:10:51.473 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T05:10:51.473 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T05:10:51.473 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T05:10:51.473 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T05:10:51.474 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T05:10:51.474 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T05:10:51.474 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T05:10:51.474 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T05:10:51.474 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T05:10:51.474 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T05:10:51.474 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T05:10:51.474 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T05:10:51.474 INFO:teuthology.orchestra.run.smithi098.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T05:10:51.474 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T05:10:51.474 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T05:10:51.474 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T05:10:51.474 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T05:10:51.474 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T05:10:51.475 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T05:10:51.475 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T05:10:51.475 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T05:10:51.475 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T05:10:51.475 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T05:10:51.475 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T05:10:51.475 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T05:10:51.475 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T05:10:51.475 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T05:10:51.475 INFO:teuthology.orchestra.run.smithi098.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T05:10:51.475 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T05:10:51.475 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T05:10:51.475 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T05:10:51.476 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T05:10:51.476 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T05:10:51.476 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T05:10:51.476 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T05:10:51.476 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T05:10:51.476 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T05:10:51.476 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T05:10:51.476 INFO:teuthology.orchestra.run.smithi098.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T05:10:51.476 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T05:10:51.476 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T05:10:51.476 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T05:10:51.476 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T05:10:51.476 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T05:10:51.476 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T05:10:51.476 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T05:10:51.476 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T05:10:51.476 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T05:10:51.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T05:10:51.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T05:10:51.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T05:10:51.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T05:10:51.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T05:10:51.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T05:10:51.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T05:10:51.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T05:10:51.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T05:10:51.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T05:10:51.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T05:10:51.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T05:10:51.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T05:10:51.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T05:10:51.477 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T05:10:51.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T05:10:51.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T05:10:51.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T05:10:51.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T05:10:51.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T05:10:51.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T05:10:51.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T05:10:51.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T05:10:51.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T05:10:51.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T05:10:51.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T05:10:51.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T05:10:51.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T05:10:51.478 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T05:10:51.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T05:10:51.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T05:10:51.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T05:10:51.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T05:10:51.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T05:10:51.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T05:10:51.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T05:10:51.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T05:10:51.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T05:10:51.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T05:10:51.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T05:10:51.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T05:10:51.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T05:10:51.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T05:10:51.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T05:10:51.479 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T05:10:51.480 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T05:10:51.480 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T05:10:51.480 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T05:10:51.480 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T05:10:51.480 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T05:10:51.480 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T05:10:51.480 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T05:10:51.480 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T05:10:51.480 INFO:teuthology.orchestra.run.smithi098.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T05:10:51.480 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T05:10:51.480 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T05:10:51.480 INFO:teuthology.orchestra.run.smithi098.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T05:10:51.480 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T05:10:51.480 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T05:10:51.480 INFO:teuthology.orchestra.run.smithi098.stdout:Installing weak dependencies: 2024-03-28T05:10:51.480 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T05:10:51.481 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-28T05:10:51.481 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-28T05:10:51.481 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-28T05:10:51.481 INFO:teuthology.orchestra.run.smithi098.stdout:Install 150 Packages 2024-03-28T05:10:51.481 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 24 Packages 2024-03-28T05:10:51.481 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-28T05:10:51.481 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 225 M 2024-03-28T05:10:51.481 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-28T05:10:51.789 INFO:teuthology.orchestra.run.smithi023.stdout:(2/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 7.9 MB/s | 5.4 MB 00:00 2024-03-28T05:10:51.848 INFO:teuthology.orchestra.run.smithi023.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 2.4 MB/s | 145 kB 00:00 2024-03-28T05:10:51.861 INFO:teuthology.orchestra.run.smithi113.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-03-28T05:10:52.015 INFO:teuthology.orchestra.run.smithi023.stdout:(4/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.3 MB/s | 890 kB 00:00 2024-03-28T05:10:52.106 INFO:teuthology.orchestra.run.smithi023.stdout:(5/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 8.9 MB/s | 2.3 MB 00:00 2024-03-28T05:10:52.217 INFO:teuthology.orchestra.run.smithi098.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 32 kB/s | 6.6 kB 00:00 2024-03-28T05:10:52.648 INFO:teuthology.orchestra.run.smithi023.stdout:(6/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 8.7 MB/s | 4.7 MB 00:00 2024-03-28T05:10:52.840 INFO:teuthology.orchestra.run.smithi023.stdout:(7/174): ceph-common-19.0.0-2145.g3be9a5c9.el9. 12 MB/s | 20 MB 00:01 2024-03-28T05:10:52.859 INFO:teuthology.orchestra.run.smithi098.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.4 MB/s | 890 kB 00:00 2024-03-28T05:10:52.907 INFO:teuthology.orchestra.run.smithi023.stdout:(8/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 1.7 MB/s | 1.5 MB 00:00 2024-03-28T05:10:52.959 INFO:teuthology.orchestra.run.smithi098.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 1.4 MB/s | 145 kB 00:00 2024-03-28T05:10:52.991 INFO:teuthology.orchestra.run.smithi023.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 302 kB/s | 25 kB 00:00 2024-03-28T05:10:53.558 INFO:teuthology.orchestra.run.smithi023.stdout:(10/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 14 MB/s | 9.9 MB 00:00 2024-03-28T05:10:53.625 INFO:teuthology.orchestra.run.smithi023.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 3.2 MB/s | 220 kB 00:00 2024-03-28T05:10:53.683 INFO:teuthology.orchestra.run.smithi023.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 464 kB/s | 27 kB 00:00 2024-03-28T05:10:53.725 INFO:teuthology.orchestra.run.smithi023.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 755 kB/s | 31 kB 00:00 2024-03-28T05:10:53.768 INFO:teuthology.orchestra.run.smithi098.stdout:(4/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 3.0 MB/s | 5.4 MB 00:01 2024-03-28T05:10:53.817 INFO:teuthology.orchestra.run.smithi023.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 7.9 MB/s | 746 kB 00:00 2024-03-28T05:10:53.818 INFO:teuthology.orchestra.run.smithi098.stdout:(5/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 2.7 MB/s | 2.3 MB 00:00 2024-03-28T05:10:53.892 INFO:teuthology.orchestra.run.smithi023.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 2.1 MB/s | 163 kB 00:00 2024-03-28T05:10:53.967 INFO:teuthology.orchestra.run.smithi023.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.7 MB/s | 127 kB 00:00 2024-03-28T05:10:54.027 INFO:teuthology.orchestra.run.smithi098.stdout:(6/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 5.8 MB/s | 1.5 MB 00:00 2024-03-28T05:10:54.043 INFO:teuthology.orchestra.run.smithi023.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 6.5 MB/s | 501 kB 00:00 2024-03-28T05:10:54.237 INFO:teuthology.orchestra.run.smithi113.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T05:10:54.237 INFO:teuthology.orchestra.run.smithi113.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T05:10:54.249 INFO:teuthology.orchestra.run.smithi113.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T05:10:54.250 INFO:teuthology.orchestra.run.smithi113.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T05:10:54.251 INFO:teuthology.orchestra.run.smithi113.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T05:10:54.252 INFO:teuthology.orchestra.run.smithi113.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T05:10:54.253 INFO:teuthology.orchestra.run.smithi113.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T05:10:54.256 INFO:teuthology.orchestra.run.smithi113.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T05:10:54.257 INFO:teuthology.orchestra.run.smithi113.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T05:10:54.337 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-28T05:10:54.357 INFO:teuthology.orchestra.run.smithi113.stdout:============================================================================================ 2024-03-28T05:10:54.357 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-03-28T05:10:54.357 INFO:teuthology.orchestra.run.smithi113.stdout:============================================================================================ 2024-03-28T05:10:54.357 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-03-28T05:10:54.357 INFO:teuthology.orchestra.run.smithi113.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T05:10:54.358 INFO:teuthology.orchestra.run.smithi113.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T05:10:54.358 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T05:10:54.358 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T05:10:54.358 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T05:10:54.358 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T05:10:54.358 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T05:10:54.358 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T05:10:54.358 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T05:10:54.358 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T05:10:54.358 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T05:10:54.358 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T05:10:54.358 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T05:10:54.358 INFO:teuthology.orchestra.run.smithi113.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T05:10:54.358 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T05:10:54.359 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T05:10:54.359 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T05:10:54.359 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T05:10:54.359 INFO:teuthology.orchestra.run.smithi113.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T05:10:54.359 INFO:teuthology.orchestra.run.smithi113.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T05:10:54.359 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T05:10:54.359 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T05:10:54.359 INFO:teuthology.orchestra.run.smithi113.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T05:10:54.359 INFO:teuthology.orchestra.run.smithi113.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T05:10:54.359 INFO:teuthology.orchestra.run.smithi113.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T05:10:54.359 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T05:10:54.359 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T05:10:54.359 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T05:10:54.359 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T05:10:54.359 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T05:10:54.359 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T05:10:54.360 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T05:10:54.360 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T05:10:54.360 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T05:10:54.360 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T05:10:54.360 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T05:10:54.360 INFO:teuthology.orchestra.run.smithi113.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T05:10:54.360 INFO:teuthology.orchestra.run.smithi113.stdout:Upgrading: 2024-03-28T05:10:54.360 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T05:10:54.360 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T05:10:54.360 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T05:10:54.360 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T05:10:54.360 INFO:teuthology.orchestra.run.smithi113.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T05:10:54.360 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T05:10:54.360 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T05:10:54.361 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T05:10:54.361 INFO:teuthology.orchestra.run.smithi113.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T05:10:54.361 INFO:teuthology.orchestra.run.smithi113.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T05:10:54.361 INFO:teuthology.orchestra.run.smithi113.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T05:10:54.361 INFO:teuthology.orchestra.run.smithi113.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T05:10:54.361 INFO:teuthology.orchestra.run.smithi113.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T05:10:54.361 INFO:teuthology.orchestra.run.smithi113.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T05:10:54.361 INFO:teuthology.orchestra.run.smithi113.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T05:10:54.361 INFO:teuthology.orchestra.run.smithi113.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T05:10:54.361 INFO:teuthology.orchestra.run.smithi113.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T05:10:54.361 INFO:teuthology.orchestra.run.smithi113.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T05:10:54.361 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T05:10:54.361 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T05:10:54.361 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T05:10:54.362 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T05:10:54.362 INFO:teuthology.orchestra.run.smithi113.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T05:10:54.362 INFO:teuthology.orchestra.run.smithi113.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T05:10:54.362 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-03-28T05:10:54.362 INFO:teuthology.orchestra.run.smithi113.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T05:10:54.362 INFO:teuthology.orchestra.run.smithi113.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T05:10:54.362 INFO:teuthology.orchestra.run.smithi113.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T05:10:54.362 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T05:10:54.362 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T05:10:54.362 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T05:10:54.362 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T05:10:54.362 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T05:10:54.362 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T05:10:54.363 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T05:10:54.363 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T05:10:54.363 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T05:10:54.363 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T05:10:54.363 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T05:10:54.363 INFO:teuthology.orchestra.run.smithi113.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T05:10:54.363 INFO:teuthology.orchestra.run.smithi113.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T05:10:54.363 INFO:teuthology.orchestra.run.smithi113.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T05:10:54.363 INFO:teuthology.orchestra.run.smithi113.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T05:10:54.363 INFO:teuthology.orchestra.run.smithi113.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T05:10:54.363 INFO:teuthology.orchestra.run.smithi113.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T05:10:54.363 INFO:teuthology.orchestra.run.smithi113.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T05:10:54.363 INFO:teuthology.orchestra.run.smithi113.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T05:10:54.363 INFO:teuthology.orchestra.run.smithi113.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T05:10:54.363 INFO:teuthology.orchestra.run.smithi113.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T05:10:54.364 INFO:teuthology.orchestra.run.smithi113.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T05:10:54.364 INFO:teuthology.orchestra.run.smithi113.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T05:10:54.364 INFO:teuthology.orchestra.run.smithi113.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T05:10:54.364 INFO:teuthology.orchestra.run.smithi113.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T05:10:54.364 INFO:teuthology.orchestra.run.smithi113.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T05:10:54.364 INFO:teuthology.orchestra.run.smithi113.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T05:10:54.364 INFO:teuthology.orchestra.run.smithi113.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T05:10:54.364 INFO:teuthology.orchestra.run.smithi113.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T05:10:54.364 INFO:teuthology.orchestra.run.smithi113.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T05:10:54.364 INFO:teuthology.orchestra.run.smithi113.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T05:10:54.364 INFO:teuthology.orchestra.run.smithi113.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T05:10:54.364 INFO:teuthology.orchestra.run.smithi113.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T05:10:54.364 INFO:teuthology.orchestra.run.smithi113.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T05:10:54.364 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T05:10:54.364 INFO:teuthology.orchestra.run.smithi113.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T05:10:54.364 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T05:10:54.365 INFO:teuthology.orchestra.run.smithi113.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T05:10:54.365 INFO:teuthology.orchestra.run.smithi113.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T05:10:54.365 INFO:teuthology.orchestra.run.smithi113.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T05:10:54.365 INFO:teuthology.orchestra.run.smithi113.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T05:10:54.365 INFO:teuthology.orchestra.run.smithi113.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T05:10:54.365 INFO:teuthology.orchestra.run.smithi113.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T05:10:54.365 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T05:10:54.365 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T05:10:54.365 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T05:10:54.365 INFO:teuthology.orchestra.run.smithi113.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T05:10:54.365 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T05:10:54.365 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T05:10:54.365 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T05:10:54.365 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T05:10:54.365 INFO:teuthology.orchestra.run.smithi113.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T05:10:54.365 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T05:10:54.366 INFO:teuthology.orchestra.run.smithi113.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T05:10:54.366 INFO:teuthology.orchestra.run.smithi113.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T05:10:54.366 INFO:teuthology.orchestra.run.smithi113.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T05:10:54.366 INFO:teuthology.orchestra.run.smithi113.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T05:10:54.366 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T05:10:54.366 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T05:10:54.366 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T05:10:54.366 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T05:10:54.366 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T05:10:54.366 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T05:10:54.366 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T05:10:54.366 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T05:10:54.366 INFO:teuthology.orchestra.run.smithi113.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T05:10:54.367 INFO:teuthology.orchestra.run.smithi113.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T05:10:54.367 INFO:teuthology.orchestra.run.smithi113.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T05:10:54.367 INFO:teuthology.orchestra.run.smithi113.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T05:10:54.367 INFO:teuthology.orchestra.run.smithi113.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T05:10:54.367 INFO:teuthology.orchestra.run.smithi113.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T05:10:54.367 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T05:10:54.367 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T05:10:54.367 INFO:teuthology.orchestra.run.smithi113.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T05:10:54.367 INFO:teuthology.orchestra.run.smithi113.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T05:10:54.367 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T05:10:54.367 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T05:10:54.367 INFO:teuthology.orchestra.run.smithi113.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T05:10:54.367 INFO:teuthology.orchestra.run.smithi113.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T05:10:54.367 INFO:teuthology.orchestra.run.smithi113.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T05:10:54.367 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T05:10:54.368 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T05:10:54.368 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T05:10:54.368 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T05:10:54.368 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T05:10:54.368 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T05:10:54.368 INFO:teuthology.orchestra.run.smithi113.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T05:10:54.368 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T05:10:54.368 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T05:10:54.368 INFO:teuthology.orchestra.run.smithi113.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T05:10:54.368 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T05:10:54.368 INFO:teuthology.orchestra.run.smithi113.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T05:10:54.368 INFO:teuthology.orchestra.run.smithi113.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T05:10:54.368 INFO:teuthology.orchestra.run.smithi113.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T05:10:54.368 INFO:teuthology.orchestra.run.smithi113.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T05:10:54.368 INFO:teuthology.orchestra.run.smithi113.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T05:10:54.368 INFO:teuthology.orchestra.run.smithi113.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T05:10:54.368 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T05:10:54.369 INFO:teuthology.orchestra.run.smithi113.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T05:10:54.369 INFO:teuthology.orchestra.run.smithi113.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T05:10:54.369 INFO:teuthology.orchestra.run.smithi113.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T05:10:54.369 INFO:teuthology.orchestra.run.smithi113.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T05:10:54.369 INFO:teuthology.orchestra.run.smithi113.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T05:10:54.369 INFO:teuthology.orchestra.run.smithi113.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T05:10:54.369 INFO:teuthology.orchestra.run.smithi113.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T05:10:54.369 INFO:teuthology.orchestra.run.smithi113.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T05:10:54.369 INFO:teuthology.orchestra.run.smithi113.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T05:10:54.369 INFO:teuthology.orchestra.run.smithi113.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T05:10:54.369 INFO:teuthology.orchestra.run.smithi113.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T05:10:54.369 INFO:teuthology.orchestra.run.smithi113.stdout:Installing weak dependencies: 2024-03-28T05:10:54.369 INFO:teuthology.orchestra.run.smithi113.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T05:10:54.369 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-28T05:10:54.369 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-03-28T05:10:54.369 INFO:teuthology.orchestra.run.smithi113.stdout:============================================================================================ 2024-03-28T05:10:54.370 INFO:teuthology.orchestra.run.smithi113.stdout:Install 150 Packages 2024-03-28T05:10:54.370 INFO:teuthology.orchestra.run.smithi113.stdout:Upgrade 24 Packages 2024-03-28T05:10:54.370 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-28T05:10:54.370 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 225 M 2024-03-28T05:10:54.370 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-03-28T05:10:54.560 INFO:teuthology.orchestra.run.smithi023.stdout:(18/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 10 MB/s | 5.1 MB 00:00 2024-03-28T05:10:54.618 INFO:teuthology.orchestra.run.smithi023.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 775 kB/s | 45 kB 00:00 2024-03-28T05:10:54.677 INFO:teuthology.orchestra.run.smithi023.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 2.2 MB/s | 131 kB 00:00 2024-03-28T05:10:54.743 INFO:teuthology.orchestra.run.smithi023.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 2.4 MB/s | 162 kB 00:00 2024-03-28T05:10:54.819 INFO:teuthology.orchestra.run.smithi023.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 4.2 MB/s | 321 kB 00:00 2024-03-28T05:10:54.894 INFO:teuthology.orchestra.run.smithi023.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 3.9 MB/s | 298 kB 00:00 2024-03-28T05:10:54.936 INFO:teuthology.orchestra.run.smithi098.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 4.2 MB/s | 4.7 MB 00:01 2024-03-28T05:10:54.945 INFO:teuthology.orchestra.run.smithi023.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.9 MB/s | 99 kB 00:00 2024-03-28T05:10:55.011 INFO:teuthology.orchestra.run.smithi023.stdout:(25/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.3 MB/s | 85 kB 00:00 2024-03-28T05:10:55.378 INFO:teuthology.orchestra.run.smithi023.stdout:(26/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 8.4 MB/s | 3.1 MB 00:00 2024-03-28T05:10:55.387 INFO:teuthology.orchestra.run.smithi113.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-03-28T05:10:55.579 INFO:teuthology.orchestra.run.smithi023.stdout:(27/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 5.7 MB/s | 17 MB 00:02 2024-03-28T05:10:55.604 INFO:teuthology.orchestra.run.smithi023.stdout:(28/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 758 kB/s | 170 kB 00:00 2024-03-28T05:10:55.662 INFO:teuthology.orchestra.run.smithi023.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2145.g 407 kB/s | 34 kB 00:00 2024-03-28T05:10:55.688 INFO:teuthology.orchestra.run.smithi023.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 1.6 MB/s | 139 kB 00:00 2024-03-28T05:10:55.737 INFO:teuthology.orchestra.run.smithi113.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 2.5 MB/s | 890 kB 00:00 2024-03-28T05:10:56.130 INFO:teuthology.orchestra.run.smithi023.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 3.8 MB/s | 1.8 MB 00:00 2024-03-28T05:10:56.187 INFO:teuthology.orchestra.run.smithi113.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 322 kB/s | 145 kB 00:00 2024-03-28T05:10:56.222 INFO:teuthology.orchestra.run.smithi023.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 2.7 MB/s | 248 kB 00:00 2024-03-28T05:10:56.322 INFO:teuthology.orchestra.run.smithi023.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 493 kB/s | 49 kB 00:00 2024-03-28T05:10:56.414 INFO:teuthology.orchestra.run.smithi023.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 180 kB/s | 16 kB 00:00 2024-03-28T05:10:56.514 INFO:teuthology.orchestra.run.smithi023.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 2.6 MB/s | 264 kB 00:00 2024-03-28T05:10:56.790 INFO:teuthology.orchestra.run.smithi023.stdout:(36/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 2.7 MB/s | 748 kB 00:00 2024-03-28T05:10:56.882 INFO:teuthology.orchestra.run.smithi023.stdout:(37/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 199 kB/s | 18 kB 00:00 2024-03-28T05:10:57.007 INFO:teuthology.orchestra.run.smithi023.stdout:(38/174): ceph-mgr-diskprediction-local-19.0.0- 5.6 MB/s | 7.4 MB 00:01 2024-03-28T05:10:57.046 INFO:teuthology.orchestra.run.smithi113.stdout:(4/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 2.7 MB/s | 2.3 MB 00:00 2024-03-28T05:10:57.102 INFO:teuthology.orchestra.run.smithi098.stdout:(8/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x86 5.5 MB/s | 17 MB 00:03 2024-03-28T05:10:57.107 INFO:teuthology.orchestra.run.smithi023.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.9 MB/s | 662 kB 00:00 2024-03-28T05:10:57.149 INFO:teuthology.orchestra.run.smithi023.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 6.4 MB/s | 920 kB 00:00 2024-03-28T05:10:57.161 INFO:teuthology.orchestra.run.smithi098.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 434 kB/s | 25 kB 00:00 2024-03-28T05:10:57.174 INFO:teuthology.orchestra.run.smithi023.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-28T05:10:57.200 INFO:teuthology.orchestra.run.smithi023.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-28T05:10:57.225 INFO:teuthology.orchestra.run.smithi023.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 639 kB/s | 32 kB 00:00 2024-03-28T05:10:57.300 INFO:teuthology.orchestra.run.smithi023.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-28T05:10:57.311 INFO:teuthology.orchestra.run.smithi098.stdout:(10/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 4.1 MB/s | 9.9 MB 00:02 2024-03-28T05:10:57.325 INFO:teuthology.orchestra.run.smithi023.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 176 kB/s | 17 kB 00:00 2024-03-28T05:10:57.351 INFO:teuthology.orchestra.run.smithi023.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-03-28T05:10:57.363 INFO:teuthology.orchestra.run.smithi113.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 4.7 MB/s | 1.5 MB 00:00 2024-03-28T05:10:57.376 INFO:teuthology.orchestra.run.smithi023.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-28T05:10:57.386 INFO:teuthology.orchestra.run.smithi098.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 2.9 MB/s | 220 kB 00:00 2024-03-28T05:10:57.401 INFO:teuthology.orchestra.run.smithi023.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 886 kB/s | 44 kB 00:00 2024-03-28T05:10:57.427 INFO:teuthology.orchestra.run.smithi023.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-28T05:10:57.437 INFO:teuthology.orchestra.run.smithi098.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 542 kB/s | 27 kB 00:00 2024-03-28T05:10:57.460 INFO:teuthology.orchestra.run.smithi023.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.9 MB/s | 578 kB 00:00 2024-03-28T05:10:57.485 INFO:teuthology.orchestra.run.smithi023.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 806 kB/s | 46 kB 00:00 2024-03-28T05:10:57.495 INFO:teuthology.orchestra.run.smithi098.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 538 kB/s | 31 kB 00:00 2024-03-28T05:10:57.511 INFO:teuthology.orchestra.run.smithi023.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-28T05:10:57.536 INFO:teuthology.orchestra.run.smithi023.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-28T05:10:57.561 INFO:teuthology.orchestra.run.smithi023.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-28T05:10:57.586 INFO:teuthology.orchestra.run.smithi023.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 688 kB/s | 34 kB 00:00 2024-03-28T05:10:57.679 INFO:teuthology.orchestra.run.smithi098.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 4.0 MB/s | 746 kB 00:00 2024-03-28T05:10:57.695 INFO:teuthology.orchestra.run.smithi023.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 36 MB/s | 4.7 MB 00:00 2024-03-28T05:10:57.729 INFO:teuthology.orchestra.run.smithi023.stdout:(57/174): protobuf-3.14.0-13.el9.x86_64.rpm 7.3 MB/s | 1.0 MB 00:00 2024-03-28T05:10:57.754 INFO:teuthology.orchestra.run.smithi023.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-28T05:10:57.755 INFO:teuthology.orchestra.run.smithi098.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 2.1 MB/s | 163 kB 00:00 2024-03-28T05:10:57.838 INFO:teuthology.orchestra.run.smithi098.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.5 MB/s | 127 kB 00:00 2024-03-28T05:10:57.972 INFO:teuthology.orchestra.run.smithi098.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 3.7 MB/s | 501 kB 00:00 2024-03-28T05:10:57.979 INFO:teuthology.orchestra.run.smithi023.stdout:(59/174): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2024-03-28T05:10:57.996 INFO:teuthology.orchestra.run.smithi023.stdout:(60/174): python3-jsonpatch-1.21-16.el9.noarch. 102 kB/s | 24 kB 00:00 2024-03-28T05:10:58.038 INFO:teuthology.orchestra.run.smithi023.stdout:(61/174): python3-jsonpointer-2.0-4.el9.noarch. 434 kB/s | 18 kB 00:00 2024-03-28T05:10:58.063 INFO:teuthology.orchestra.run.smithi023.stdout:(62/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-28T05:10:58.089 INFO:teuthology.orchestra.run.smithi023.stdout:(63/174): python3-markupsafe-1.1.1-12.el9.x86_6 632 kB/s | 32 kB 00:00 2024-03-28T05:10:58.239 INFO:teuthology.orchestra.run.smithi023.stdout:(64/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 4.8 MB 00:00 2024-03-28T05:10:58.265 INFO:teuthology.orchestra.run.smithi023.stdout:(65/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2024-03-28T05:10:58.290 INFO:teuthology.orchestra.run.smithi023.stdout:(66/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-28T05:10:58.315 INFO:teuthology.orchestra.run.smithi023.stdout:(67/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-28T05:10:58.330 INFO:teuthology.orchestra.run.smithi113.stdout:(6/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 1.6 MB/s | 5.4 MB 00:03 2024-03-28T05:10:58.340 INFO:teuthology.orchestra.run.smithi023.stdout:(68/174): python3-prettytable-0.7.2-27.el9.noar 822 kB/s | 41 kB 00:00 2024-03-28T05:10:58.366 INFO:teuthology.orchestra.run.smithi023.stdout:(69/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-28T05:10:58.391 INFO:teuthology.orchestra.run.smithi023.stdout:(70/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-28T05:10:58.416 INFO:teuthology.orchestra.run.smithi023.stdout:(71/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-28T05:10:58.441 INFO:teuthology.orchestra.run.smithi023.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 950 kB/s | 47 kB 00:00 2024-03-28T05:10:58.455 INFO:teuthology.orchestra.run.smithi113.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 4.3 MB/s | 4.7 MB 00:01 2024-03-28T05:10:58.467 INFO:teuthology.orchestra.run.smithi023.stdout:(73/174): python3-requests-oauthlib-1.3.0-12.el 857 kB/s | 43 kB 00:00 2024-03-28T05:10:58.684 INFO:teuthology.orchestra.run.smithi023.stdout:(74/174): python3-toml-0.10.2-6.el9.noarch.rpm 179 kB/s | 39 kB 00:00 2024-03-28T05:10:58.689 INFO:teuthology.orchestra.run.smithi113.stdout:(8/174): ceph-common-19.0.0-2145.g3be9a5c9.el9. 5.7 MB/s | 20 MB 00:03 2024-03-28T05:10:58.747 INFO:teuthology.orchestra.run.smithi113.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 434 kB/s | 25 kB 00:00 2024-03-28T05:10:58.839 INFO:teuthology.orchestra.run.smithi098.stdout:(18/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 3.0 MB/s | 20 MB 00:06 2024-03-28T05:10:58.914 INFO:teuthology.orchestra.run.smithi098.stdout:(19/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 5.5 MB/s | 5.1 MB 00:00 2024-03-28T05:10:58.931 INFO:teuthology.orchestra.run.smithi098.stdout:(20/174): python3-ceph-argparse-19.0.0-2145.g3b 492 kB/s | 45 kB 00:00 2024-03-28T05:10:58.990 INFO:teuthology.orchestra.run.smithi098.stdout:(21/174): python3-ceph-common-19.0.0-2145.g3be9 1.7 MB/s | 131 kB 00:00 2024-03-28T05:10:59.009 INFO:teuthology.orchestra.run.smithi023.stdout:(75/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-03-28T05:10:59.015 INFO:teuthology.orchestra.run.smithi098.stdout:(22/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 1.9 MB/s | 162 kB 00:00 2024-03-28T05:10:59.034 INFO:teuthology.orchestra.run.smithi023.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 118 kB/s | 41 kB 00:00 2024-03-28T05:10:59.060 INFO:teuthology.orchestra.run.smithi023.stdout:(77/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-28T05:10:59.085 INFO:teuthology.orchestra.run.smithi023.stdout:(78/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-28T05:10:59.107 INFO:teuthology.orchestra.run.smithi098.stdout:(23/174): python3-rados-19.0.0-2145.g3be9a5c9.e 2.7 MB/s | 321 kB 00:00 2024-03-28T05:10:59.110 INFO:teuthology.orchestra.run.smithi023.stdout:(79/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-28T05:10:59.135 INFO:teuthology.orchestra.run.smithi023.stdout:(80/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 878 kB/s | 44 kB 00:00 2024-03-28T05:10:59.157 INFO:teuthology.orchestra.run.smithi098.stdout:(24/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 2.1 MB/s | 298 kB 00:00 2024-03-28T05:10:59.160 INFO:teuthology.orchestra.run.smithi023.stdout:(81/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-28T05:10:59.183 INFO:teuthology.orchestra.run.smithi098.stdout:(25/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.3 MB/s | 99 kB 00:00 2024-03-28T05:10:59.186 INFO:teuthology.orchestra.run.smithi023.stdout:(82/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-28T05:10:59.228 INFO:teuthology.orchestra.run.smithi023.stdout:(83/174): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-03-28T05:10:59.241 INFO:teuthology.orchestra.run.smithi098.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.0 MB/s | 85 kB 00:00 2024-03-28T05:10:59.245 INFO:teuthology.orchestra.run.smithi023.stdout:(84/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-28T05:10:59.270 INFO:teuthology.orchestra.run.smithi023.stdout:(85/174): python3-pycparser-2.20-6.el9.noarch.r 2.9 MB/s | 124 kB 00:00 2024-03-28T05:10:59.295 INFO:teuthology.orchestra.run.smithi023.stdout:(86/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-28T05:10:59.333 INFO:teuthology.orchestra.run.smithi098.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-28T05:10:59.425 INFO:teuthology.orchestra.run.smithi098.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 370 kB/s | 34 kB 00:00 2024-03-28T05:10:59.525 INFO:teuthology.orchestra.run.smithi098.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 1.4 MB/s | 139 kB 00:00 2024-03-28T05:10:59.817 INFO:teuthology.orchestra.run.smithi098.stdout:(30/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 4.9 MB/s | 3.1 MB 00:00 2024-03-28T05:10:59.946 INFO:teuthology.orchestra.run.smithi023.stdout:(87/174): jq-1.6-16.el9.x86_64.rpm 277 kB/s | 187 kB 00:00 2024-03-28T05:11:00.085 INFO:teuthology.orchestra.run.smithi098.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 3.2 MB/s | 1.8 MB 00:00 2024-03-28T05:11:00.201 INFO:teuthology.orchestra.run.smithi098.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 2.1 MB/s | 248 kB 00:00 2024-03-28T05:11:00.287 INFO:teuthology.orchestra.run.smithi023.stdout:(88/174): libgfortran-11.4.1-3.el9.x86_64.rpm 810 kB/s | 803 kB 00:00 2024-03-28T05:11:00.302 INFO:teuthology.orchestra.run.smithi098.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 493 kB/s | 49 kB 00:00 2024-03-28T05:11:00.363 INFO:teuthology.orchestra.run.smithi023.stdout:(89/174): libquadmath-11.4.1-3.el9.x86_64.rpm 469 kB/s | 195 kB 00:00 2024-03-28T05:11:00.394 INFO:teuthology.orchestra.run.smithi098.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 180 kB/s | 16 kB 00:00 2024-03-28T05:11:00.438 INFO:teuthology.orchestra.run.smithi023.stdout:(90/174): elfutils-libelf-devel-0.190-2.el9.x86 286 kB/s | 21 kB 00:00 2024-03-28T05:11:00.488 INFO:teuthology.orchestra.run.smithi023.stdout:(91/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 508 kB/s | 25 kB 00:00 2024-03-28T05:11:00.502 INFO:teuthology.orchestra.run.smithi098.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 2.4 MB/s | 264 kB 00:00 2024-03-28T05:11:00.581 INFO:teuthology.orchestra.run.smithi113.stdout:(10/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 4.6 MB/s | 9.9 MB 00:02 2024-03-28T05:11:00.664 INFO:teuthology.orchestra.run.smithi023.stdout:(92/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 24 MB/s | 4.1 MB 00:00 2024-03-28T05:11:00.664 INFO:teuthology.orchestra.run.smithi113.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 2.6 MB/s | 220 kB 00:00 2024-03-28T05:11:00.722 INFO:teuthology.orchestra.run.smithi023.stdout:(93/174): postgresql-13.14-1.el9.x86_64.rpm 27 MB/s | 1.6 MB 00:00 2024-03-28T05:11:00.731 INFO:teuthology.orchestra.run.smithi113.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 405 kB/s | 27 kB 00:00 2024-03-28T05:11:00.732 INFO:teuthology.orchestra.run.smithi098.stdout:(36/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 3.2 MB/s | 748 kB 00:00 2024-03-28T05:11:00.773 INFO:teuthology.orchestra.run.smithi023.stdout:(94/174): postgresql-contrib-13.14-1.el9.x86_64 18 MB/s | 911 kB 00:00 2024-03-28T05:11:00.790 INFO:teuthology.orchestra.run.smithi023.stdout:(95/174): python3-requests-2.25.1-8.el9.noarch. 249 kB/s | 125 kB 00:00 2024-03-28T05:11:00.806 INFO:teuthology.orchestra.run.smithi113.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 418 kB/s | 31 kB 00:00 2024-03-28T05:11:00.815 INFO:teuthology.orchestra.run.smithi023.stdout:(96/174): postgresql-private-libs-13.14-1.el9.x 3.1 MB/s | 133 kB 00:00 2024-03-28T05:11:00.816 INFO:teuthology.orchestra.run.smithi098.stdout:(37/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 219 kB/s | 18 kB 00:00 2024-03-28T05:11:00.857 INFO:teuthology.orchestra.run.smithi023.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 5.8 MB/s | 245 kB 00:00 2024-03-28T05:11:00.899 INFO:teuthology.orchestra.run.smithi023.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 5.8 MB/s | 248 kB 00:00 2024-03-28T05:11:00.941 INFO:teuthology.orchestra.run.smithi023.stdout:(99/174): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 221 kB 00:00 2024-03-28T05:11:00.967 INFO:teuthology.orchestra.run.smithi098.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 5.2 MB/s | 662 kB 00:00 2024-03-28T05:11:00.983 INFO:teuthology.orchestra.run.smithi023.stdout:(100/174): qatlib-service-24.02.0-1.el9.x86_64. 859 kB/s | 36 kB 00:00 2024-03-28T05:11:00.992 INFO:teuthology.orchestra.run.smithi098.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 18 MB/s | 920 kB 00:00 2024-03-28T05:11:01.024 INFO:teuthology.orchestra.run.smithi023.stdout:(101/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-03-28T05:11:01.026 INFO:teuthology.orchestra.run.smithi098.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-28T05:11:01.032 INFO:teuthology.orchestra.run.smithi113.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 3.2 MB/s | 746 kB 00:00 2024-03-28T05:11:01.076 INFO:teuthology.orchestra.run.smithi098.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-28T05:11:01.099 INFO:teuthology.orchestra.run.smithi113.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 2.4 MB/s | 163 kB 00:00 2024-03-28T05:11:01.100 INFO:teuthology.orchestra.run.smithi023.stdout:(102/174): postgresql-server-13.14-1.el9.x86_64 19 MB/s | 5.8 MB 00:00 2024-03-28T05:11:01.109 INFO:teuthology.orchestra.run.smithi098.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 956 kB/s | 32 kB 00:00 2024-03-28T05:11:01.125 INFO:teuthology.orchestra.run.smithi023.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 934 kB/s | 94 kB 00:00 2024-03-28T05:11:01.174 INFO:teuthology.orchestra.run.smithi113.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.7 MB/s | 127 kB 00:00 2024-03-28T05:11:01.202 INFO:teuthology.orchestra.run.smithi098.stdout:(43/174): ceph-mgr-diskprediction-local-19.0.0- 5.3 MB/s | 7.4 MB 00:01 2024-03-28T05:11:01.209 INFO:teuthology.orchestra.run.smithi023.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 206 kB/s | 22 kB 00:00 2024-03-28T05:11:01.260 INFO:teuthology.orchestra.run.smithi098.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 20 MB/s | 3.0 MB 00:00 2024-03-28T05:11:01.286 INFO:teuthology.orchestra.run.smithi098.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 211 kB/s | 17 kB 00:00 2024-03-28T05:11:01.307 INFO:teuthology.orchestra.run.smithi113.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 3.7 MB/s | 501 kB 00:00 2024-03-28T05:11:01.311 INFO:teuthology.orchestra.run.smithi098.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 200 kB/s | 10 kB 00:00 2024-03-28T05:11:01.336 INFO:teuthology.orchestra.run.smithi098.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-28T05:11:01.361 INFO:teuthology.orchestra.run.smithi098.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 885 kB/s | 44 kB 00:00 2024-03-28T05:11:01.386 INFO:teuthology.orchestra.run.smithi098.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-28T05:11:01.412 INFO:teuthology.orchestra.run.smithi098.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-28T05:11:01.417 INFO:teuthology.orchestra.run.smithi023.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.9 MB/s | 863 kB 00:00 2024-03-28T05:11:01.429 INFO:teuthology.orchestra.run.smithi098.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-28T05:11:01.463 INFO:teuthology.orchestra.run.smithi098.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-28T05:11:01.497 INFO:teuthology.orchestra.run.smithi098.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2024-03-28T05:11:01.522 INFO:teuthology.orchestra.run.smithi098.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-03-28T05:11:01.539 INFO:teuthology.orchestra.run.smithi098.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 823 kB/s | 34 kB 00:00 2024-03-28T05:11:01.648 INFO:teuthology.orchestra.run.smithi098.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-28T05:11:01.681 INFO:teuthology.orchestra.run.smithi098.stdout:(57/174): protobuf-3.14.0-13.el9.x86_64.rpm 7.2 MB/s | 1.0 MB 00:00 2024-03-28T05:11:01.707 INFO:teuthology.orchestra.run.smithi098.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-28T05:11:01.824 INFO:teuthology.orchestra.run.smithi098.stdout:(59/174): python3-babel-2.9.1-2.el9.noarch.rpm 41 MB/s | 5.8 MB 00:00 2024-03-28T05:11:01.840 INFO:teuthology.orchestra.run.smithi098.stdout:(60/174): python3-jsonpatch-1.21-16.el9.noarch. 186 kB/s | 24 kB 00:00 2024-03-28T05:11:01.857 INFO:teuthology.orchestra.run.smithi098.stdout:(61/174): python3-jsonpointer-2.0-4.el9.noarch. 544 kB/s | 18 kB 00:00 2024-03-28T05:11:01.883 INFO:teuthology.orchestra.run.smithi098.stdout:(62/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-28T05:11:01.900 INFO:teuthology.orchestra.run.smithi098.stdout:(63/174): python3-markupsafe-1.1.1-12.el9.x86_6 755 kB/s | 32 kB 00:00 2024-03-28T05:11:02.008 INFO:teuthology.orchestra.run.smithi098.stdout:(64/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-28T05:11:02.026 INFO:teuthology.orchestra.run.smithi023.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 671 kB/s | 548 kB 00:00 2024-03-28T05:11:02.034 INFO:teuthology.orchestra.run.smithi098.stdout:(65/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 217 kB 00:00 2024-03-28T05:11:02.067 INFO:teuthology.orchestra.run.smithi098.stdout:(66/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 174 kB 00:00 2024-03-28T05:11:02.093 INFO:teuthology.orchestra.run.smithi098.stdout:(67/174): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 69 kB 00:00 2024-03-28T05:11:02.118 INFO:teuthology.orchestra.run.smithi023.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 442 kB/s | 309 kB 00:00 2024-03-28T05:11:02.126 INFO:teuthology.orchestra.run.smithi098.stdout:(68/174): python3-prettytable-0.7.2-27.el9.noar 706 kB/s | 41 kB 00:00 2024-03-28T05:11:02.151 INFO:teuthology.orchestra.run.smithi098.stdout:(69/174): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 238 kB 00:00 2024-03-28T05:11:02.160 INFO:teuthology.orchestra.run.smithi023.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 146 kB/s | 19 kB 00:00 2024-03-28T05:11:02.185 INFO:teuthology.orchestra.run.smithi098.stdout:(70/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-28T05:11:02.208 INFO:teuthology.orchestra.run.smithi113.stdout:(18/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 5.7 MB/s | 5.1 MB 00:00 2024-03-28T05:11:02.210 INFO:teuthology.orchestra.run.smithi098.stdout:(71/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.5 MB/s | 211 kB 00:00 2024-03-28T05:11:02.244 INFO:teuthology.orchestra.run.smithi098.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 803 kB/s | 47 kB 00:00 2024-03-28T05:11:02.258 INFO:teuthology.orchestra.run.smithi113.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 905 kB/s | 45 kB 00:00 2024-03-28T05:11:02.269 INFO:teuthology.orchestra.run.smithi098.stdout:(73/174): python3-requests-oauthlib-1.3.0-12.el 728 kB/s | 43 kB 00:00 2024-03-28T05:11:02.293 INFO:teuthology.orchestra.run.smithi023.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 195 kB/s | 26 kB 00:00 2024-03-28T05:11:02.308 INFO:teuthology.orchestra.run.smithi113.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 2.6 MB/s | 131 kB 00:00 2024-03-28T05:11:02.367 INFO:teuthology.orchestra.run.smithi113.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 2.7 MB/s | 162 kB 00:00 2024-03-28T05:11:02.435 INFO:teuthology.orchestra.run.smithi023.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 343 kB/s | 49 kB 00:00 2024-03-28T05:11:02.475 INFO:teuthology.orchestra.run.smithi113.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 2.9 MB/s | 321 kB 00:00 2024-03-28T05:11:02.503 INFO:teuthology.orchestra.run.smithi098.stdout:(74/174): python3-toml-0.10.2-6.el9.noarch.rpm 165 kB/s | 39 kB 00:00 2024-03-28T05:11:02.567 INFO:teuthology.orchestra.run.smithi113.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 3.2 MB/s | 298 kB 00:00 2024-03-28T05:11:02.577 INFO:teuthology.orchestra.run.smithi023.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 476 kB/s | 67 kB 00:00 2024-03-28T05:11:02.626 INFO:teuthology.orchestra.run.smithi113.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.7 MB/s | 99 kB 00:00 2024-03-28T05:11:02.676 INFO:teuthology.orchestra.run.smithi113.stdout:(25/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.7 MB/s | 85 kB 00:00 2024-03-28T05:11:02.727 INFO:teuthology.orchestra.run.smithi023.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-28T05:11:02.754 INFO:teuthology.orchestra.run.smithi098.stdout:(75/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-03-28T05:11:02.779 INFO:teuthology.orchestra.run.smithi098.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 149 kB/s | 41 kB 00:00 2024-03-28T05:11:02.804 INFO:teuthology.orchestra.run.smithi098.stdout:(77/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-28T05:11:02.829 INFO:teuthology.orchestra.run.smithi098.stdout:(78/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-28T05:11:02.855 INFO:teuthology.orchestra.run.smithi098.stdout:(79/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-28T05:11:02.880 INFO:teuthology.orchestra.run.smithi098.stdout:(80/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 874 kB/s | 44 kB 00:00 2024-03-28T05:11:02.914 INFO:teuthology.orchestra.run.smithi098.stdout:(81/174): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2024-03-28T05:11:02.948 INFO:teuthology.orchestra.run.smithi098.stdout:(82/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-28T05:11:02.969 INFO:teuthology.orchestra.run.smithi023.stdout:(113/174): libarrow-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 4.4 MB 00:00 2024-03-28T05:11:02.998 INFO:teuthology.orchestra.run.smithi098.stdout:(83/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-28T05:11:03.019 INFO:teuthology.orchestra.run.smithi023.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.8 MB/s | 838 kB 00:00 2024-03-28T05:11:03.023 INFO:teuthology.orchestra.run.smithi098.stdout:(84/174): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-28T05:11:03.054 INFO:teuthology.orchestra.run.smithi098.stdout:(85/174): python3-pycparser-2.20-6.el9.noarch.r 2.2 MB/s | 124 kB 00:00 2024-03-28T05:11:03.079 INFO:teuthology.orchestra.run.smithi098.stdout:(86/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.3 MB/s | 187 kB 00:00 2024-03-28T05:11:03.112 INFO:teuthology.orchestra.run.smithi023.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 3.8 MB/s | 548 kB 00:00 2024-03-28T05:11:03.145 INFO:teuthology.orchestra.run.smithi023.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 347 kB/s | 43 kB 00:00 2024-03-28T05:11:03.218 INFO:teuthology.orchestra.run.smithi113.stdout:(26/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 5.7 MB/s | 3.1 MB 00:00 2024-03-28T05:11:03.229 INFO:teuthology.orchestra.run.smithi023.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 275 kB/s | 32 kB 00:00 2024-03-28T05:11:03.238 INFO:teuthology.orchestra.run.smithi098.stdout:(87/174): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-28T05:11:03.262 INFO:teuthology.orchestra.run.smithi023.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 128 kB/s | 15 kB 00:00 2024-03-28T05:11:03.269 INFO:teuthology.orchestra.run.smithi113.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 3.3 MB/s | 170 kB 00:00 2024-03-28T05:11:03.288 INFO:teuthology.orchestra.run.smithi098.stdout:(88/174): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-03-28T05:11:03.311 INFO:teuthology.orchestra.run.smithi113.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 818 kB/s | 34 kB 00:00 2024-03-28T05:11:03.330 INFO:teuthology.orchestra.run.smithi098.stdout:(89/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.1 MB/s | 803 kB 00:00 2024-03-28T05:11:03.346 INFO:teuthology.orchestra.run.smithi023.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-03-28T05:11:03.356 INFO:teuthology.orchestra.run.smithi098.stdout:(90/174): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-03-28T05:11:03.361 INFO:teuthology.orchestra.run.smithi113.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.7 MB/s | 139 kB 00:00 2024-03-28T05:11:03.405 INFO:teuthology.orchestra.run.smithi023.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 2.5 MB/s | 358 kB 00:00 2024-03-28T05:11:03.480 INFO:teuthology.orchestra.run.smithi023.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-03-28T05:11:03.530 INFO:teuthology.orchestra.run.smithi023.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 282 kB/s | 35 kB 00:00 2024-03-28T05:11:03.605 INFO:teuthology.orchestra.run.smithi023.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 1.6 MB/s | 203 kB 00:00 2024-03-28T05:11:03.739 INFO:teuthology.orchestra.run.smithi023.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-03-28T05:11:03.823 INFO:teuthology.orchestra.run.smithi023.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 7.0 MB/s | 2.0 MB 00:00 2024-03-28T05:11:03.853 INFO:teuthology.orchestra.run.smithi113.stdout:(30/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 9.3 MB/s | 48 MB 00:05 2024-03-28T05:11:03.856 INFO:teuthology.orchestra.run.smithi023.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 91 kB/s | 11 kB 00:00 2024-03-28T05:11:03.895 INFO:teuthology.orchestra.run.smithi113.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 3.4 MB/s | 1.8 MB 00:00 2024-03-28T05:11:03.932 INFO:teuthology.orchestra.run.smithi023.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 162 kB/s | 18 kB 00:00 2024-03-28T05:11:03.962 INFO:teuthology.orchestra.run.smithi113.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 3.6 MB/s | 248 kB 00:00 2024-03-28T05:11:03.965 INFO:teuthology.orchestra.run.smithi023.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 213 kB/s | 23 kB 00:00 2024-03-28T05:11:04.020 INFO:teuthology.orchestra.run.smithi113.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 848 kB/s | 49 kB 00:00 2024-03-28T05:11:04.032 INFO:teuthology.orchestra.run.smithi023.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2024-03-28T05:11:04.066 INFO:teuthology.orchestra.run.smithi023.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-03-28T05:11:04.079 INFO:teuthology.orchestra.run.smithi113.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 282 kB/s | 16 kB 00:00 2024-03-28T05:11:04.179 INFO:teuthology.orchestra.run.smithi113.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 2.6 MB/s | 264 kB 00:00 2024-03-28T05:11:04.189 INFO:teuthology.orchestra.run.smithi098.stdout:(91/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 6.8 MB/s | 48 MB 00:07 2024-03-28T05:11:04.191 INFO:teuthology.orchestra.run.smithi023.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 6.4 MB/s | 1.0 MB 00:00 2024-03-28T05:11:04.208 INFO:teuthology.orchestra.run.smithi023.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 326 kB/s | 46 kB 00:00 2024-03-28T05:11:04.313 INFO:teuthology.orchestra.run.smithi113.stdout:(36/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 5.5 MB/s | 748 kB 00:00 2024-03-28T05:11:04.316 INFO:teuthology.orchestra.run.smithi023.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 630 kB/s | 79 kB 00:00 2024-03-28T05:11:04.333 INFO:teuthology.orchestra.run.smithi023.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 461 kB/s | 58 kB 00:00 2024-03-28T05:11:04.372 INFO:teuthology.orchestra.run.smithi113.stdout:(37/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 314 kB/s | 18 kB 00:00 2024-03-28T05:11:04.381 INFO:teuthology.orchestra.run.smithi098.stdout:(92/174): elfutils-libelf-devel-0.190-2.el9.x86 20 kB/s | 21 kB 00:01 2024-03-28T05:11:04.398 INFO:teuthology.orchestra.run.smithi098.stdout:(93/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 24 kB/s | 25 kB 00:01 2024-03-28T05:11:04.434 INFO:teuthology.orchestra.run.smithi023.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-03-28T05:11:04.451 INFO:teuthology.orchestra.run.smithi023.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 141 kB/s | 16 kB 00:00 2024-03-28T05:11:04.543 INFO:teuthology.orchestra.run.smithi023.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 830 kB/s | 90 kB 00:00 2024-03-28T05:11:04.555 INFO:teuthology.orchestra.run.smithi113.stdout:(38/174): ceph-mgr-diskprediction-local-19.0.0- 11 MB/s | 7.4 MB 00:00 2024-03-28T05:11:04.560 INFO:teuthology.orchestra.run.smithi023.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 260 kB/s | 28 kB 00:00 2024-03-28T05:11:04.643 INFO:teuthology.orchestra.run.smithi023.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 307 kB/s | 31 kB 00:00 2024-03-28T05:11:04.668 INFO:teuthology.orchestra.run.smithi023.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-03-28T05:11:04.714 INFO:teuthology.orchestra.run.smithi113.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 1.9 MB/s | 662 kB 00:00 2024-03-28T05:11:04.744 INFO:teuthology.orchestra.run.smithi023.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 590 kB/s | 59 kB 00:00 2024-03-28T05:11:04.769 INFO:teuthology.orchestra.run.smithi023.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 357 kB/s | 36 kB 00:00 2024-03-28T05:11:04.781 INFO:teuthology.orchestra.run.smithi113.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.0 MB/s | 920 kB 00:00 2024-03-28T05:11:04.807 INFO:teuthology.orchestra.run.smithi113.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 106 kB 00:00 2024-03-28T05:11:04.832 INFO:teuthology.orchestra.run.smithi113.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-28T05:11:04.844 INFO:teuthology.orchestra.run.smithi023.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 509 kB/s | 51 kB 00:00 2024-03-28T05:11:04.857 INFO:teuthology.orchestra.run.smithi113.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 640 kB/s | 32 kB 00:00 2024-03-28T05:11:04.878 INFO:teuthology.orchestra.run.smithi023.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-28T05:11:04.891 INFO:teuthology.orchestra.run.smithi113.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 512 kB/s | 17 kB 00:00 2024-03-28T05:11:04.953 INFO:teuthology.orchestra.run.smithi023.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 826 kB/s | 90 kB 00:00 2024-03-28T05:11:04.995 INFO:teuthology.orchestra.run.smithi023.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-03-28T05:11:05.058 INFO:teuthology.orchestra.run.smithi113.stdout:(45/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 2.5 MB/s | 17 MB 00:06 2024-03-28T05:11:05.079 INFO:teuthology.orchestra.run.smithi023.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 159 kB/s | 20 kB 00:00 2024-03-28T05:11:05.083 INFO:teuthology.orchestra.run.smithi113.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 52 kB/s | 10 kB 00:00 2024-03-28T05:11:05.112 INFO:teuthology.orchestra.run.smithi023.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-03-28T05:11:05.150 INFO:teuthology.orchestra.run.smithi113.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 159 kB 00:00 2024-03-28T05:11:05.209 INFO:teuthology.orchestra.run.smithi113.stdout:(48/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.9 MB/s | 3.0 MB 00:00 2024-03-28T05:11:05.234 INFO:teuthology.orchestra.run.smithi113.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 296 kB/s | 44 kB 00:00 2024-03-28T05:11:05.255 INFO:teuthology.orchestra.run.smithi023.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-03-28T05:11:05.259 INFO:teuthology.orchestra.run.smithi113.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 661 kB 00:00 2024-03-28T05:11:05.284 INFO:teuthology.orchestra.run.smithi113.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 952 kB/s | 46 kB 00:00 2024-03-28T05:11:05.310 INFO:teuthology.orchestra.run.smithi113.stdout:(52/174): libtool-2.4.6-45.el9.x86_64.rpm 7.6 MB/s | 578 kB 00:00 2024-03-28T05:11:05.335 INFO:teuthology.orchestra.run.smithi113.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-28T05:11:05.360 INFO:teuthology.orchestra.run.smithi113.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.5 MB/s | 187 kB 00:00 2024-03-28T05:11:05.385 INFO:teuthology.orchestra.run.smithi113.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 689 kB/s | 34 kB 00:00 2024-03-28T05:11:05.418 INFO:teuthology.orchestra.run.smithi113.stdout:(56/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.6 MB/s | 217 kB 00:00 2024-03-28T05:11:05.588 INFO:teuthology.orchestra.run.smithi023.stdout:(150/174): librados2-19.0.0-2145.g3be9a5c9.el9. 7.1 MB/s | 3.4 MB 00:00 2024-03-28T05:11:05.610 INFO:teuthology.orchestra.run.smithi113.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 19 MB/s | 4.7 MB 00:00 2024-03-28T05:11:05.639 INFO:teuthology.orchestra.run.smithi023.stdout:(151/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 8.2 MB/s | 3.1 MB 00:00 2024-03-28T05:11:05.644 INFO:teuthology.orchestra.run.smithi113.stdout:(58/174): protobuf-3.14.0-13.el9.x86_64.rpm 3.9 MB/s | 1.0 MB 00:00 2024-03-28T05:11:05.664 INFO:teuthology.orchestra.run.smithi023.stdout:(152/174): xfsdump-3.1.12-4.el9.x86_64.rpm 4.2 MB/s | 321 kB 00:00 2024-03-28T05:11:05.669 INFO:teuthology.orchestra.run.smithi113.stdout:(59/174): python3-attrs-20.3.0-7.el9.noarch.rpm 311 kB/s | 78 kB 00:00 2024-03-28T05:11:05.811 INFO:teuthology.orchestra.run.smithi113.stdout:(60/174): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 5.8 MB 00:00 2024-03-28T05:11:05.828 INFO:teuthology.orchestra.run.smithi113.stdout:(61/174): python3-jsonpatch-1.21-16.el9.noarch. 157 kB/s | 24 kB 00:00 2024-03-28T05:11:05.845 INFO:teuthology.orchestra.run.smithi113.stdout:(62/174): python3-jsonpointer-2.0-4.el9.noarch. 105 kB/s | 18 kB 00:00 2024-03-28T05:11:05.870 INFO:teuthology.orchestra.run.smithi113.stdout:(63/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-28T05:11:05.987 INFO:teuthology.orchestra.run.smithi113.stdout:(64/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-28T05:11:06.013 INFO:teuthology.orchestra.run.smithi113.stdout:(65/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-03-28T05:11:06.030 INFO:teuthology.orchestra.run.smithi113.stdout:(66/174): python3-markupsafe-1.1.1-12.el9.x86_6 171 kB/s | 32 kB 00:00 2024-03-28T05:11:06.063 INFO:teuthology.orchestra.run.smithi113.stdout:(67/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.3 MB/s | 174 kB 00:00 2024-03-28T05:11:06.105 INFO:teuthology.orchestra.run.smithi113.stdout:(68/174): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-03-28T05:11:06.107 INFO:teuthology.orchestra.run.smithi098.stdout:(94/174): postgresql-contrib-13.14-1.el9.x86_64 533 kB/s | 911 kB 00:01 2024-03-28T05:11:06.130 INFO:teuthology.orchestra.run.smithi113.stdout:(69/174): python3-packaging-20.9-5.el9.noarch.r 586 kB/s | 69 kB 00:00 2024-03-28T05:11:06.147 INFO:teuthology.orchestra.run.smithi113.stdout:(70/174): python3-prettytable-0.7.2-27.el9.noar 351 kB/s | 41 kB 00:00 2024-03-28T05:11:06.173 INFO:teuthology.orchestra.run.smithi113.stdout:(71/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-28T05:11:06.189 INFO:teuthology.orchestra.run.smithi113.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-28T05:11:06.215 INFO:teuthology.orchestra.run.smithi113.stdout:(73/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2024-03-28T05:11:06.231 INFO:teuthology.orchestra.run.smithi113.stdout:(74/174): python3-requests-oauthlib-1.3.0-12.el 732 kB/s | 43 kB 00:00 2024-03-28T05:11:06.357 INFO:teuthology.orchestra.run.smithi113.stdout:(75/174): python3-toml-0.10.2-6.el9.noarch.rpm 316 kB/s | 39 kB 00:00 2024-03-28T05:11:06.374 INFO:teuthology.orchestra.run.smithi113.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 295 kB/s | 41 kB 00:00 2024-03-28T05:11:06.431 INFO:teuthology.orchestra.run.smithi023.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 2.4 MB/s | 1.8 MB 00:00 2024-03-28T05:11:06.556 INFO:teuthology.orchestra.run.smithi023.stdout:(154/174): NetworkManager-1.47.2-1.el9.x86_64.r 2.5 MB/s | 2.3 MB 00:00 2024-03-28T05:11:06.566 INFO:teuthology.orchestra.run.smithi113.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-28T05:11:06.591 INFO:teuthology.orchestra.run.smithi113.stdout:(78/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 300 kB 00:00 2024-03-28T05:11:06.616 INFO:teuthology.orchestra.run.smithi113.stdout:(79/174): uuid-1.6.2-55.el9.x86_64.rpm 235 kB/s | 56 kB 00:00 2024-03-28T05:11:06.642 INFO:teuthology.orchestra.run.smithi113.stdout:(80/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 839 kB/s | 63 kB 00:00 2024-03-28T05:11:06.695 INFO:teuthology.orchestra.run.smithi113.stdout:(81/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 564 kB/s | 44 kB 00:00 2024-03-28T05:11:06.717 INFO:teuthology.orchestra.run.smithi113.stdout:(82/174): fuse-2.9.9-15.el9.x86_64.rpm 784 kB/s | 79 kB 00:00 2024-03-28T05:11:06.742 INFO:teuthology.orchestra.run.smithi113.stdout:(83/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 241 kB 00:00 2024-03-28T05:11:06.768 INFO:teuthology.orchestra.run.smithi113.stdout:(84/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-28T05:11:06.801 INFO:teuthology.orchestra.run.smithi113.stdout:(85/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-28T05:11:06.823 INFO:teuthology.orchestra.run.smithi023.stdout:(155/174): NetworkManager-team-1.47.2-1.el9.x86 107 kB/s | 42 kB 00:00 2024-03-28T05:11:06.826 INFO:teuthology.orchestra.run.smithi113.stdout:(86/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-28T05:11:06.852 INFO:teuthology.orchestra.run.smithi113.stdout:(87/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-28T05:11:06.857 INFO:teuthology.orchestra.run.smithi098.stdout:(95/174): postgresql-private-libs-13.14-1.el9.x 178 kB/s | 133 kB 00:00 2024-03-28T05:11:06.982 INFO:teuthology.orchestra.run.smithi098.stdout:(96/174): postgresql-13.14-1.el9.x86_64.rpm 626 kB/s | 1.6 MB 00:02 2024-03-28T05:11:07.069 INFO:teuthology.orchestra.run.smithi113.stdout:(88/174): jq-1.6-16.el9.x86_64.rpm 775 kB/s | 187 kB 00:00 2024-03-28T05:11:07.073 INFO:teuthology.orchestra.run.smithi023.stdout:(156/174): NetworkManager-tui-1.47.2-1.el9.x86_ 480 kB/s | 248 kB 00:00 2024-03-28T05:11:07.094 INFO:teuthology.orchestra.run.smithi113.stdout:(89/174): libquadmath-11.4.1-3.el9.x86_64.rpm 807 kB/s | 195 kB 00:00 2024-03-28T05:11:07.144 INFO:teuthology.orchestra.run.smithi113.stdout:(90/174): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-03-28T05:11:07.165 INFO:teuthology.orchestra.run.smithi023.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 209 kB/s | 71 kB 00:00 2024-03-28T05:11:07.178 INFO:teuthology.orchestra.run.smithi113.stdout:(91/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 803 kB 00:00 2024-03-28T05:11:07.204 INFO:teuthology.orchestra.run.smithi113.stdout:(92/174): elfutils-libelf-devel-0.190-2.el9.x86 197 kB/s | 21 kB 00:00 2024-03-28T05:11:07.237 INFO:teuthology.orchestra.run.smithi113.stdout:(93/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 276 kB/s | 25 kB 00:00 2024-03-28T05:11:07.371 INFO:teuthology.orchestra.run.smithi113.stdout:(94/174): postgresql-13.14-1.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-03-28T05:11:07.382 INFO:teuthology.orchestra.run.smithi023.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 118 kB/s | 36 kB 00:00 2024-03-28T05:11:07.396 INFO:teuthology.orchestra.run.smithi113.stdout:(95/174): postgresql-contrib-13.14-1.el9.x86_64 5.6 MB/s | 911 kB 00:00 2024-03-28T05:11:07.421 INFO:teuthology.orchestra.run.smithi113.stdout:(96/174): postgresql-private-libs-13.14-1.el9.x 2.6 MB/s | 133 kB 00:00 2024-03-28T05:11:07.496 INFO:teuthology.orchestra.run.smithi113.stdout:(97/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 13 MB/s | 4.1 MB 00:00 2024-03-28T05:11:07.522 INFO:teuthology.orchestra.run.smithi113.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-03-28T05:11:07.547 INFO:teuthology.orchestra.run.smithi113.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-03-28T05:11:07.572 INFO:teuthology.orchestra.run.smithi113.stdout:(100/174): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-03-28T05:11:07.582 INFO:teuthology.orchestra.run.smithi023.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 461 kB/s | 192 kB 00:00 2024-03-28T05:11:07.589 INFO:teuthology.orchestra.run.smithi113.stdout:(101/174): qatlib-service-24.02.0-1.el9.x86_64. 860 kB/s | 36 kB 00:00 2024-03-28T05:11:07.615 INFO:teuthology.orchestra.run.smithi113.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-03-28T05:11:07.640 INFO:teuthology.orchestra.run.smithi113.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-28T05:11:07.740 INFO:teuthology.orchestra.run.smithi113.stdout:(104/174): postgresql-server-13.14-1.el9.x86_64 17 MB/s | 5.8 MB 00:00 2024-03-28T05:11:07.849 INFO:teuthology.orchestra.run.smithi023.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 86 kB/s | 23 kB 00:00 2024-03-28T05:11:07.866 INFO:teuthology.orchestra.run.smithi113.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 89 kB/s | 22 kB 00:00 2024-03-28T05:11:07.891 INFO:teuthology.orchestra.run.smithi098.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 270 kB/s | 245 kB 00:00 2024-03-28T05:11:07.891 INFO:teuthology.orchestra.run.smithi023.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 503 kB/s | 256 kB 00:00 2024-03-28T05:11:07.924 INFO:teuthology.orchestra.run.smithi113.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.0 MB/s | 863 kB 00:00 2024-03-28T05:11:08.049 INFO:teuthology.orchestra.run.smithi113.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.7 MB/s | 548 kB 00:00 2024-03-28T05:11:08.075 INFO:teuthology.orchestra.run.smithi113.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 130 kB/s | 19 kB 00:00 2024-03-28T05:11:08.142 INFO:teuthology.orchestra.run.smithi113.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.1 MB/s | 309 kB 00:00 2024-03-28T05:11:08.167 INFO:teuthology.orchestra.run.smithi113.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-03-28T05:11:08.200 INFO:teuthology.orchestra.run.smithi113.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 831 kB/s | 49 kB 00:00 2024-03-28T05:11:08.275 INFO:teuthology.orchestra.run.smithi023.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 256 kB/s | 108 kB 00:00 2024-03-28T05:11:08.276 INFO:teuthology.orchestra.run.smithi113.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-03-28T05:11:08.298 INFO:teuthology.orchestra.run.smithi113.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 514 kB/s | 67 kB 00:00 2024-03-28T05:11:08.317 INFO:teuthology.orchestra.run.smithi023.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 364 kB/s | 155 kB 00:00 2024-03-28T05:11:08.323 INFO:teuthology.orchestra.run.smithi113.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-03-28T05:11:08.357 INFO:teuthology.orchestra.run.smithi113.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 10 MB/s | 838 kB 00:00 2024-03-28T05:11:08.382 INFO:teuthology.orchestra.run.smithi113.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 741 kB/s | 43 kB 00:00 2024-03-28T05:11:08.416 INFO:teuthology.orchestra.run.smithi113.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 551 kB/s | 32 kB 00:00 2024-03-28T05:11:08.441 INFO:teuthology.orchestra.run.smithi113.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 256 kB/s | 15 kB 00:00 2024-03-28T05:11:08.475 INFO:teuthology.orchestra.run.smithi113.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-03-28T05:11:08.500 INFO:teuthology.orchestra.run.smithi113.stdout:(120/174): python3-asyncssh-2.13.2-5.el9.noarch 2.7 MB/s | 548 kB 00:00 2024-03-28T05:11:08.534 INFO:teuthology.orchestra.run.smithi113.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-03-28T05:11:08.559 INFO:teuthology.orchestra.run.smithi113.stdout:(122/174): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-03-28T05:11:08.584 INFO:teuthology.orchestra.run.smithi113.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 420 kB/s | 35 kB 00:00 2024-03-28T05:11:08.610 INFO:teuthology.orchestra.run.smithi113.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-03-28T05:11:08.643 INFO:teuthology.orchestra.run.smithi113.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-03-28T05:11:08.650 INFO:teuthology.orchestra.run.smithi023.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 192 kB/s | 64 kB 00:00 2024-03-28T05:11:08.668 INFO:teuthology.orchestra.run.smithi113.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 183 kB/s | 11 kB 00:00 2024-03-28T05:11:08.692 INFO:teuthology.orchestra.run.smithi023.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 325 kB/s | 136 kB 00:00 2024-03-28T05:11:08.694 INFO:teuthology.orchestra.run.smithi113.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-03-28T05:11:08.736 INFO:teuthology.orchestra.run.smithi113.stdout:(128/174): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-03-28T05:11:08.752 INFO:teuthology.orchestra.run.smithi113.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 276 kB/s | 23 kB 00:00 2024-03-28T05:11:08.769 INFO:teuthology.orchestra.run.smithi113.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-03-28T05:11:08.795 INFO:teuthology.orchestra.run.smithi113.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 334 kB/s | 20 kB 00:00 2024-03-28T05:11:08.800 INFO:teuthology.orchestra.run.smithi098.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 273 kB/s | 248 kB 00:00 2024-03-28T05:11:08.828 INFO:teuthology.orchestra.run.smithi113.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 14 MB/s | 1.0 MB 00:00 2024-03-28T05:11:08.854 INFO:teuthology.orchestra.run.smithi113.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 551 kB/s | 46 kB 00:00 2024-03-28T05:11:08.879 INFO:teuthology.orchestra.run.smithi113.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 942 kB/s | 79 kB 00:00 2024-03-28T05:11:08.904 INFO:teuthology.orchestra.run.smithi113.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 767 kB/s | 58 kB 00:00 2024-03-28T05:11:08.926 INFO:teuthology.orchestra.run.smithi023.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 104 kB/s | 29 kB 00:00 2024-03-28T05:11:08.929 INFO:teuthology.orchestra.run.smithi113.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-28T05:11:08.963 INFO:teuthology.orchestra.run.smithi113.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 195 kB/s | 16 kB 00:00 2024-03-28T05:11:08.988 INFO:teuthology.orchestra.run.smithi113.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 479 kB/s | 28 kB 00:00 2024-03-28T05:11:09.014 INFO:teuthology.orchestra.run.smithi113.stdout:(139/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 822 kB/s | 90 kB 00:00 2024-03-28T05:11:09.039 INFO:teuthology.orchestra.run.smithi113.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 407 kB/s | 31 kB 00:00 2024-03-28T05:11:09.064 INFO:teuthology.orchestra.run.smithi113.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-28T05:11:09.090 INFO:teuthology.orchestra.run.smithi113.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 782 kB/s | 59 kB 00:00 2024-03-28T05:11:09.115 INFO:teuthology.orchestra.run.smithi113.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 473 kB/s | 36 kB 00:00 2024-03-28T05:11:09.148 INFO:teuthology.orchestra.run.smithi113.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 608 kB/s | 51 kB 00:00 2024-03-28T05:11:09.174 INFO:teuthology.orchestra.run.smithi113.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-03-28T05:11:09.199 INFO:teuthology.orchestra.run.smithi113.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-28T05:11:09.224 INFO:teuthology.orchestra.run.smithi113.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-03-28T05:11:09.250 INFO:teuthology.orchestra.run.smithi113.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 264 kB/s | 20 kB 00:00 2024-03-28T05:11:09.275 INFO:teuthology.orchestra.run.smithi113.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-28T05:11:09.375 INFO:teuthology.orchestra.run.smithi113.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-28T05:11:09.485 INFO:teuthology.orchestra.run.smithi023.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 1.5 MB/s | 1.2 MB 00:00 2024-03-28T05:11:09.492 INFO:teuthology.orchestra.run.smithi113.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.7 MB/s | 321 kB 00:00 2024-03-28T05:11:09.700 INFO:teuthology.orchestra.run.smithi098.stdout:(99/174): qatlib-24.02.0-1.el9.x86_64.rpm 245 kB/s | 221 kB 00:00 2024-03-28T05:11:09.734 INFO:teuthology.orchestra.run.smithi113.stdout:(152/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 6.9 MB/s | 3.1 MB 00:00 2024-03-28T05:11:09.843 INFO:teuthology.orchestra.run.smithi113.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 6.6 MB/s | 2.3 MB 00:00 2024-03-28T05:11:09.868 INFO:teuthology.orchestra.run.smithi023.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 2.3 MB/s | 2.2 MB 00:00 2024-03-28T05:11:09.901 INFO:teuthology.orchestra.run.smithi113.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 723 kB/s | 42 kB 00:00 2024-03-28T05:11:09.958 INFO:teuthology.orchestra.run.smithi098.stdout:(100/174): openssl-devel-3.0.7-27.el9.x86_64.rp 733 kB/s | 4.1 MB 00:05 2024-03-28T05:11:09.985 INFO:teuthology.orchestra.run.smithi113.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.9 MB/s | 248 kB 00:00 2024-03-28T05:11:10.052 INFO:teuthology.orchestra.run.smithi113.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2024-03-28T05:11:10.110 INFO:teuthology.orchestra.run.smithi113.stdout:(157/174): librados2-19.0.0-2145.g3be9a5c9.el9. 3.9 MB/s | 3.4 MB 00:00 2024-03-28T05:11:10.135 INFO:teuthology.orchestra.run.smithi113.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 436 kB/s | 36 kB 00:00 2024-03-28T05:11:10.177 INFO:teuthology.orchestra.run.smithi113.stdout:(159/174): NetworkManager-libnm-1.47.2-1.el9.x8 4.1 MB/s | 1.8 MB 00:00 2024-03-28T05:11:10.236 INFO:teuthology.orchestra.run.smithi113.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.5 MB/s | 256 kB 00:00 2024-03-28T05:11:10.261 INFO:teuthology.orchestra.run.smithi113.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 275 kB/s | 23 kB 00:00 2024-03-28T05:11:10.320 INFO:teuthology.orchestra.run.smithi113.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 108 kB 00:00 2024-03-28T05:11:10.345 INFO:teuthology.orchestra.run.smithi113.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 155 kB 00:00 2024-03-28T05:11:10.375 INFO:teuthology.orchestra.run.smithi098.stdout:(101/174): qatlib-service-24.02.0-1.el9.x86_64. 53 kB/s | 36 kB 00:00 2024-03-28T05:11:10.404 INFO:teuthology.orchestra.run.smithi113.stdout:(164/174): elfutils-libelf-0.190-2.el9.x86_64.r 656 kB/s | 192 kB 00:00 2024-03-28T05:11:10.429 INFO:teuthology.orchestra.run.smithi113.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 765 kB/s | 64 kB 00:00 2024-03-28T05:11:10.452 INFO:teuthology.orchestra.run.smithi023.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 798 kB/s | 465 kB 00:00 2024-03-28T05:11:10.455 INFO:teuthology.orchestra.run.smithi113.stdout:(166/174): libmount-2.37.4-18.el9.x86_64.rpm 1.0 MB/s | 136 kB 00:00 2024-03-28T05:11:10.472 INFO:teuthology.orchestra.run.smithi113.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 427 kB/s | 29 kB 00:00 2024-03-28T05:11:10.494 INFO:teuthology.orchestra.run.smithi023.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 2.3 MB 00:01 2024-03-28T05:11:10.514 INFO:teuthology.orchestra.run.smithi113.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 14 MB/s | 1.2 MB 00:00 2024-03-28T05:11:10.664 INFO:teuthology.orchestra.run.smithi113.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 10 MB/s | 2.2 MB 00:00 2024-03-28T05:11:10.675 INFO:teuthology.orchestra.run.smithi098.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 69 kB/s | 47 kB 00:00 2024-03-28T05:11:10.689 INFO:teuthology.orchestra.run.smithi113.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 2.6 MB/s | 465 kB 00:00 2024-03-28T05:11:10.711 INFO:teuthology.orchestra.run.smithi023.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 120 kB/s | 31 kB 00:00 2024-03-28T05:11:10.748 INFO:teuthology.orchestra.run.smithi113.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 373 kB/s | 31 kB 00:00 2024-03-28T05:11:10.781 INFO:teuthology.orchestra.run.smithi113.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 12 MB/s | 1.1 MB 00:00 2024-03-28T05:11:10.794 INFO:teuthology.orchestra.run.smithi023.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 893 kB/s | 74 kB 00:00 2024-03-28T05:11:10.801 INFO:teuthology.orchestra.run.smithi098.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 226 kB/s | 22 kB 00:00 2024-03-28T05:11:10.832 INFO:teuthology.orchestra.run.smithi113.stdout:(173/174): util-linux-2.37.4-18.el9.x86_64.rpm 6.4 MB/s | 2.3 MB 00:00 2024-03-28T05:11:10.857 INFO:teuthology.orchestra.run.smithi113.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 684 kB/s | 74 kB 00:00 2024-03-28T05:11:10.861 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-03-28T05:11:10.861 INFO:teuthology.orchestra.run.smithi113.stdout:Total 14 MB/s | 225 MB 00:16 2024-03-28T05:11:11.010 INFO:teuthology.orchestra.run.smithi098.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 4.1 MB/s | 863 kB 00:00 2024-03-28T05:11:11.201 INFO:teuthology.orchestra.run.smithi098.stdout:(105/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 113 kB/s | 94 kB 00:00 2024-03-28T05:11:11.336 INFO:teuthology.orchestra.run.smithi023.stdout:(173/174): ceph-test-19.0.0-2145.g3be9a5c9.el9. 2.6 MB/s | 48 MB 00:18 2024-03-28T05:11:11.444 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-03-28T05:11:11.686 INFO:teuthology.orchestra.run.smithi023.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 922 kB/s | 1.1 MB 00:01 2024-03-28T05:11:11.691 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2024-03-28T05:11:11.691 INFO:teuthology.orchestra.run.smithi023.stdout:Total 11 MB/s | 225 MB 00:21 2024-03-28T05:11:11.714 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-03-28T05:11:11.714 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-03-28T05:11:11.743 INFO:teuthology.orchestra.run.smithi098.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 747 kB/s | 548 kB 00:00 2024-03-28T05:11:11.852 INFO:teuthology.orchestra.run.smithi098.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 179 kB/s | 19 kB 00:00 2024-03-28T05:11:11.877 INFO:teuthology.orchestra.run.smithi098.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 458 kB/s | 309 kB 00:00 2024-03-28T05:11:11.986 INFO:teuthology.orchestra.run.smithi098.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-03-28T05:11:12.102 INFO:teuthology.orchestra.run.smithi098.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 417 kB/s | 49 kB 00:00 2024-03-28T05:11:12.220 INFO:teuthology.orchestra.run.smithi098.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 578 kB/s | 67 kB 00:00 2024-03-28T05:11:12.278 INFO:teuthology.orchestra.run.smithi098.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-03-28T05:11:12.289 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2024-03-28T05:11:12.354 INFO:teuthology.orchestra.run.smithi098.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-03-28T05:11:12.396 INFO:teuthology.orchestra.run.smithi098.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-03-28T05:11:12.504 INFO:teuthology.orchestra.run.smithi098.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 401 kB/s | 43 kB 00:00 2024-03-28T05:11:12.588 INFO:teuthology.orchestra.run.smithi098.stdout:(116/174): python3-asyncssh-2.13.2-5.el9.noarch 2.3 MB/s | 548 kB 00:00 2024-03-28T05:11:12.588 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2024-03-28T05:11:12.588 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2024-03-28T05:11:12.605 INFO:teuthology.orchestra.run.smithi098.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 321 kB/s | 32 kB 00:00 2024-03-28T05:11:12.689 INFO:teuthology.orchestra.run.smithi098.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-03-28T05:11:12.714 INFO:teuthology.orchestra.run.smithi098.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-03-28T05:11:12.831 INFO:teuthology.orchestra.run.smithi098.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 2.5 MB/s | 358 kB 00:00 2024-03-28T05:11:12.856 INFO:teuthology.orchestra.run.smithi098.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.1 MB/s | 303 kB 00:00 2024-03-28T05:11:12.940 INFO:teuthology.orchestra.run.smithi098.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 325 kB/s | 35 kB 00:00 2024-03-28T05:11:12.982 INFO:teuthology.orchestra.run.smithi098.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 1.6 MB/s | 203 kB 00:00 2024-03-28T05:11:13.099 INFO:teuthology.orchestra.run.smithi098.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-03-28T05:11:13.207 INFO:teuthology.orchestra.run.smithi098.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 99 kB/s | 11 kB 00:00 2024-03-28T05:11:13.258 INFO:teuthology.orchestra.run.smithi098.stdout:(126/174): python3-grpcio-1.46.7-10.el9.x86_64. 6.4 MB/s | 2.0 MB 00:00 2024-03-28T05:11:13.308 INFO:teuthology.orchestra.run.smithi098.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-03-28T05:11:13.358 INFO:teuthology.orchestra.run.smithi098.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 230 kB/s | 23 kB 00:00 2024-03-28T05:11:13.409 INFO:teuthology.orchestra.run.smithi098.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2024-03-28T05:11:13.440 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-03-28T05:11:13.441 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-03-28T05:11:13.459 INFO:teuthology.orchestra.run.smithi098.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-03-28T05:11:13.543 INFO:teuthology.orchestra.run.smithi098.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 7.6 MB/s | 1.0 MB 00:00 2024-03-28T05:11:13.568 INFO:teuthology.orchestra.run.smithi098.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 426 kB/s | 46 kB 00:00 2024-03-28T05:11:13.652 INFO:teuthology.orchestra.run.smithi098.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 727 kB/s | 79 kB 00:00 2024-03-28T05:11:13.677 INFO:teuthology.orchestra.run.smithi098.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 531 kB/s | 58 kB 00:00 2024-03-28T05:11:13.761 INFO:teuthology.orchestra.run.smithi098.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-28T05:11:13.786 INFO:teuthology.orchestra.run.smithi098.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-03-28T05:11:13.869 INFO:teuthology.orchestra.run.smithi098.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 828 kB/s | 90 kB 00:00 2024-03-28T05:11:13.895 INFO:teuthology.orchestra.run.smithi098.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 259 kB/s | 28 kB 00:00 2024-03-28T05:11:13.978 INFO:teuthology.orchestra.run.smithi098.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 284 kB/s | 31 kB 00:00 2024-03-28T05:11:14.004 INFO:teuthology.orchestra.run.smithi098.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-03-28T05:11:14.087 INFO:teuthology.orchestra.run.smithi098.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 543 kB/s | 59 kB 00:00 2024-03-28T05:11:14.112 INFO:teuthology.orchestra.run.smithi098.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 329 kB/s | 36 kB 00:00 2024-03-28T05:11:14.196 INFO:teuthology.orchestra.run.smithi098.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 470 kB/s | 51 kB 00:00 2024-03-28T05:11:14.230 INFO:teuthology.orchestra.run.smithi098.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-03-28T05:11:14.305 INFO:teuthology.orchestra.run.smithi098.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 825 kB/s | 90 kB 00:00 2024-03-28T05:11:14.347 INFO:teuthology.orchestra.run.smithi098.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-03-28T05:11:14.414 INFO:teuthology.orchestra.run.smithi098.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2024-03-28T05:11:14.464 INFO:teuthology.orchestra.run.smithi098.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-03-28T05:11:14.556 INFO:teuthology.orchestra.run.smithi098.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-28T05:11:14.700 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2024-03-28T05:11:14.700 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2024-03-28T05:11:14.865 INFO:teuthology.orchestra.run.smithi098.stdout:(150/174): librados2-19.0.0-2145.g3be9a5c9.el9. 8.5 MB/s | 3.4 MB 00:00 2024-03-28T05:11:14.940 INFO:teuthology.orchestra.run.smithi098.stdout:(151/174): postgresql-server-13.14-1.el9.x86_64 732 kB/s | 5.8 MB 00:08 2024-03-28T05:11:14.974 INFO:teuthology.orchestra.run.smithi098.stdout:(152/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.9 MB/s | 321 kB 00:00 2024-03-28T05:11:15.024 INFO:teuthology.orchestra.run.smithi098.stdout:(153/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 6.7 MB/s | 3.1 MB 00:00 2024-03-28T05:11:15.125 INFO:teuthology.orchestra.run.smithi098.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 421 kB/s | 42 kB 00:00 2024-03-28T05:11:15.200 INFO:teuthology.orchestra.run.smithi098.stdout:(155/174): NetworkManager-1.47.2-1.el9.x86_64.r 8.9 MB/s | 2.3 MB 00:00 2024-03-28T05:11:15.242 INFO:teuthology.orchestra.run.smithi098.stdout:(156/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.1 MB/s | 248 kB 00:00 2024-03-28T05:11:15.275 INFO:teuthology.orchestra.run.smithi098.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 948 kB/s | 71 kB 00:00 2024-03-28T05:11:15.293 INFO:teuthology.orchestra.run.smithi098.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 726 kB/s | 36 kB 00:00 2024-03-28T05:11:15.343 INFO:teuthology.orchestra.run.smithi098.stdout:(159/174): NetworkManager-libnm-1.47.2-1.el9.x8 5.0 MB/s | 1.8 MB 00:00 2024-03-28T05:11:15.368 INFO:teuthology.orchestra.run.smithi098.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.0 MB/s | 192 kB 00:00 2024-03-28T05:11:15.394 INFO:teuthology.orchestra.run.smithi098.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 459 kB/s | 23 kB 00:00 2024-03-28T05:11:15.419 INFO:teuthology.orchestra.run.smithi098.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.0 MB/s | 256 kB 00:00 2024-03-28T05:11:15.444 INFO:teuthology.orchestra.run.smithi098.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 108 kB 00:00 2024-03-28T05:11:15.469 INFO:teuthology.orchestra.run.smithi098.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 155 kB 00:00 2024-03-28T05:11:15.495 INFO:teuthology.orchestra.run.smithi098.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 136 kB 00:00 2024-03-28T05:11:15.520 INFO:teuthology.orchestra.run.smithi098.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 845 kB/s | 64 kB 00:00 2024-03-28T05:11:15.537 INFO:teuthology.orchestra.run.smithi098.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 427 kB/s | 29 kB 00:00 2024-03-28T05:11:15.587 INFO:teuthology.orchestra.run.smithi098.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 18 MB/s | 1.2 MB 00:00 2024-03-28T05:11:15.654 INFO:teuthology.orchestra.run.smithi098.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 16 MB/s | 2.2 MB 00:00 2024-03-28T05:11:15.696 INFO:teuthology.orchestra.run.smithi098.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 14 MB/s | 2.3 MB 00:00 2024-03-28T05:11:15.755 INFO:teuthology.orchestra.run.smithi098.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 310 kB/s | 31 kB 00:00 2024-03-28T05:11:15.838 INFO:teuthology.orchestra.run.smithi098.stdout:(172/174): util-linux-core-2.37.4-18.el9.x86_64 1.8 MB/s | 465 kB 00:00 2024-03-28T05:11:15.872 INFO:teuthology.orchestra.run.smithi098.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 6.1 MB/s | 1.1 MB 00:00 2024-03-28T05:11:16.439 INFO:teuthology.orchestra.run.smithi098.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 109 kB/s | 74 kB 00:00 2024-03-28T05:11:16.445 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-28T05:11:16.446 INFO:teuthology.orchestra.run.smithi098.stdout:Total 9.0 MB/s | 225 MB 00:24 2024-03-28T05:11:17.004 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-28T05:11:17.274 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-28T05:11:17.274 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-28T05:11:19.003 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-28T05:11:19.003 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-28T05:11:20.566 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-03-28T05:11:20.701 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T05:11:20.736 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T05:11:20.766 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T05:11:20.776 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T05:11:20.824 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T05:11:20.877 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T05:11:20.912 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T05:11:20.937 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T05:11:20.979 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T05:11:21.306 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T05:11:21.363 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T05:11:21.467 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T05:11:21.553 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T05:11:21.630 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T05:11:21.660 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T05:11:21.700 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T05:11:21.875 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T05:11:21.925 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T05:11:21.953 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T05:11:21.991 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2024-03-28T05:11:22.039 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T05:11:22.062 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T05:11:22.099 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T05:11:22.109 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T05:11:22.136 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T05:11:22.141 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T05:11:22.164 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T05:11:22.166 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T05:11:22.176 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T05:11:22.179 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T05:11:22.225 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T05:11:22.281 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T05:11:22.312 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T05:11:22.345 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T05:11:22.372 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T05:11:22.390 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T05:11:22.403 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T05:11:22.443 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T05:11:22.548 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T05:11:22.646 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T05:11:22.706 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T05:11:22.733 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T05:11:22.779 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T05:11:22.788 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T05:11:22.840 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T05:11:22.912 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T05:11:23.056 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T05:11:23.063 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T05:11:23.148 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T05:11:23.176 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T05:11:23.177 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T05:11:23.219 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T05:11:23.267 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T05:11:23.304 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T05:11:23.331 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T05:11:23.375 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T05:11:23.411 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T05:11:23.449 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T05:11:23.467 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T05:11:23.494 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T05:11:23.503 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T05:11:23.549 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T05:11:23.582 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T05:11:23.611 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T05:11:23.614 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T05:11:23.648 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T05:11:23.668 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T05:11:23.692 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T05:11:23.707 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T05:11:23.725 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T05:11:23.729 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T05:11:23.745 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T05:11:23.752 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T05:11:23.823 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T05:11:23.866 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T05:11:23.902 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T05:11:23.905 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T05:11:23.930 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T05:11:23.940 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T05:11:23.993 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T05:11:23.997 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T05:11:24.044 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T05:11:24.117 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T05:11:24.181 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T05:11:24.250 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T05:11:24.306 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T05:11:24.365 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T05:11:24.414 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T05:11:24.500 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T05:11:24.548 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T05:11:24.742 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T05:11:24.817 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T05:11:24.856 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T05:11:24.902 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T05:11:24.961 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T05:11:24.981 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T05:11:24.995 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T05:11:25.013 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T05:11:25.022 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T05:11:25.023 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T05:11:25.052 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T05:11:25.066 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T05:11:25.101 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T05:11:25.139 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T05:11:25.142 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T05:11:25.169 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T05:11:25.204 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T05:11:25.241 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T05:11:25.315 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T05:11:25.348 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T05:11:25.390 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T05:11:25.391 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T05:11:25.424 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T05:11:25.451 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T05:11:25.523 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T05:11:25.556 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T05:11:25.583 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T05:11:25.597 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T05:11:25.628 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T05:11:25.655 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T05:11:25.698 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T05:11:25.701 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T05:11:25.751 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T05:11:25.828 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T05:11:25.830 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T05:11:25.885 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T05:11:25.949 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T05:11:25.973 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T05:11:26.038 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T05:11:26.080 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-28T05:11:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T05:11:26.216 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T05:11:26.219 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T05:11:26.249 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T05:11:26.258 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T05:11:26.262 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T05:11:26.298 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T05:11:26.298 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T05:11:26.346 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T05:11:26.352 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T05:11:26.381 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T05:11:26.386 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T05:11:26.412 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T05:11:26.424 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T05:11:26.453 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T05:11:26.478 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T05:11:26.597 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T05:11:26.614 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T05:11:26.667 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T05:11:26.678 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T05:11:26.696 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T05:11:26.738 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T05:11:26.773 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T05:11:26.820 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T05:11:26.830 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T05:11:26.831 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T05:11:26.838 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T05:11:26.863 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T05:11:26.864 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T05:11:26.871 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T05:11:26.903 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T05:11:26.910 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T05:11:26.929 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T05:11:26.953 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T05:11:26.964 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T05:11:26.992 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T05:11:26.995 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T05:11:27.014 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T05:11:27.026 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T05:11:27.105 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T05:11:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T05:11:27.171 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T05:11:27.204 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T05:11:27.293 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T05:11:27.307 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T05:11:27.345 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T05:11:27.355 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T05:11:27.394 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T05:11:27.399 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T05:11:27.428 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T05:11:27.448 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T05:11:27.530 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T05:11:27.554 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T05:11:27.574 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T05:11:27.617 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T05:11:27.640 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T05:11:27.664 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T05:11:27.665 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T05:11:27.688 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T05:11:27.691 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T05:11:27.704 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T05:11:27.708 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T05:11:27.820 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T05:11:27.856 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T05:11:27.860 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T05:11:27.887 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T05:11:27.888 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T05:11:27.902 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T05:11:27.920 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T05:11:27.920 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T05:11:27.947 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T05:11:27.951 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T05:11:27.959 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T05:11:27.990 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T05:11:28.029 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T05:11:28.033 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T05:11:28.047 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T05:11:28.061 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T05:11:28.077 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T05:11:28.091 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T05:11:28.115 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T05:11:28.145 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T05:11:28.161 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T05:11:28.167 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T05:11:28.197 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T05:11:28.201 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T05:11:28.214 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T05:11:28.240 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T05:11:28.243 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T05:11:28.282 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T05:11:28.284 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T05:11:28.289 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T05:11:28.319 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T05:11:28.332 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T05:11:28.408 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T05:11:28.421 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T05:11:28.471 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T05:11:28.497 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T05:11:28.501 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T05:11:28.526 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T05:11:28.541 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T05:11:28.543 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T05:11:28.573 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T05:11:28.612 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T05:11:28.640 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T05:11:28.675 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T05:11:28.708 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T05:11:28.716 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T05:11:28.739 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T05:11:28.741 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T05:11:28.758 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T05:11:28.766 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T05:11:28.845 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T05:11:28.973 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T05:11:29.006 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T05:11:29.037 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T05:11:29.064 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T05:11:29.080 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T05:11:29.099 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T05:11:29.124 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T05:11:29.172 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T05:11:29.183 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T05:11:29.227 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T05:11:29.247 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T05:11:29.273 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T05:11:29.337 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T05:11:29.373 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T05:11:29.416 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T05:11:29.438 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T05:11:29.449 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T05:11:29.474 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T05:11:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T05:11:29.487 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T05:11:29.522 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T05:11:29.546 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T05:11:29.556 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T05:11:29.582 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T05:11:29.604 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T05:11:29.625 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T05:11:29.631 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T05:11:29.654 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T05:11:29.668 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T05:11:29.684 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T05:11:29.700 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T05:11:29.722 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T05:11:29.723 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T05:11:29.731 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T05:11:29.759 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T05:11:29.768 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T05:11:29.785 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T05:11:29.825 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T05:11:29.841 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T05:11:29.864 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T05:11:29.894 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T05:11:29.899 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T05:11:29.930 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T05:11:29.973 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T05:11:29.973 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T05:11:30.004 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T05:11:30.006 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T05:11:30.029 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T05:11:30.037 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T05:11:30.079 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T05:11:30.113 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T05:11:30.206 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T05:11:30.303 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T05:11:30.345 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T05:11:30.384 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T05:11:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T05:11:30.535 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T05:11:30.566 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T05:11:30.586 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T05:11:30.640 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T05:11:30.649 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T05:11:30.727 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T05:11:30.742 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T05:11:30.770 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T05:11:30.779 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T05:11:30.810 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T05:11:30.851 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T05:11:30.888 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T05:11:30.926 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T05:11:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T05:11:31.355 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T05:11:31.356 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T05:11:31.407 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T05:11:31.429 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T05:11:31.441 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T05:11:31.469 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T05:11:31.491 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T05:11:31.582 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T05:11:31.610 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T05:11:31.616 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T05:11:31.655 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T05:11:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T05:11:31.773 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T05:11:31.801 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T05:11:31.806 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T05:11:31.839 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T05:11:31.842 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T05:11:31.889 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T05:11:31.919 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T05:11:31.933 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T05:11:31.951 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T05:11:31.977 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T05:11:32.019 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T05:11:32.038 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T05:11:32.070 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T05:11:32.101 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T05:11:32.128 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T05:11:32.152 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T05:11:32.191 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T05:11:32.221 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T05:11:32.263 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T05:11:32.364 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T05:11:32.404 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T05:11:32.440 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T05:11:32.444 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T05:11:32.467 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T05:11:32.470 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T05:11:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T05:11:32.549 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T05:11:32.568 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T05:11:32.580 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T05:11:32.616 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T05:11:32.616 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T05:11:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T05:11:32.732 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T05:11:32.793 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T05:11:32.845 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T05:11:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T05:11:32.958 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T05:11:33.075 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T05:11:33.104 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T05:11:33.141 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T05:11:33.174 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T05:11:33.191 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T05:11:33.196 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T05:11:33.328 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T05:11:33.364 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T05:11:33.396 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T05:11:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T05:11:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T05:11:33.490 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T05:11:33.528 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T05:11:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T05:11:33.599 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T05:11:33.673 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T05:11:33.709 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T05:11:33.742 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T05:11:33.784 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T05:11:33.818 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T05:11:33.907 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T05:11:33.988 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T05:11:34.032 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T05:11:34.037 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T05:11:34.064 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T05:11:34.110 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T05:11:34.131 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T05:11:34.162 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T05:11:34.193 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T05:11:34.207 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T05:11:34.221 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T05:11:34.257 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T05:11:34.542 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T05:11:34.575 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T05:11:34.667 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T05:11:34.697 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T05:11:34.734 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T05:11:34.975 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T05:11:35.021 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T05:11:35.055 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T05:11:35.103 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T05:11:35.181 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T05:11:35.214 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T05:11:35.218 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T05:11:35.245 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T05:11:35.300 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T05:11:35.330 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T05:11:35.382 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T05:11:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T05:11:35.441 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T05:11:35.481 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T05:11:35.485 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T05:11:36.020 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T05:11:36.054 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T05:11:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T05:11:36.137 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T05:11:36.138 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T05:11:36.138 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T05:11:36.138 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-28T05:11:36.166 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T05:11:36.233 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T05:11:37.424 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T05:11:37.513 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T05:11:37.544 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T05:11:37.575 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T05:11:37.603 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T05:11:37.617 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T05:11:37.668 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T05:11:37.915 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T05:11:37.941 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T05:11:38.032 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T05:11:38.063 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T05:11:38.098 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T05:11:38.238 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T05:11:38.280 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T05:11:38.373 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T05:11:38.373 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T05:11:38.373 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T05:11:38.373 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:11:38.414 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T05:11:38.557 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T05:11:38.586 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T05:11:38.640 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T05:11:38.663 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T05:11:40.804 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T05:11:40.835 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T05:11:41.368 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T05:11:41.404 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T05:11:41.486 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T05:11:41.486 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T05:11:41.486 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T05:11:41.486 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-28T05:11:41.514 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T05:11:47.853 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T05:11:47.853 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /sys 2024-03-28T05:11:47.853 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /proc 2024-03-28T05:11:47.853 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /mnt 2024-03-28T05:11:47.853 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /var/tmp 2024-03-28T05:11:47.853 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /home 2024-03-28T05:11:47.854 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /root 2024-03-28T05:11:47.854 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /tmp 2024-03-28T05:11:47.854 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-28T05:11:48.013 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T05:11:48.029 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T05:11:48.029 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:48.029 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T05:11:48.029 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-28T05:11:49.509 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T05:11:49.534 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T05:11:49.534 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:49.535 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T05:11:49.535 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T05:11:49.535 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T05:11:49.535 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-28T05:11:49.759 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T05:11:49.781 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T05:11:49.782 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:49.782 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T05:11:49.782 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T05:11:49.782 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T05:11:49.782 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-28T05:11:50.166 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T05:11:50.166 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /sys 2024-03-28T05:11:50.166 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /proc 2024-03-28T05:11:50.167 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /mnt 2024-03-28T05:11:50.167 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /var/tmp 2024-03-28T05:11:50.167 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /home 2024-03-28T05:11:50.167 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /root 2024-03-28T05:11:50.167 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /tmp 2024-03-28T05:11:50.167 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:11:50.218 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T05:11:50.241 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T05:11:50.241 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:50.241 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T05:11:50.241 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T05:11:50.241 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T05:11:50.241 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-28T05:11:50.336 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T05:11:50.344 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T05:11:50.359 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T05:11:50.359 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:50.360 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T05:11:50.360 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:11:50.746 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T05:11:50.777 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T05:11:50.809 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T05:11:50.868 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T05:11:50.891 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T05:11:51.124 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T05:11:51.151 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T05:11:51.880 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T05:11:51.906 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T05:11:51.906 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:51.906 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T05:11:51.906 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T05:11:51.906 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T05:11:51.907 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:11:52.104 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T05:11:52.126 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T05:11:52.139 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T05:11:52.162 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T05:11:52.162 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:52.162 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T05:11:52.162 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T05:11:52.162 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T05:11:52.162 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:11:52.225 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T05:11:52.396 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T05:11:52.428 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T05:11:52.451 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T05:11:52.451 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:52.451 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T05:11:52.452 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T05:11:52.452 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T05:11:52.452 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-28T05:11:52.495 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T05:11:52.506 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T05:11:52.551 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T05:11:52.623 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T05:11:52.646 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T05:11:52.646 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:52.646 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T05:11:52.646 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T05:11:52.646 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T05:11:52.647 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:11:52.742 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T05:11:53.151 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T05:11:53.182 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T05:11:53.215 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T05:11:53.296 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T05:11:53.322 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T05:11:53.383 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T05:11:53.383 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2024-03-28T05:11:53.383 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2024-03-28T05:11:53.383 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2024-03-28T05:11:53.383 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2024-03-28T05:11:53.384 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2024-03-28T05:11:53.384 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /root 2024-03-28T05:11:53.384 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2024-03-28T05:11:53.384 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-28T05:11:53.421 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T05:11:53.448 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T05:11:53.471 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T05:11:53.471 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:53.471 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T05:11:53.471 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T05:11:53.471 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T05:11:53.471 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-28T05:11:53.515 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T05:11:53.537 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T05:11:53.537 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:53.537 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T05:11:53.537 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-28T05:11:53.548 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T05:11:53.563 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T05:11:53.563 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:53.563 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T05:11:53.563 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-28T05:11:53.567 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T05:11:53.582 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T05:11:53.597 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T05:11:53.604 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T05:11:53.605 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:53.605 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T05:11:53.605 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T05:11:53.605 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T05:11:53.605 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-28T05:11:53.908 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T05:11:53.931 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T05:11:53.931 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:53.931 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T05:11:53.931 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T05:11:53.931 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T05:11:53.931 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-28T05:11:54.610 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T05:11:54.640 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T05:11:54.746 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T05:11:54.909 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T05:11:54.933 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T05:11:54.956 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T05:11:54.957 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:54.957 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T05:11:54.957 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T05:11:54.957 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T05:11:54.957 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:11:55.017 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T05:11:55.029 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T05:11:55.051 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T05:11:55.068 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T05:11:55.076 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T05:11:55.076 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:55.076 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T05:11:55.076 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T05:11:55.077 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T05:11:55.077 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-28T05:11:55.302 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T05:11:55.323 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T05:11:55.324 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:55.324 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T05:11:55.324 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T05:11:55.324 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T05:11:55.324 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-28T05:11:55.761 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T05:11:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T05:11:55.783 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:55.783 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T05:11:55.783 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T05:11:55.783 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T05:11:55.783 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-28T05:11:55.871 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T05:11:55.984 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T05:11:56.011 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T05:11:56.033 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T05:11:56.033 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:56.033 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T05:11:56.034 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T05:11:56.034 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T05:11:56.034 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:11:56.078 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T05:11:56.100 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T05:11:56.100 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:56.100 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T05:11:56.101 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:11:56.154 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T05:11:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T05:11:56.176 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:56.177 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T05:11:56.177 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T05:11:56.177 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T05:11:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:11:56.221 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T05:11:56.253 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T05:11:56.285 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T05:11:56.344 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T05:11:56.367 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T05:11:56.504 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T05:11:56.527 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T05:11:56.527 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:56.527 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T05:11:56.527 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T05:11:56.527 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T05:11:56.527 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:11:56.599 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T05:11:56.627 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T05:11:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T05:11:57.602 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T05:11:57.699 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T05:11:57.889 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T05:11:57.921 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T05:11:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T05:11:57.943 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:57.943 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T05:11:57.944 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T05:11:57.944 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T05:11:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-28T05:11:57.988 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T05:11:57.999 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T05:11:58.043 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T05:11:58.353 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T05:11:58.403 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T05:11:58.434 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T05:11:58.461 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T05:11:58.642 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T05:11:58.752 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T05:11:58.792 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T05:11:58.831 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T05:11:58.893 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T05:11:58.920 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T05:11:58.936 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T05:11:58.949 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T05:11:58.968 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T05:11:58.971 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T05:11:58.971 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:58.971 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T05:11:58.971 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T05:11:58.971 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T05:11:58.971 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-28T05:11:59.009 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T05:11:59.016 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T05:11:59.037 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T05:11:59.037 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:59.037 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T05:11:59.038 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-28T05:11:59.039 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T05:11:59.083 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T05:11:59.085 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T05:11:59.105 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T05:11:59.105 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:59.105 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T05:11:59.105 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T05:11:59.105 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T05:11:59.106 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-28T05:11:59.137 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T05:11:59.171 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T05:11:59.409 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T05:11:59.431 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T05:11:59.431 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T05:11:59.431 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T05:11:59.431 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T05:11:59.431 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T05:11:59.431 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-28T05:11:59.748 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T05:11:59.794 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T05:11:59.814 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T05:12:00.625 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T05:12:00.706 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T05:12:00.747 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T05:12:00.828 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T05:12:00.868 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T05:12:00.893 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T05:12:00.919 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T05:12:00.945 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T05:12:00.945 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T05:12:01.033 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T05:12:01.063 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T05:12:01.063 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T05:12:01.131 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T05:12:01.157 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T05:12:01.199 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T05:12:01.224 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T05:12:01.242 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T05:12:01.249 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T05:12:01.282 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T05:12:01.292 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T05:12:01.321 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T05:12:01.322 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T05:12:01.349 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T05:12:01.354 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T05:12:01.379 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T05:12:01.404 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T05:12:01.429 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T05:12:01.455 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T05:12:01.462 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T05:12:01.462 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T05:12:01.549 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T05:12:01.650 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T05:12:01.650 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T05:12:01.669 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T05:12:01.706 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T05:12:01.724 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T05:12:01.749 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T05:12:01.764 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T05:12:01.772 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T05:12:01.800 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T05:12:01.814 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T05:12:01.825 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T05:12:01.849 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T05:12:01.881 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T05:12:01.889 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T05:12:01.922 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T05:12:01.952 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T05:12:02.008 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T05:12:02.063 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T05:12:02.092 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T05:12:02.703 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T05:12:02.740 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T05:12:02.759 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T05:12:03.640 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T05:12:03.726 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T05:12:03.768 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T05:12:03.847 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T05:12:03.860 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T05:12:03.888 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T05:12:03.896 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T05:12:03.919 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T05:12:03.921 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T05:12:03.946 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T05:12:03.947 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T05:12:03.973 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T05:12:03.974 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T05:12:04.062 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T05:12:04.099 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T05:12:04.099 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T05:12:04.122 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T05:12:04.184 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T05:12:04.210 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T05:12:04.243 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T05:12:04.245 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T05:12:04.269 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T05:12:04.294 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T05:12:04.294 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T05:12:04.329 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T05:12:04.333 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T05:12:04.377 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T05:12:04.398 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T05:12:04.409 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T05:12:04.429 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T05:12:04.434 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T05:12:04.459 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T05:12:04.462 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T05:12:04.494 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T05:12:04.501 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T05:12:04.524 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T05:12:04.526 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T05:12:04.533 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T05:12:04.533 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T05:12:04.591 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T05:12:04.653 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T05:12:04.690 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T05:12:04.729 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T05:12:04.729 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T05:12:04.820 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T05:12:04.845 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T05:12:04.876 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T05:12:04.904 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T05:12:04.929 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T05:12:04.998 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T05:12:05.265 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T05:12:05.304 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T05:12:05.323 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T05:12:06.129 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T05:12:06.216 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T05:12:06.257 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T05:12:06.336 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T05:12:06.370 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T05:12:06.395 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T05:12:06.421 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T05:12:06.447 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T05:12:06.447 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T05:12:06.535 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T05:12:06.573 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T05:12:06.573 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T05:12:06.666 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T05:12:06.692 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T05:12:06.726 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T05:12:06.751 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T05:12:06.776 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T05:12:06.813 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T05:12:06.813 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T05:12:06.813 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T05:12:06.813 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T05:12:06.813 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T05:12:06.814 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T05:12:06.814 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T05:12:06.814 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T05:12:06.814 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T05:12:06.814 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T05:12:06.814 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T05:12:06.814 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T05:12:06.814 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T05:12:06.814 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T05:12:06.814 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T05:12:06.815 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T05:12:06.815 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T05:12:06.815 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T05:12:06.815 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T05:12:06.815 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T05:12:06.815 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T05:12:06.815 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T05:12:06.815 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T05:12:06.815 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T05:12:06.815 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T05:12:06.815 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T05:12:06.816 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T05:12:06.816 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T05:12:06.816 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T05:12:06.816 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T05:12:06.816 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T05:12:06.816 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T05:12:06.816 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T05:12:06.816 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T05:12:06.816 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T05:12:06.816 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T05:12:06.816 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T05:12:06.816 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T05:12:06.817 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T05:12:06.817 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T05:12:06.817 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T05:12:06.817 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T05:12:06.817 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T05:12:06.817 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T05:12:06.817 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T05:12:06.817 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T05:12:06.817 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T05:12:06.817 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T05:12:06.817 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T05:12:06.818 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T05:12:06.818 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T05:12:06.818 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T05:12:06.818 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T05:12:06.818 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T05:12:06.818 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T05:12:06.818 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T05:12:06.818 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T05:12:06.818 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T05:12:06.818 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T05:12:06.818 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T05:12:06.819 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T05:12:06.819 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T05:12:06.819 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T05:12:06.819 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T05:12:06.819 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T05:12:06.819 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T05:12:06.819 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T05:12:06.819 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T05:12:06.819 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T05:12:06.819 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T05:12:06.819 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T05:12:06.819 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T05:12:06.820 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T05:12:06.820 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T05:12:06.820 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T05:12:06.820 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T05:12:06.820 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T05:12:06.820 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T05:12:06.820 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T05:12:06.820 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T05:12:06.820 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T05:12:06.820 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T05:12:06.820 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T05:12:06.821 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T05:12:06.821 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T05:12:06.821 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T05:12:06.821 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T05:12:06.821 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T05:12:06.821 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T05:12:06.821 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T05:12:06.821 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T05:12:06.821 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T05:12:06.821 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T05:12:06.821 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T05:12:06.821 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T05:12:06.821 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T05:12:06.822 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T05:12:06.822 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T05:12:06.822 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T05:12:06.822 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T05:12:06.822 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T05:12:06.822 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T05:12:06.822 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T05:12:06.822 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T05:12:06.822 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T05:12:06.822 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T05:12:06.822 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T05:12:06.823 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T05:12:06.823 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T05:12:06.823 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T05:12:06.823 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T05:12:06.823 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T05:12:06.823 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T05:12:06.823 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T05:12:06.823 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T05:12:06.823 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T05:12:06.823 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T05:12:06.823 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T05:12:06.823 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T05:12:06.824 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T05:12:06.824 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T05:12:06.824 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T05:12:06.824 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T05:12:06.824 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T05:12:06.824 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T05:12:06.824 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T05:12:06.824 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T05:12:06.824 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T05:12:06.824 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T05:12:06.825 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T05:12:06.825 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T05:12:06.825 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T05:12:06.825 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T05:12:06.825 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T05:12:06.825 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T05:12:06.825 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T05:12:06.825 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T05:12:06.825 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T05:12:06.825 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T05:12:06.825 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T05:12:06.825 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T05:12:06.826 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T05:12:06.826 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T05:12:06.826 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T05:12:06.826 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T05:12:06.826 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T05:12:06.826 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T05:12:06.826 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T05:12:06.826 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T05:12:06.826 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T05:12:06.826 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T05:12:06.826 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T05:12:06.827 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T05:12:06.827 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T05:12:06.827 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T05:12:06.827 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T05:12:06.827 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T05:12:06.827 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T05:12:06.827 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T05:12:06.827 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T05:12:06.827 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T05:12:06.827 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T05:12:06.827 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T05:12:06.827 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T05:12:06.828 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T05:12:06.828 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T05:12:06.828 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T05:12:06.828 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T05:12:06.828 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T05:12:06.828 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T05:12:06.828 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T05:12:06.828 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T05:12:06.828 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T05:12:06.828 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T05:12:06.828 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T05:12:06.828 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T05:12:06.829 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T05:12:06.829 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T05:12:06.829 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T05:12:06.829 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T05:12:06.829 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T05:12:06.829 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T05:12:06.829 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T05:12:06.829 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T05:12:06.829 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T05:12:06.829 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T05:12:06.829 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T05:12:06.829 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T05:12:06.829 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T05:12:06.830 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T05:12:06.830 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T05:12:06.830 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T05:12:06.830 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T05:12:06.830 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T05:12:06.830 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T05:12:06.830 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T05:12:06.830 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T05:12:06.830 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T05:12:06.831 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T05:12:06.848 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T05:12:06.881 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T05:12:06.906 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T05:12:06.960 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T05:12:06.991 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T05:12:07.016 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T05:12:07.022 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T05:12:07.022 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T05:12:07.202 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T05:12:07.202 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T05:12:07.268 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T05:12:07.293 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T05:12:07.324 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T05:12:07.352 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T05:12:07.377 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T05:12:07.441 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T05:12:09.651 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T05:12:09.652 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-28T05:12:09.652 INFO:teuthology.orchestra.run.smithi113.stdout:Upgraded: 2024-03-28T05:12:09.652 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T05:12:09.652 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T05:12:09.652 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T05:12:09.652 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T05:12:09.652 INFO:teuthology.orchestra.run.smithi113.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T05:12:09.652 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T05:12:09.652 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T05:12:09.652 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T05:12:09.653 INFO:teuthology.orchestra.run.smithi113.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T05:12:09.653 INFO:teuthology.orchestra.run.smithi113.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T05:12:09.653 INFO:teuthology.orchestra.run.smithi113.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T05:12:09.653 INFO:teuthology.orchestra.run.smithi113.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T05:12:09.653 INFO:teuthology.orchestra.run.smithi113.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T05:12:09.653 INFO:teuthology.orchestra.run.smithi113.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.653 INFO:teuthology.orchestra.run.smithi113.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.653 INFO:teuthology.orchestra.run.smithi113.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T05:12:09.653 INFO:teuthology.orchestra.run.smithi113.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T05:12:09.653 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T05:12:09.653 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T05:12:09.654 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T05:12:09.654 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T05:12:09.654 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T05:12:09.654 INFO:teuthology.orchestra.run.smithi113.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T05:12:09.654 INFO:teuthology.orchestra.run.smithi113.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T05:12:09.654 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-03-28T05:12:09.654 INFO:teuthology.orchestra.run.smithi113.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T05:12:09.654 INFO:teuthology.orchestra.run.smithi113.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T05:12:09.654 INFO:teuthology.orchestra.run.smithi113.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T05:12:09.654 INFO:teuthology.orchestra.run.smithi113.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T05:12:09.655 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.655 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.655 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.655 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.655 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:09.655 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.655 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.655 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.655 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:09.655 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:09.656 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:09.656 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:09.656 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:09.656 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.656 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.656 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:09.656 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.656 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.656 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.656 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:09.656 INFO:teuthology.orchestra.run.smithi113.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:09.657 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.657 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.657 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:09.657 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T05:12:09.657 INFO:teuthology.orchestra.run.smithi113.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T05:12:09.657 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T05:12:09.657 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T05:12:09.657 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T05:12:09.657 INFO:teuthology.orchestra.run.smithi113.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T05:12:09.657 INFO:teuthology.orchestra.run.smithi113.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T05:12:09.658 INFO:teuthology.orchestra.run.smithi113.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T05:12:09.658 INFO:teuthology.orchestra.run.smithi113.stdout: jq-1.6-16.el9.x86_64 2024-03-28T05:12:09.658 INFO:teuthology.orchestra.run.smithi113.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T05:12:09.658 INFO:teuthology.orchestra.run.smithi113.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T05:12:09.658 INFO:teuthology.orchestra.run.smithi113.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T05:12:09.658 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.658 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.658 INFO:teuthology.orchestra.run.smithi113.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.658 INFO:teuthology.orchestra.run.smithi113.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T05:12:09.658 INFO:teuthology.orchestra.run.smithi113.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T05:12:09.658 INFO:teuthology.orchestra.run.smithi113.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T05:12:09.659 INFO:teuthology.orchestra.run.smithi113.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T05:12:09.659 INFO:teuthology.orchestra.run.smithi113.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T05:12:09.659 INFO:teuthology.orchestra.run.smithi113.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.659 INFO:teuthology.orchestra.run.smithi113.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.659 INFO:teuthology.orchestra.run.smithi113.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T05:12:09.659 INFO:teuthology.orchestra.run.smithi113.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.659 INFO:teuthology.orchestra.run.smithi113.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T05:12:09.659 INFO:teuthology.orchestra.run.smithi113.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T05:12:09.659 INFO:teuthology.orchestra.run.smithi113.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T05:12:09.659 INFO:teuthology.orchestra.run.smithi113.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T05:12:09.659 INFO:teuthology.orchestra.run.smithi113.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T05:12:09.660 INFO:teuthology.orchestra.run.smithi113.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T05:12:09.660 INFO:teuthology.orchestra.run.smithi113.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T05:12:09.660 INFO:teuthology.orchestra.run.smithi113.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T05:12:09.660 INFO:teuthology.orchestra.run.smithi113.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T05:12:09.660 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T05:12:09.660 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T05:12:09.660 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T05:12:09.660 INFO:teuthology.orchestra.run.smithi113.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T05:12:09.660 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T05:12:09.660 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T05:12:09.660 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T05:12:09.661 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T05:12:09.661 INFO:teuthology.orchestra.run.smithi113.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T05:12:09.661 INFO:teuthology.orchestra.run.smithi113.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T05:12:09.661 INFO:teuthology.orchestra.run.smithi113.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T05:12:09.661 INFO:teuthology.orchestra.run.smithi113.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T05:12:09.661 INFO:teuthology.orchestra.run.smithi113.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T05:12:09.661 INFO:teuthology.orchestra.run.smithi113.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T05:12:09.661 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T05:12:09.661 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.661 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.661 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.661 INFO:teuthology.orchestra.run.smithi113.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T05:12:09.662 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T05:12:09.662 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T05:12:09.662 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T05:12:09.662 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T05:12:09.662 INFO:teuthology.orchestra.run.smithi113.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T05:12:09.662 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T05:12:09.662 INFO:teuthology.orchestra.run.smithi113.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T05:12:09.662 INFO:teuthology.orchestra.run.smithi113.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T05:12:09.662 INFO:teuthology.orchestra.run.smithi113.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T05:12:09.662 INFO:teuthology.orchestra.run.smithi113.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T05:12:09.662 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T05:12:09.663 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T05:12:09.663 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T05:12:09.663 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T05:12:09.663 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T05:12:09.663 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T05:12:09.663 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T05:12:09.663 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T05:12:09.663 INFO:teuthology.orchestra.run.smithi113.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T05:12:09.663 INFO:teuthology.orchestra.run.smithi113.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T05:12:09.663 INFO:teuthology.orchestra.run.smithi113.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T05:12:09.663 INFO:teuthology.orchestra.run.smithi113.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T05:12:09.663 INFO:teuthology.orchestra.run.smithi113.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T05:12:09.664 INFO:teuthology.orchestra.run.smithi113.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T05:12:09.664 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T05:12:09.664 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T05:12:09.664 INFO:teuthology.orchestra.run.smithi113.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T05:12:09.664 INFO:teuthology.orchestra.run.smithi113.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T05:12:09.664 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T05:12:09.664 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T05:12:09.664 INFO:teuthology.orchestra.run.smithi113.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T05:12:09.664 INFO:teuthology.orchestra.run.smithi113.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T05:12:09.664 INFO:teuthology.orchestra.run.smithi113.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T05:12:09.664 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T05:12:09.664 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T05:12:09.665 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T05:12:09.665 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T05:12:09.665 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T05:12:09.665 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T05:12:09.665 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.665 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.665 INFO:teuthology.orchestra.run.smithi113.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T05:12:09.665 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T05:12:09.665 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T05:12:09.665 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.665 INFO:teuthology.orchestra.run.smithi113.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T05:12:09.666 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T05:12:09.666 INFO:teuthology.orchestra.run.smithi113.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T05:12:09.666 INFO:teuthology.orchestra.run.smithi113.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T05:12:09.666 INFO:teuthology.orchestra.run.smithi113.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T05:12:09.666 INFO:teuthology.orchestra.run.smithi113.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T05:12:09.666 INFO:teuthology.orchestra.run.smithi113.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T05:12:09.666 INFO:teuthology.orchestra.run.smithi113.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T05:12:09.666 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T05:12:09.666 INFO:teuthology.orchestra.run.smithi113.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T05:12:09.666 INFO:teuthology.orchestra.run.smithi113.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T05:12:09.666 INFO:teuthology.orchestra.run.smithi113.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T05:12:09.667 INFO:teuthology.orchestra.run.smithi113.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T05:12:09.667 INFO:teuthology.orchestra.run.smithi113.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T05:12:09.667 INFO:teuthology.orchestra.run.smithi113.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T05:12:09.667 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.667 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.667 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:09.667 INFO:teuthology.orchestra.run.smithi113.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T05:12:09.667 INFO:teuthology.orchestra.run.smithi113.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T05:12:09.667 INFO:teuthology.orchestra.run.smithi113.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T05:12:09.667 INFO:teuthology.orchestra.run.smithi113.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T05:12:09.667 INFO:teuthology.orchestra.run.smithi113.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T05:12:09.667 INFO:teuthology.orchestra.run.smithi113.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T05:12:09.668 INFO:teuthology.orchestra.run.smithi113.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T05:12:09.668 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-28T05:12:09.668 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-28T05:12:09.879 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T05:12:09.879 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T05:12:09.879 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T05:12:09.879 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T05:12:09.879 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T05:12:09.880 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T05:12:09.880 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T05:12:09.880 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T05:12:09.880 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T05:12:09.880 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T05:12:09.880 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T05:12:09.880 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T05:12:09.880 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T05:12:09.880 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T05:12:09.880 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T05:12:09.880 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T05:12:09.880 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T05:12:09.881 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T05:12:09.881 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T05:12:09.881 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T05:12:09.881 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T05:12:09.881 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T05:12:09.881 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T05:12:09.881 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T05:12:09.881 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T05:12:09.881 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T05:12:09.881 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T05:12:09.881 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T05:12:09.882 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T05:12:09.882 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T05:12:09.882 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T05:12:09.882 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T05:12:09.882 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T05:12:09.882 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T05:12:09.882 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T05:12:09.882 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T05:12:09.882 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T05:12:09.882 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T05:12:09.883 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T05:12:09.883 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T05:12:09.883 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T05:12:09.883 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T05:12:09.883 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T05:12:09.883 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T05:12:09.883 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T05:12:09.883 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T05:12:09.883 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T05:12:09.883 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T05:12:09.883 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T05:12:09.883 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T05:12:09.883 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T05:12:09.884 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T05:12:09.884 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T05:12:09.884 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T05:12:09.884 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T05:12:09.884 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T05:12:09.884 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T05:12:09.884 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T05:12:09.884 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T05:12:09.884 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T05:12:09.884 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T05:12:09.884 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T05:12:09.884 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T05:12:09.885 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T05:12:09.885 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T05:12:09.885 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T05:12:09.885 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T05:12:09.885 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T05:12:09.885 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T05:12:09.885 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T05:12:09.885 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T05:12:09.885 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T05:12:09.885 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T05:12:09.885 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T05:12:09.885 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T05:12:09.886 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T05:12:09.886 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T05:12:09.886 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T05:12:09.886 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T05:12:09.886 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T05:12:09.886 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T05:12:09.886 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T05:12:09.886 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T05:12:09.886 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T05:12:09.886 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T05:12:09.886 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T05:12:09.886 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T05:12:09.887 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T05:12:09.887 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T05:12:09.887 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T05:12:09.887 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T05:12:09.887 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T05:12:09.887 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T05:12:09.887 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T05:12:09.887 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T05:12:09.887 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T05:12:09.887 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T05:12:09.887 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T05:12:09.887 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T05:12:09.887 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T05:12:09.888 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T05:12:09.888 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T05:12:09.888 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T05:12:09.888 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T05:12:09.888 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T05:12:09.888 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T05:12:09.888 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T05:12:09.888 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T05:12:09.888 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T05:12:09.888 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T05:12:09.888 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T05:12:09.888 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T05:12:09.888 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T05:12:09.889 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T05:12:09.889 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T05:12:09.889 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T05:12:09.889 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T05:12:09.889 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T05:12:09.889 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T05:12:09.889 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T05:12:09.889 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T05:12:09.889 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T05:12:09.889 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T05:12:09.889 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T05:12:09.889 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T05:12:09.889 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T05:12:09.890 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T05:12:09.890 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T05:12:09.890 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T05:12:09.890 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T05:12:09.890 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T05:12:09.890 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T05:12:09.890 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T05:12:09.890 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T05:12:09.890 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T05:12:09.890 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T05:12:09.890 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T05:12:09.890 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T05:12:09.891 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T05:12:09.891 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T05:12:09.891 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T05:12:09.891 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T05:12:09.891 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T05:12:09.891 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T05:12:09.891 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T05:12:09.891 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T05:12:09.891 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T05:12:09.891 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T05:12:09.891 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T05:12:09.891 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T05:12:09.891 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T05:12:09.892 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T05:12:09.892 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T05:12:09.892 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T05:12:09.892 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T05:12:09.892 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T05:12:09.892 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T05:12:09.892 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T05:12:09.892 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T05:12:09.892 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T05:12:09.892 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T05:12:09.892 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T05:12:09.892 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T05:12:09.893 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T05:12:09.893 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T05:12:09.893 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T05:12:09.893 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T05:12:09.893 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T05:12:09.893 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T05:12:09.893 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T05:12:09.893 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T05:12:09.893 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T05:12:09.893 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T05:12:09.893 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T05:12:09.893 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T05:12:09.893 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T05:12:09.894 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T05:12:09.894 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T05:12:09.894 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T05:12:09.894 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T05:12:09.894 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T05:12:09.894 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T05:12:09.894 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T05:12:09.894 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T05:12:09.894 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T05:12:09.894 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T05:12:09.894 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T05:12:09.895 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T05:12:09.895 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T05:12:09.895 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T05:12:09.895 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T05:12:09.895 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T05:12:09.895 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T05:12:09.895 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T05:12:09.895 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T05:12:09.895 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T05:12:09.895 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T05:12:09.895 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T05:12:10.169 DEBUG:teuthology.parallel:result is None 2024-03-28T05:12:12.311 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T05:12:12.311 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T05:12:12.311 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T05:12:12.311 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T05:12:12.311 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T05:12:12.311 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T05:12:12.311 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T05:12:12.311 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T05:12:12.311 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T05:12:12.311 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T05:12:12.312 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T05:12:12.312 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T05:12:12.312 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T05:12:12.312 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T05:12:12.312 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T05:12:12.312 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T05:12:12.312 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T05:12:12.312 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T05:12:12.312 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T05:12:12.312 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T05:12:12.312 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T05:12:12.312 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T05:12:12.312 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T05:12:12.312 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T05:12:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T05:12:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T05:12:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T05:12:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T05:12:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T05:12:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T05:12:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T05:12:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T05:12:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T05:12:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T05:12:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T05:12:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T05:12:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T05:12:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T05:12:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T05:12:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T05:12:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T05:12:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T05:12:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T05:12:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T05:12:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T05:12:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T05:12:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T05:12:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T05:12:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T05:12:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T05:12:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T05:12:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T05:12:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T05:12:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T05:12:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T05:12:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T05:12:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T05:12:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T05:12:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T05:12:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T05:12:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T05:12:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T05:12:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T05:12:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T05:12:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T05:12:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T05:12:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T05:12:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T05:12:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T05:12:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T05:12:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T05:12:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T05:12:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T05:12:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T05:12:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T05:12:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T05:12:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T05:12:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T05:12:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T05:12:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T05:12:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T05:12:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T05:12:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T05:12:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T05:12:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T05:12:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T05:12:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T05:12:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T05:12:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T05:12:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T05:12:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T05:12:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T05:12:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T05:12:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T05:12:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T05:12:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T05:12:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T05:12:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T05:12:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T05:12:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T05:12:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T05:12:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T05:12:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T05:12:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T05:12:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T05:12:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T05:12:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T05:12:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T05:12:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T05:12:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T05:12:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T05:12:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T05:12:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T05:12:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T05:12:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T05:12:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T05:12:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T05:12:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T05:12:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T05:12:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T05:12:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T05:12:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T05:12:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T05:12:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T05:12:12.320 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T05:12:12.320 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T05:12:12.320 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T05:12:12.320 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T05:12:12.320 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T05:12:12.320 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T05:12:12.320 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T05:12:12.320 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T05:12:12.320 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T05:12:12.320 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T05:12:12.320 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T05:12:12.320 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T05:12:12.320 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T05:12:12.320 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T05:12:12.320 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T05:12:12.321 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T05:12:12.321 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T05:12:12.321 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T05:12:12.321 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T05:12:12.321 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T05:12:12.321 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T05:12:12.321 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T05:12:12.321 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T05:12:12.321 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T05:12:12.321 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T05:12:12.321 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T05:12:12.321 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T05:12:12.321 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T05:12:12.321 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T05:12:12.321 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T05:12:12.322 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T05:12:12.322 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T05:12:12.322 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T05:12:12.322 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T05:12:12.322 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T05:12:12.322 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T05:12:12.322 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T05:12:12.322 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T05:12:12.322 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T05:12:12.322 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T05:12:12.322 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T05:12:12.322 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T05:12:12.322 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T05:12:12.322 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T05:12:12.323 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T05:12:12.323 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T05:12:12.323 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T05:12:12.323 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T05:12:12.323 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T05:12:12.323 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T05:12:12.323 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T05:12:12.323 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T05:12:12.323 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T05:12:12.323 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T05:12:12.323 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T05:12:12.323 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T05:12:12.323 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T05:12:12.323 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T05:12:12.323 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T05:12:12.324 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T05:12:12.324 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T05:12:12.324 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T05:12:12.324 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T05:12:12.324 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T05:12:12.324 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T05:12:12.324 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T05:12:12.324 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T05:12:12.324 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T05:12:12.324 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T05:12:12.324 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T05:12:12.324 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T05:12:12.324 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T05:12:12.324 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T05:12:12.324 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T05:12:12.963 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T05:12:12.963 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:12:12.964 INFO:teuthology.orchestra.run.smithi023.stdout:Upgraded: 2024-03-28T05:12:12.964 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T05:12:12.964 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T05:12:12.964 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T05:12:12.964 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T05:12:12.964 INFO:teuthology.orchestra.run.smithi023.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T05:12:12.964 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T05:12:12.964 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T05:12:12.964 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T05:12:12.964 INFO:teuthology.orchestra.run.smithi023.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T05:12:12.964 INFO:teuthology.orchestra.run.smithi023.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T05:12:12.965 INFO:teuthology.orchestra.run.smithi023.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T05:12:12.965 INFO:teuthology.orchestra.run.smithi023.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T05:12:12.965 INFO:teuthology.orchestra.run.smithi023.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T05:12:12.965 INFO:teuthology.orchestra.run.smithi023.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.965 INFO:teuthology.orchestra.run.smithi023.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.965 INFO:teuthology.orchestra.run.smithi023.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T05:12:12.965 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T05:12:12.965 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T05:12:12.965 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T05:12:12.965 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T05:12:12.965 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T05:12:12.965 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T05:12:12.966 INFO:teuthology.orchestra.run.smithi023.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T05:12:12.966 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T05:12:12.966 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2024-03-28T05:12:12.966 INFO:teuthology.orchestra.run.smithi023.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T05:12:12.966 INFO:teuthology.orchestra.run.smithi023.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T05:12:12.966 INFO:teuthology.orchestra.run.smithi023.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T05:12:12.966 INFO:teuthology.orchestra.run.smithi023.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T05:12:12.966 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.966 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.966 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.967 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.967 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:12.967 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.967 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.967 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.967 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:12.967 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:12.967 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:12.967 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:12.967 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:12.967 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.967 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.968 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:12.968 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.968 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.968 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.968 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:12.968 INFO:teuthology.orchestra.run.smithi023.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:12.968 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.968 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.968 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:12.968 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T05:12:12.968 INFO:teuthology.orchestra.run.smithi023.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T05:12:12.968 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T05:12:12.969 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T05:12:12.969 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T05:12:12.969 INFO:teuthology.orchestra.run.smithi023.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T05:12:12.969 INFO:teuthology.orchestra.run.smithi023.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T05:12:12.969 INFO:teuthology.orchestra.run.smithi023.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T05:12:12.969 INFO:teuthology.orchestra.run.smithi023.stdout: jq-1.6-16.el9.x86_64 2024-03-28T05:12:12.969 INFO:teuthology.orchestra.run.smithi023.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T05:12:12.969 INFO:teuthology.orchestra.run.smithi023.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T05:12:12.969 INFO:teuthology.orchestra.run.smithi023.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T05:12:12.969 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.969 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.969 INFO:teuthology.orchestra.run.smithi023.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.970 INFO:teuthology.orchestra.run.smithi023.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T05:12:12.970 INFO:teuthology.orchestra.run.smithi023.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T05:12:12.970 INFO:teuthology.orchestra.run.smithi023.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T05:12:12.970 INFO:teuthology.orchestra.run.smithi023.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T05:12:12.970 INFO:teuthology.orchestra.run.smithi023.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T05:12:12.970 INFO:teuthology.orchestra.run.smithi023.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.970 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.970 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T05:12:12.970 INFO:teuthology.orchestra.run.smithi023.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.970 INFO:teuthology.orchestra.run.smithi023.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T05:12:12.970 INFO:teuthology.orchestra.run.smithi023.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T05:12:12.971 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T05:12:12.971 INFO:teuthology.orchestra.run.smithi023.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T05:12:12.971 INFO:teuthology.orchestra.run.smithi023.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T05:12:12.971 INFO:teuthology.orchestra.run.smithi023.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T05:12:12.971 INFO:teuthology.orchestra.run.smithi023.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T05:12:12.971 INFO:teuthology.orchestra.run.smithi023.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T05:12:12.971 INFO:teuthology.orchestra.run.smithi023.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T05:12:12.971 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T05:12:12.971 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T05:12:12.971 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T05:12:12.971 INFO:teuthology.orchestra.run.smithi023.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T05:12:12.971 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T05:12:12.972 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T05:12:12.972 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T05:12:12.972 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T05:12:12.972 INFO:teuthology.orchestra.run.smithi023.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T05:12:12.972 INFO:teuthology.orchestra.run.smithi023.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T05:12:12.972 INFO:teuthology.orchestra.run.smithi023.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T05:12:12.972 INFO:teuthology.orchestra.run.smithi023.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T05:12:12.972 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T05:12:12.972 INFO:teuthology.orchestra.run.smithi023.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T05:12:12.972 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T05:12:12.972 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.972 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.973 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.973 INFO:teuthology.orchestra.run.smithi023.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T05:12:12.973 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T05:12:12.973 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T05:12:12.973 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T05:12:12.973 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T05:12:12.973 INFO:teuthology.orchestra.run.smithi023.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T05:12:12.973 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T05:12:12.973 INFO:teuthology.orchestra.run.smithi023.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T05:12:12.973 INFO:teuthology.orchestra.run.smithi023.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T05:12:12.973 INFO:teuthology.orchestra.run.smithi023.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T05:12:12.973 INFO:teuthology.orchestra.run.smithi023.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T05:12:12.974 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T05:12:12.974 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T05:12:12.974 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T05:12:12.974 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T05:12:12.974 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T05:12:12.974 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T05:12:12.974 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T05:12:12.974 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T05:12:12.974 INFO:teuthology.orchestra.run.smithi023.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T05:12:12.974 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T05:12:12.974 INFO:teuthology.orchestra.run.smithi023.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T05:12:12.974 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T05:12:12.975 INFO:teuthology.orchestra.run.smithi023.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T05:12:12.975 INFO:teuthology.orchestra.run.smithi023.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T05:12:12.975 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T05:12:12.975 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T05:12:12.975 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T05:12:12.975 INFO:teuthology.orchestra.run.smithi023.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T05:12:12.975 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T05:12:12.975 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T05:12:12.975 INFO:teuthology.orchestra.run.smithi023.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T05:12:12.975 INFO:teuthology.orchestra.run.smithi023.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T05:12:12.975 INFO:teuthology.orchestra.run.smithi023.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T05:12:12.975 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T05:12:12.976 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T05:12:12.976 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T05:12:12.976 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T05:12:12.976 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T05:12:12.976 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T05:12:12.976 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.976 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.976 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T05:12:12.976 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T05:12:12.976 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T05:12:12.976 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.976 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T05:12:12.977 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T05:12:12.977 INFO:teuthology.orchestra.run.smithi023.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T05:12:12.977 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T05:12:12.977 INFO:teuthology.orchestra.run.smithi023.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T05:12:12.977 INFO:teuthology.orchestra.run.smithi023.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T05:12:12.977 INFO:teuthology.orchestra.run.smithi023.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T05:12:12.977 INFO:teuthology.orchestra.run.smithi023.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T05:12:12.977 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T05:12:12.977 INFO:teuthology.orchestra.run.smithi023.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T05:12:12.977 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T05:12:12.977 INFO:teuthology.orchestra.run.smithi023.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T05:12:12.977 INFO:teuthology.orchestra.run.smithi023.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T05:12:12.977 INFO:teuthology.orchestra.run.smithi023.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T05:12:12.978 INFO:teuthology.orchestra.run.smithi023.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T05:12:12.978 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.978 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.978 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:12.978 INFO:teuthology.orchestra.run.smithi023.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T05:12:12.978 INFO:teuthology.orchestra.run.smithi023.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T05:12:12.978 INFO:teuthology.orchestra.run.smithi023.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T05:12:12.978 INFO:teuthology.orchestra.run.smithi023.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T05:12:12.978 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T05:12:12.978 INFO:teuthology.orchestra.run.smithi023.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T05:12:12.979 INFO:teuthology.orchestra.run.smithi023.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T05:12:12.979 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:12:12.979 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2024-03-28T05:12:13.466 DEBUG:teuthology.parallel:result is None 2024-03-28T05:12:15.250 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T05:12:15.250 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-28T05:12:15.250 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2024-03-28T05:12:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T05:12:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T05:12:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T05:12:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T05:12:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T05:12:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T05:12:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T05:12:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T05:12:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T05:12:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T05:12:15.251 INFO:teuthology.orchestra.run.smithi098.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T05:12:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T05:12:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T05:12:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T05:12:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T05:12:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T05:12:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T05:12:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T05:12:15.252 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T05:12:15.253 INFO:teuthology.orchestra.run.smithi098.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T05:12:15.253 INFO:teuthology.orchestra.run.smithi098.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T05:12:15.253 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T05:12:15.253 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-28T05:12:15.253 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T05:12:15.253 INFO:teuthology.orchestra.run.smithi098.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T05:12:15.253 INFO:teuthology.orchestra.run.smithi098.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T05:12:15.253 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T05:12:15.253 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.253 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.253 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:15.254 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.255 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T05:12:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T05:12:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T05:12:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T05:12:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T05:12:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T05:12:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T05:12:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T05:12:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T05:12:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: jq-1.6-16.el9.x86_64 2024-03-28T05:12:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T05:12:15.256 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T05:12:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T05:12:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T05:12:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T05:12:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T05:12:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T05:12:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T05:12:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.257 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T05:12:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T05:12:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T05:12:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T05:12:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T05:12:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T05:12:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T05:12:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T05:12:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T05:12:15.258 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T05:12:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T05:12:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T05:12:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T05:12:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T05:12:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T05:12:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T05:12:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T05:12:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T05:12:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T05:12:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T05:12:15.259 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T05:12:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T05:12:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T05:12:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T05:12:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T05:12:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T05:12:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T05:12:15.260 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T05:12:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T05:12:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T05:12:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T05:12:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T05:12:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T05:12:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T05:12:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T05:12:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T05:12:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T05:12:15.261 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T05:12:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T05:12:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T05:12:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T05:12:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T05:12:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T05:12:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T05:12:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T05:12:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T05:12:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T05:12:15.262 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T05:12:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T05:12:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T05:12:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T05:12:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T05:12:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T05:12:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T05:12:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T05:12:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T05:12:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T05:12:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T05:12:15.263 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T05:12:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T05:12:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T05:12:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T05:12:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T05:12:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T05:12:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T05:12:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T05:12:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T05:12:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T05:12:15.264 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T05:12:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T05:12:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T05:12:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T05:12:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T05:12:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T05:12:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T05:12:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T05:12:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T05:12:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T05:12:15.265 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T05:12:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T05:12:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T05:12:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T05:12:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T05:12:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T05:12:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T05:12:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T05:12:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T05:12:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T05:12:15.266 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T05:12:15.267 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T05:12:15.267 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T05:12:15.267 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-28T05:12:15.267 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-28T05:12:15.749 DEBUG:teuthology.parallel:result is None 2024-03-28T05:12:15.750 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T05:12:15.975 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T05:12:16.001 INFO:teuthology.orchestra.run.smithi023.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T05:12:16.002 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T05:12:16.002 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T05:12:16.004 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T05:12:16.222 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T05:12:16.247 INFO:teuthology.orchestra.run.smithi098.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T05:12:16.248 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T05:12:16.248 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T05:12:16.249 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T05:12:16.470 DEBUG:teuthology.orchestra.run.smithi113:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T05:12:16.495 INFO:teuthology.orchestra.run.smithi113.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T05:12:16.495 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T05:12:16.495 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T05:12:16.496 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-28T05:12:16.497 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T05:12:16.497 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T05:12:16.531 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-28T05:12:16.531 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T05:12:16.566 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-28T05:12:16.566 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T05:12:16.599 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-28T05:12:16.599 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T05:12:16.599 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T05:12:16.630 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T05:12:16.700 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-28T05:12:16.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T05:12:16.731 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T05:12:16.802 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-28T05:12:16.802 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T05:12:16.832 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T05:12:16.901 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-28T05:12:16.901 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T05:12:16.901 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T05:12:16.933 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T05:12:17.004 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-28T05:12:17.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T05:12:17.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T05:12:17.105 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-28T05:12:17.105 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T05:12:17.136 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T05:12:17.205 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-28T05:12:17.205 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T05:12:17.205 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T05:12:17.236 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T05:12:17.306 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-28T05:12:17.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T05:12:17.338 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T05:12:17.408 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-28T05:12:17.408 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T05:12:17.436 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T05:12:17.506 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-28T05:12:17.601 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 1, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'subvols': {'create': 2}} 2024-03-28T05:12:17.601 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T05:12:17.602 INFO:tasks.cephadm:Cluster fsid is bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:12:17.602 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-28T05:12:17.602 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.23', 'mon.b': '172.21.15.98', 'mon.c': '172.21.15.113'} 2024-03-28T05:12:17.602 INFO:tasks.cephadm:First mon is mon.a on smithi023 2024-03-28T05:12:17.602 INFO:tasks.cephadm:First mgr is x 2024-03-28T05:12:17.602 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-28T05:12:17.602 DEBUG:teuthology.orchestra.run.smithi023:> sudo hostname $(hostname -s) 2024-03-28T05:12:17.634 DEBUG:teuthology.orchestra.run.smithi098:> sudo hostname $(hostname -s) 2024-03-28T05:12:17.665 DEBUG:teuthology.orchestra.run.smithi113:> sudo hostname $(hostname -s) 2024-03-28T05:12:17.693 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-28T05:12:17.693 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T05:12:17.835 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77877/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2145-g3be9a5c9', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2145.g3be9a5c9'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-22 05:51:42.267384', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-mchangir-testing-main-20240318.032620', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-28T05:12:17.974 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-mchangir-testing-main-20240318.032620, sha1 3be9a5c9bc793e11e2800a8c0c696e8b46742033 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T05:12:17.975 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm 2024-03-28T05:12:17.976 DEBUG:teuthology.orchestra.run.smithi023:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T05:12:18.634 INFO:teuthology.orchestra.run.smithi023.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 05:12 /home/ubuntu/cephtest/cephadm 2024-03-28T05:12:18.634 DEBUG:teuthology.orchestra.run.smithi098:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T05:12:19.112 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 05:12 /home/ubuntu/cephtest/cephadm 2024-03-28T05:12:19.112 DEBUG:teuthology.orchestra.run.smithi113:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T05:12:19.720 INFO:teuthology.orchestra.run.smithi113.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 05:12 /home/ubuntu/cephtest/cephadm 2024-03-28T05:12:19.720 DEBUG:teuthology.orchestra.run.smithi023:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T05:12:19.739 DEBUG:teuthology.orchestra.run.smithi098:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T05:12:19.758 DEBUG:teuthology.orchestra.run.smithi113:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T05:12:19.792 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 on all hosts... 2024-03-28T05:12:19.792 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T05:12:19.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T05:12:19.800 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T05:12:20.579 INFO:teuthology.orchestra.run.smithi113.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T05:12:20.612 INFO:teuthology.orchestra.run.smithi023.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T05:12:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T05:12:52.930 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-28T05:12:52.931 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T05:12:52.931 INFO:teuthology.orchestra.run.smithi098.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T05:12:52.931 INFO:teuthology.orchestra.run.smithi098.stdout: "repo_digests": [ 2024-03-28T05:12:52.931 INFO:teuthology.orchestra.run.smithi098.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T05:12:52.931 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-28T05:12:52.931 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-28T05:13:03.843 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-28T05:13:03.843 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T05:13:03.843 INFO:teuthology.orchestra.run.smithi113.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T05:13:03.844 INFO:teuthology.orchestra.run.smithi113.stdout: "repo_digests": [ 2024-03-28T05:13:03.844 INFO:teuthology.orchestra.run.smithi113.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T05:13:03.844 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-28T05:13:03.844 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-28T05:13:04.197 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T05:13:04.197 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T05:13:04.197 INFO:teuthology.orchestra.run.smithi023.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T05:13:04.197 INFO:teuthology.orchestra.run.smithi023.stdout: "repo_digests": [ 2024-03-28T05:13:04.197 INFO:teuthology.orchestra.run.smithi023.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T05:13:04.198 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T05:13:04.198 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T05:13:04.221 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /etc/ceph 2024-03-28T05:13:04.252 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph 2024-03-28T05:13:04.287 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /etc/ceph 2024-03-28T05:13:04.316 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 777 /etc/ceph 2024-03-28T05:13:04.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /etc/ceph 2024-03-28T05:13:04.372 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 777 /etc/ceph 2024-03-28T05:13:04.399 INFO:tasks.cephadm:Writing seed config... 2024-03-28T05:13:04.400 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-28T05:13:04.400 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-28T05:13:04.400 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-28T05:13:04.400 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-28T05:13:04.400 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-28T05:13:04.400 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-28T05:13:04.400 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-28T05:13:04.400 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-28T05:13:04.400 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-28T05:13:04.400 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-28T05:13:04.401 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-28T05:13:04.401 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-28T05:13:04.401 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-28T05:13:04.401 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-28T05:13:04.401 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-28T05:13:04.401 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-28T05:13:04.401 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-28T05:13:04.401 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-28T05:13:04.401 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-28T05:13:04.401 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-28T05:13:04.401 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-28T05:13:04.401 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-28T05:13:04.401 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-28T05:13:04.401 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-28T05:13:04.401 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-28T05:13:04.401 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-28T05:13:04.401 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-28T05:13:04.402 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-28T05:13:04.402 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-28T05:13:04.402 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-28T05:13:04.402 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-28T05:13:04.402 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-28T05:13:04.402 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-28T05:13:04.402 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-28T05:13:04.402 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-28T05:13:04.402 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-28T05:13:04.402 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-28T05:13:04.402 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-28T05:13:04.402 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-28T05:13:04.402 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-28T05:13:04.402 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-28T05:13:04.402 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-28T05:13:04.402 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-28T05:13:04.403 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T05:13:04.403 DEBUG:teuthology.orchestra.run.smithi023:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-28T05:13:04.417 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = bf37ebc4-ecc1-11ee-95d0-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-28T05:13:04.418 DEBUG:teuthology.orchestra.run.smithi023:mon.a> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mon.a.service 2024-03-28T05:13:04.460 DEBUG:teuthology.orchestra.run.smithi023:mgr.x> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mgr.x.service 2024-03-28T05:13:04.503 INFO:tasks.cephadm:Bootstrapping... 2024-03-28T05:13:04.503 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 -v bootstrap --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.23 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-28T05:13:04.700 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2024-03-28T05:13:04.700 INFO:teuthology.orchestra.run.smithi023.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033', '-v', 'bootstrap', '--fsid', 'bf37ebc4-ecc1-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.23', '--skip-admin-label'] 2024-03-28T05:13:04.724 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T05:13:04.725 INFO:teuthology.orchestra.run.smithi023.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-28T05:13:04.725 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying podman|docker is present... 2024-03-28T05:13:04.747 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T05:13:04.748 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying lvm2 is present... 2024-03-28T05:13:04.748 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying time synchronization is in place... 2024-03-28T05:13:04.755 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-28T05:13:04.755 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-28T05:13:04.761 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-28T05:13:04.761 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-28T05:13:04.767 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout enabled 2024-03-28T05:13:04.773 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout active 2024-03-28T05:13:04.773 INFO:teuthology.orchestra.run.smithi023.stdout:Unit chronyd.service is enabled and running 2024-03-28T05:13:04.773 INFO:teuthology.orchestra.run.smithi023.stdout:Repeating the final host check... 2024-03-28T05:13:04.795 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T05:13:04.795 INFO:teuthology.orchestra.run.smithi023.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-28T05:13:04.795 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl is present 2024-03-28T05:13:04.796 INFO:teuthology.orchestra.run.smithi023.stdout:lvcreate is present 2024-03-28T05:13:04.802 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-28T05:13:04.802 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-28T05:13:04.807 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-28T05:13:04.808 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-28T05:13:04.814 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout enabled 2024-03-28T05:13:04.820 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout active 2024-03-28T05:13:04.820 INFO:teuthology.orchestra.run.smithi023.stdout:Unit chronyd.service is enabled and running 2024-03-28T05:13:04.820 INFO:teuthology.orchestra.run.smithi023.stdout:Host looks OK 2024-03-28T05:13:04.820 INFO:teuthology.orchestra.run.smithi023.stdout:Cluster fsid: bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:13:04.820 INFO:teuthology.orchestra.run.smithi023.stdout:Acquiring lock 140415263997376 on /run/cephadm/bf37ebc4-ecc1-11ee-95d0-87774f69a715.lock 2024-03-28T05:13:04.820 INFO:teuthology.orchestra.run.smithi023.stdout:Lock 140415263997376 acquired on /run/cephadm/bf37ebc4-ecc1-11ee-95d0-87774f69a715.lock 2024-03-28T05:13:04.821 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying IP 172.21.15.23 port 3300 ... 2024-03-28T05:13:04.821 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying IP 172.21.15.23 port 6789 ... 2024-03-28T05:13:04.821 INFO:teuthology.orchestra.run.smithi023.stdout:Base mon IP(s) is [172.21.15.23:3300, 172.21.15.23:6789], mon addrv is [v2:172.21.15.23:3300,v1:172.21.15.23:6789] 2024-03-28T05:13:04.824 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.23 metric 100 2024-03-28T05:13:04.824 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.23 metric 100 2024-03-28T05:13:04.827 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-28T05:13:04.827 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-28T05:13:04.827 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-28T05:13:04.829 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-28T05:13:04.829 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-28T05:13:04.829 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-28T05:13:04.829 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-28T05:13:04.829 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe59:ce42/64 scope link noprefixroute 2024-03-28T05:13:04.829 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-28T05:13:04.830 INFO:teuthology.orchestra.run.smithi023.stdout:Mon IP `172.21.15.23` is in CIDR network `172.21.0.0/20` 2024-03-28T05:13:04.830 INFO:teuthology.orchestra.run.smithi023.stdout:Mon IP `172.21.15.23` is in CIDR network `172.21.0.0/20` 2024-03-28T05:13:04.830 INFO:teuthology.orchestra.run.smithi023.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-28T05:13:04.830 INFO:teuthology.orchestra.run.smithi023.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-28T05:13:04.833 INFO:teuthology.orchestra.run.smithi023.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T05:13:05.300 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-28T05:13:05.301 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T05:13:05.301 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Getting image source signatures 2024-03-28T05:13:05.301 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Copying blob sha256:4dd83c6ee8921fe17396d77c31e205985ae94c87d3f752465b5f04aaf9ae340a 2024-03-28T05:13:05.301 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-28T05:13:05.301 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Copying config sha256:c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-28T05:13:05.301 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-28T05:13:06.188 INFO:teuthology.orchestra.run.smithi023.stdout:ceph: stdout ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T05:13:06.188 INFO:teuthology.orchestra.run.smithi023.stdout:Ceph version: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T05:13:06.188 INFO:teuthology.orchestra.run.smithi023.stdout:Extracting ceph user uid/gid from container image... 2024-03-28T05:13:07.085 INFO:teuthology.orchestra.run.smithi023.stdout:stat: stdout 167 167 2024-03-28T05:13:07.085 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial keys... 2024-03-28T05:13:07.957 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQBj/ARmgffJGhAAY7vh+zz0bxgOled/BH10GA== 2024-03-28T05:13:08.871 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQBk/ARmbe3JERAAANbqcyDcBcDrDpTV/E5w1w== 2024-03-28T05:13:09.843 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQBl/ARmNLqNChAAFVj2Y0H7cOh79j+ekwFY7A== 2024-03-28T05:13:09.844 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial monmap... 2024-03-28T05:13:10.858 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-28T05:13:10.858 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-28T05:13:10.858 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:13:10.858 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-28T05:13:10.858 INFO:teuthology.orchestra.run.smithi023.stdout:monmaptool for a [v2:172.21.15.23:3300,v1:172.21.15.23:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-28T05:13:10.858 INFO:teuthology.orchestra.run.smithi023.stdout:setting min_mon_release = quincy 2024-03-28T05:13:10.858 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: set fsid to bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:13:10.858 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-28T05:13:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:13:10.859 INFO:teuthology.orchestra.run.smithi023.stdout:Creating mon... 2024-03-28T05:13:11.831 INFO:teuthology.orchestra.run.smithi023.stdout:create mon.a on 2024-03-28T05:13:12.040 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-28T05:13:12.196 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-28T05:13:12.515 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715.target. 2024-03-28T05:13:12.515 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715.target. 2024-03-28T05:13:12.749 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mon.a 2024-03-28T05:13:12.749 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to reset failed state of unit ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-28T05:13:12.906 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715.target.wants/ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@.service. 2024-03-28T05:13:13.486 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T05:13:13.487 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-28T05:13:13.493 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T05:13:13.493 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-28T05:13:13.493 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-28T05:13:13.493 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to enable service . firewalld.service is not available 2024-03-28T05:13:13.494 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mon to start... 2024-03-28T05:13:13.494 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mon... 2024-03-28T05:13:13.617 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:13 smithi023 systemd[1]: Started Ceph mon.a for bf37ebc4-ecc1-11ee-95d0-87774f69a715. 2024-03-28T05:13:14.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:14 smithi023 ceph-mon[31582]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T05:13:14.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:14 smithi023 ceph-mon[31582]: monmap e1: 1 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T05:13:14.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:14 smithi023 ceph-mon[31582]: fsmap 2024-03-28T05:13:14.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:14 smithi023 ceph-mon[31582]: osdmap e1: 0 total, 0 up, 0 in 2024-03-28T05:13:14.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:14 smithi023 ceph-mon[31582]: mgrmap e1: no daemons active 2024-03-28T05:13:14.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:14 smithi023 ceph-mon[31582]: from='client.? 172.21.15.23:0/1500109207' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-28T05:13:14.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout cluster: 2024-03-28T05:13:14.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout id: bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:13:14.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-28T05:13:14.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T05:13:14.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout services: 2024-03-28T05:13:14.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.532778s) 2024-03-28T05:13:14.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-28T05:13:14.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-28T05:13:14.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T05:13:14.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout data: 2024-03-28T05:13:14.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-28T05:13:14.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-28T05:13:14.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-28T05:13:14.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout pgs: 2024-03-28T05:13:14.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T05:13:14.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.905+0000 7f2d17535700 1 Processor -- start 2024-03-28T05:13:14.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.905+0000 7f2d17535700 1 -- start start 2024-03-28T05:13:14.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.906+0000 7f2d17535700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d10107a20 0x7f2d10107e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:14.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.906+0000 7f2d17535700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d10108410 con 0x7f2d10107a20 2024-03-28T05:13:14.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.906+0000 7f2d152d1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d10107a20 0x7f2d10107e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:14.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.906+0000 7f2d152d1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d10107a20 0x7f2d10107e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57420/0 (socket says 172.21.15.23:57420) 2024-03-28T05:13:14.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.906+0000 7f2d152d1700 1 -- 172.21.15.23:0/3750860933 learned_addr learned my addr 172.21.15.23:0/3750860933 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:14.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.906+0000 7f2d152d1700 1 -- 172.21.15.23:0/3750860933 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d10108c20 con 0x7f2d10107a20 2024-03-28T05:13:14.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.906+0000 7f2d152d1700 1 --2- 172.21.15.23:0/3750860933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d10107a20 0x7f2d10107e40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2d080098a0 tx=0x7f2d08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=941e27c1bf12278e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:14.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.907+0000 7f2d0ffff700 1 -- 172.21.15.23:0/3750860933 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d08004160 con 0x7f2d10107a20 2024-03-28T05:13:14.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.907+0000 7f2d0ffff700 1 -- 172.21.15.23:0/3750860933 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d08004300 con 0x7f2d10107a20 2024-03-28T05:13:14.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.907+0000 7f2d0ffff700 1 -- 172.21.15.23:0/3750860933 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d080036a0 con 0x7f2d10107a20 2024-03-28T05:13:14.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.907+0000 7f2d17535700 1 -- 172.21.15.23:0/3750860933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d10107a20 msgr2=0x7f2d10107e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:14.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.907+0000 7f2d17535700 1 --2- 172.21.15.23:0/3750860933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d10107a20 0x7f2d10107e40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2d080098a0 tx=0x7f2d08009bb0 comp rx=0 tx=0).stop 2024-03-28T05:13:14.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.907+0000 7f2d17535700 1 -- 172.21.15.23:0/3750860933 shutdown_connections 2024-03-28T05:13:14.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.907+0000 7f2d17535700 1 --2- 172.21.15.23:0/3750860933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d10107a20 0x7f2d10107e40 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:14.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.907+0000 7f2d17535700 1 -- 172.21.15.23:0/3750860933 >> 172.21.15.23:0/3750860933 conn(0x7f2d10103040 msgr2=0x7f2d10105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:14.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.907+0000 7f2d17535700 1 -- 172.21.15.23:0/3750860933 shutdown_connections 2024-03-28T05:13:14.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.907+0000 7f2d17535700 1 -- 172.21.15.23:0/3750860933 wait complete. 2024-03-28T05:13:14.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.908+0000 7f2d17535700 1 Processor -- start 2024-03-28T05:13:14.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.908+0000 7f2d17535700 1 -- start start 2024-03-28T05:13:14.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.908+0000 7f2d17535700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d10107a20 0x7f2d1007cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:14.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.908+0000 7f2d17535700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d10109150 con 0x7f2d10107a20 2024-03-28T05:13:14.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.908+0000 7f2d152d1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d10107a20 0x7f2d1007cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:14.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.908+0000 7f2d152d1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d10107a20 0x7f2d1007cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57426/0 (socket says 172.21.15.23:57426) 2024-03-28T05:13:14.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.908+0000 7f2d152d1700 1 -- 172.21.15.23:0/1500109207 learned_addr learned my addr 172.21.15.23:0/1500109207 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:14.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.909+0000 7f2d152d1700 1 -- 172.21.15.23:0/1500109207 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d1007d0d0 con 0x7f2d10107a20 2024-03-28T05:13:14.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.909+0000 7f2d152d1700 1 --2- 172.21.15.23:0/1500109207 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d10107a20 0x7f2d1007cb90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2d08000c00 tx=0x7f2d08004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:14.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.909+0000 7f2d0e7fc700 1 -- 172.21.15.23:0/1500109207 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d08003db0 con 0x7f2d10107a20 2024-03-28T05:13:14.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.909+0000 7f2d0e7fc700 1 -- 172.21.15.23:0/1500109207 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d08011630 con 0x7f2d10107a20 2024-03-28T05:13:14.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.909+0000 7f2d17535700 1 -- 172.21.15.23:0/1500109207 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d1007b2c0 con 0x7f2d10107a20 2024-03-28T05:13:14.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.909+0000 7f2d0e7fc700 1 -- 172.21.15.23:0/1500109207 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d08011950 con 0x7f2d10107a20 2024-03-28T05:13:14.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.909+0000 7f2d17535700 1 -- 172.21.15.23:0/1500109207 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d1007b800 con 0x7f2d10107a20 2024-03-28T05:13:14.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.909+0000 7f2d0e7fc700 1 -- 172.21.15.23:0/1500109207 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f2d08011af0 con 0x7f2d10107a20 2024-03-28T05:13:14.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.909+0000 7f2d0e7fc700 1 -- 172.21.15.23:0/1500109207 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2d0801c450 con 0x7f2d10107a20 2024-03-28T05:13:14.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.910+0000 7f2d17535700 1 -- 172.21.15.23:0/1500109207 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d10069220 con 0x7f2d10107a20 2024-03-28T05:13:14.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.912+0000 7f2d0e7fc700 1 -- 172.21.15.23:0/1500109207 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f2d08004260 con 0x7f2d10107a20 2024-03-28T05:13:14.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.985+0000 7f2d17535700 1 -- 172.21.15.23:0/1500109207 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f2d10194990 con 0x7f2d10107a20 2024-03-28T05:13:14.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.985+0000 7f2d0e7fc700 1 -- 172.21.15.23:0/1500109207 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f2d0801b450 con 0x7f2d10107a20 2024-03-28T05:13:14.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.985+0000 7f2d17535700 1 -- 172.21.15.23:0/1500109207 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d10107a20 msgr2=0x7f2d1007cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:14.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.985+0000 7f2d17535700 1 --2- 172.21.15.23:0/1500109207 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d10107a20 0x7f2d1007cb90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2d08000c00 tx=0x7f2d08004750 comp rx=0 tx=0).stop 2024-03-28T05:13:14.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.986+0000 7f2d17535700 1 -- 172.21.15.23:0/1500109207 shutdown_connections 2024-03-28T05:13:14.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.986+0000 7f2d17535700 1 --2- 172.21.15.23:0/1500109207 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d10107a20 0x7f2d1007cb90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:14.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.986+0000 7f2d17535700 1 -- 172.21.15.23:0/1500109207 >> 172.21.15.23:0/1500109207 conn(0x7f2d10103040 msgr2=0x7f2d10191dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:14.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.986+0000 7f2d17535700 1 -- 172.21.15.23:0/1500109207 shutdown_connections 2024-03-28T05:13:14.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:13.986+0000 7f2d17535700 1 -- 172.21.15.23:0/1500109207 wait complete. 2024-03-28T05:13:14.588 INFO:teuthology.orchestra.run.smithi023.stdout:mon is available 2024-03-28T05:13:14.588 INFO:teuthology.orchestra.run.smithi023.stdout:Assimilating anything we can from ceph.conf... 2024-03-28T05:13:15.763 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:15 smithi023 ceph-mon[31582]: from='client.? 172.21.15.23:0/1183518603' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-28T05:13:15.763 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:15 smithi023 ceph-mon[31582]: from='client.? 172.21.15.23:0/1183518603' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-28T05:13:15.773 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T05:13:15.773 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [global] 2024-03-28T05:13:15.773 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout fsid = bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:13:15.773 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.23:3300,v1:172.21.15.23:6789] 2024-03-28T05:13:15.773 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-28T05:13:15.773 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-28T05:13:15.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-28T05:13:15.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-28T05:13:15.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T05:13:15.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-28T05:13:15.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-28T05:13:15.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T05:13:15.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [osd] 2024-03-28T05:13:15.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-28T05:13:15.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-28T05:13:15.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-28T05:13:15.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-28T05:13:15.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-28T05:13:15.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.992+0000 7fa5bfb1c700 1 Processor -- start 2024-03-28T05:13:15.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.992+0000 7fa5bfb1c700 1 -- start start 2024-03-28T05:13:15.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.992+0000 7fa5bfb1c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b8107ac0 0x7fa5b8107ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:15.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.992+0000 7fa5bfb1c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b81084b0 con 0x7fa5b8107ac0 2024-03-28T05:13:15.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.992+0000 7fa5bd8b8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b8107ac0 0x7fa5b8107ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:15.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.992+0000 7fa5bd8b8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b8107ac0 0x7fa5b8107ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57434/0 (socket says 172.21.15.23:57434) 2024-03-28T05:13:15.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.992+0000 7fa5bd8b8700 1 -- 172.21.15.23:0/35709693 learned_addr learned my addr 172.21.15.23:0/35709693 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:15.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.993+0000 7fa5bd8b8700 1 -- 172.21.15.23:0/35709693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b8108c70 con 0x7fa5b8107ac0 2024-03-28T05:13:15.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.993+0000 7fa5bd8b8700 1 --2- 172.21.15.23:0/35709693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b8107ac0 0x7fa5b8107ee0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa5b40098a0 tx=0x7fa5b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37e90d766b424467 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:15.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.993+0000 7fa5bc8b6700 1 -- 172.21.15.23:0/35709693 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5b4004160 con 0x7fa5b8107ac0 2024-03-28T05:13:15.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.993+0000 7fa5bc8b6700 1 -- 172.21.15.23:0/35709693 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b4004300 con 0x7fa5b8107ac0 2024-03-28T05:13:15.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.993+0000 7fa5bc8b6700 1 -- 172.21.15.23:0/35709693 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5b40036a0 con 0x7fa5b8107ac0 2024-03-28T05:13:15.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.993+0000 7fa5bfb1c700 1 -- 172.21.15.23:0/35709693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b8107ac0 msgr2=0x7fa5b8107ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:15.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.993+0000 7fa5bfb1c700 1 --2- 172.21.15.23:0/35709693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b8107ac0 0x7fa5b8107ee0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa5b40098a0 tx=0x7fa5b4009bb0 comp rx=0 tx=0).stop 2024-03-28T05:13:15.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.993+0000 7fa5bfb1c700 1 -- 172.21.15.23:0/35709693 shutdown_connections 2024-03-28T05:13:15.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.993+0000 7fa5bfb1c700 1 --2- 172.21.15.23:0/35709693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b8107ac0 0x7fa5b8107ee0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:15.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.993+0000 7fa5bfb1c700 1 -- 172.21.15.23:0/35709693 >> 172.21.15.23:0/35709693 conn(0x7fa5b81030c0 msgr2=0x7fa5b81054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:15.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.994+0000 7fa5bfb1c700 1 -- 172.21.15.23:0/35709693 shutdown_connections 2024-03-28T05:13:15.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.994+0000 7fa5bfb1c700 1 -- 172.21.15.23:0/35709693 wait complete. 2024-03-28T05:13:15.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.994+0000 7fa5bfb1c700 1 Processor -- start 2024-03-28T05:13:15.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.994+0000 7fa5bfb1c700 1 -- start start 2024-03-28T05:13:15.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.994+0000 7fa5bfb1c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b8107ac0 0x7fa5b819d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:15.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.994+0000 7fa5bfb1c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b81091a0 con 0x7fa5b8107ac0 2024-03-28T05:13:15.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.994+0000 7fa5bd8b8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b8107ac0 0x7fa5b819d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:15.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.994+0000 7fa5bd8b8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b8107ac0 0x7fa5b819d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57446/0 (socket says 172.21.15.23:57446) 2024-03-28T05:13:15.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.994+0000 7fa5bd8b8700 1 -- 172.21.15.23:0/1183518603 learned_addr learned my addr 172.21.15.23:0/1183518603 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:15.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.995+0000 7fa5bd8b8700 1 -- 172.21.15.23:0/1183518603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b819d920 con 0x7fa5b8107ac0 2024-03-28T05:13:15.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.995+0000 7fa5bd8b8700 1 --2- 172.21.15.23:0/1183518603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b8107ac0 0x7fa5b819d3e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4000c00 tx=0x7fa5b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:15.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.995+0000 7fa5aeffd700 1 -- 172.21.15.23:0/1183518603 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5b4003db0 con 0x7fa5b8107ac0 2024-03-28T05:13:15.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.995+0000 7fa5aeffd700 1 -- 172.21.15.23:0/1183518603 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b4011630 con 0x7fa5b8107ac0 2024-03-28T05:13:15.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.995+0000 7fa5aeffd700 1 -- 172.21.15.23:0/1183518603 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5b4011950 con 0x7fa5b8107ac0 2024-03-28T05:13:15.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.995+0000 7fa5bfb1c700 1 -- 172.21.15.23:0/1183518603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b819dbb0 con 0x7fa5b8107ac0 2024-03-28T05:13:15.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.995+0000 7fa5bfb1c700 1 -- 172.21.15.23:0/1183518603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b81a0850 con 0x7fa5b8107ac0 2024-03-28T05:13:15.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.995+0000 7fa5aeffd700 1 -- 172.21.15.23:0/1183518603 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fa5b401a460 con 0x7fa5b8107ac0 2024-03-28T05:13:15.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.995+0000 7fa5aeffd700 1 -- 172.21.15.23:0/1183518603 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa5b401b780 con 0x7fa5b8107ac0 2024-03-28T05:13:15.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.996+0000 7fa5bfb1c700 1 -- 172.21.15.23:0/1183518603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b8069220 con 0x7fa5b8107ac0 2024-03-28T05:13:15.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:14.998+0000 7fa5aeffd700 1 -- 172.21.15.23:0/1183518603 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fa5b401b980 con 0x7fa5b8107ac0 2024-03-28T05:13:15.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:15.068+0000 7fa5bfb1c700 1 -- 172.21.15.23:0/1183518603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fa5b819e090 con 0x7fa5b8107ac0 2024-03-28T05:13:15.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:15.072+0000 7fa5aeffd700 1 -- 172.21.15.23:0/1183518603 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fa5b819e090 con 0x7fa5b8107ac0 2024-03-28T05:13:15.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:15.072+0000 7fa5aeffd700 1 -- 172.21.15.23:0/1183518603 <== mon.0 v2:172.21.15.23:3300/0 8 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa5b401ada0 con 0x7fa5b8107ac0 2024-03-28T05:13:15.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:15.072+0000 7fa5bfb1c700 1 -- 172.21.15.23:0/1183518603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b8107ac0 msgr2=0x7fa5b819d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:15.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:15.072+0000 7fa5bfb1c700 1 --2- 172.21.15.23:0/1183518603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b8107ac0 0x7fa5b819d3e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4000c00 tx=0x7fa5b4004750 comp rx=0 tx=0).stop 2024-03-28T05:13:15.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:15.072+0000 7fa5bfb1c700 1 -- 172.21.15.23:0/1183518603 shutdown_connections 2024-03-28T05:13:15.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:15.072+0000 7fa5bfb1c700 1 --2- 172.21.15.23:0/1183518603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b8107ac0 0x7fa5b819d3e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:15.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:15.072+0000 7fa5bfb1c700 1 -- 172.21.15.23:0/1183518603 >> 172.21.15.23:0/1183518603 conn(0x7fa5b81030c0 msgr2=0x7fa5b807c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:15.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:15.072+0000 7fa5bfb1c700 1 -- 172.21.15.23:0/1183518603 shutdown_connections 2024-03-28T05:13:15.778 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:15.072+0000 7fa5bfb1c700 1 -- 172.21.15.23:0/1183518603 wait complete. 2024-03-28T05:13:15.778 INFO:teuthology.orchestra.run.smithi023.stdout:Generating new minimal ceph.conf... 2024-03-28T05:13:16.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:16 smithi023 ceph-mon[31582]: from='client.? 172.21.15.23:0/850694072' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:13:16.902 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.225+0000 7fe27f733700 1 Processor -- start 2024-03-28T05:13:16.902 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.225+0000 7fe27f733700 1 -- start start 2024-03-28T05:13:16.902 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.225+0000 7fe27f733700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe278107a20 0x7fe278107e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:16.903 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.225+0000 7fe27f733700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe278108410 con 0x7fe278107a20 2024-03-28T05:13:16.903 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.225+0000 7fe27d4cf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe278107a20 0x7fe278107e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:16.903 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.225+0000 7fe27d4cf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe278107a20 0x7fe278107e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57452/0 (socket says 172.21.15.23:57452) 2024-03-28T05:13:16.903 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.225+0000 7fe27d4cf700 1 -- 172.21.15.23:0/3720477068 learned_addr learned my addr 172.21.15.23:0/3720477068 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:16.903 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.225+0000 7fe27d4cf700 1 -- 172.21.15.23:0/3720477068 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe278108c20 con 0x7fe278107a20 2024-03-28T05:13:16.903 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.226+0000 7fe27d4cf700 1 --2- 172.21.15.23:0/3720477068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe278107a20 0x7fe278107e40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe26c0098a0 tx=0x7fe26c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fef72cfdaf7cd317 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:16.903 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.226+0000 7fe26bfff700 1 -- 172.21.15.23:0/3720477068 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe26c004160 con 0x7fe278107a20 2024-03-28T05:13:16.903 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.226+0000 7fe26bfff700 1 -- 172.21.15.23:0/3720477068 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe26c004300 con 0x7fe278107a20 2024-03-28T05:13:16.903 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.226+0000 7fe26bfff700 1 -- 172.21.15.23:0/3720477068 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe26c011b50 con 0x7fe278107a20 2024-03-28T05:13:16.903 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.226+0000 7fe27f733700 1 -- 172.21.15.23:0/3720477068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe278107a20 msgr2=0x7fe278107e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:16.903 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.226+0000 7fe27f733700 1 --2- 172.21.15.23:0/3720477068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe278107a20 0x7fe278107e40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe26c0098a0 tx=0x7fe26c009bb0 comp rx=0 tx=0).stop 2024-03-28T05:13:16.903 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.226+0000 7fe27f733700 1 -- 172.21.15.23:0/3720477068 shutdown_connections 2024-03-28T05:13:16.903 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.226+0000 7fe27f733700 1 --2- 172.21.15.23:0/3720477068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe278107a20 0x7fe278107e40 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:16.903 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.226+0000 7fe27f733700 1 -- 172.21.15.23:0/3720477068 >> 172.21.15.23:0/3720477068 conn(0x7fe278103040 msgr2=0x7fe278105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:16.903 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.226+0000 7fe27f733700 1 -- 172.21.15.23:0/3720477068 shutdown_connections 2024-03-28T05:13:16.903 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.227+0000 7fe27f733700 1 -- 172.21.15.23:0/3720477068 wait complete. 2024-03-28T05:13:16.904 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.227+0000 7fe27f733700 1 Processor -- start 2024-03-28T05:13:16.904 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.227+0000 7fe27f733700 1 -- start start 2024-03-28T05:13:16.904 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.227+0000 7fe27f733700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe278107a20 0x7fe27807cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:16.904 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.227+0000 7fe27f733700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27810bfd0 con 0x7fe278107a20 2024-03-28T05:13:16.904 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.227+0000 7fe27d4cf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe278107a20 0x7fe27807cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:16.904 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.227+0000 7fe27d4cf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe278107a20 0x7fe27807cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57454/0 (socket says 172.21.15.23:57454) 2024-03-28T05:13:16.904 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.227+0000 7fe27d4cf700 1 -- 172.21.15.23:0/850694072 learned_addr learned my addr 172.21.15.23:0/850694072 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:16.904 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.228+0000 7fe27d4cf700 1 -- 172.21.15.23:0/850694072 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe27807d0d0 con 0x7fe278107a20 2024-03-28T05:13:16.904 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.228+0000 7fe27d4cf700 1 --2- 172.21.15.23:0/850694072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe278107a20 0x7fe27807cb90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe26c009fb0 tx=0x7fe26c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:16.904 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.228+0000 7fe26a7fc700 1 -- 172.21.15.23:0/850694072 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe26c0038b0 con 0x7fe278107a20 2024-03-28T05:13:16.904 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.228+0000 7fe26a7fc700 1 -- 172.21.15.23:0/850694072 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe26c011e50 con 0x7fe278107a20 2024-03-28T05:13:16.904 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.228+0000 7fe27f733700 1 -- 172.21.15.23:0/850694072 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe27807b2c0 con 0x7fe278107a20 2024-03-28T05:13:16.904 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.228+0000 7fe26a7fc700 1 -- 172.21.15.23:0/850694072 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe26c019550 con 0x7fe278107a20 2024-03-28T05:13:16.904 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.228+0000 7fe27f733700 1 -- 172.21.15.23:0/850694072 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe27807b800 con 0x7fe278107a20 2024-03-28T05:13:16.904 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.228+0000 7fe26a7fc700 1 -- 172.21.15.23:0/850694072 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe26c019b90 con 0x7fe278107a20 2024-03-28T05:13:16.904 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.228+0000 7fe26a7fc700 1 -- 172.21.15.23:0/850694072 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe26c024730 con 0x7fe278107a20 2024-03-28T05:13:16.904 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.229+0000 7fe27f733700 1 -- 172.21.15.23:0/850694072 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe278069220 con 0x7fe278107a20 2024-03-28T05:13:16.905 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.231+0000 7fe26a7fc700 1 -- 172.21.15.23:0/850694072 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe26c003a50 con 0x7fe278107a20 2024-03-28T05:13:16.905 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.302+0000 7fe27f733700 1 -- 172.21.15.23:0/850694072 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fe27807bd30 con 0x7fe278107a20 2024-03-28T05:13:16.905 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.302+0000 7fe26a7fc700 1 -- 172.21.15.23:0/850694072 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fe26c024930 con 0x7fe278107a20 2024-03-28T05:13:16.905 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.303+0000 7fe27f733700 1 -- 172.21.15.23:0/850694072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe278107a20 msgr2=0x7fe27807cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:16.905 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.303+0000 7fe27f733700 1 --2- 172.21.15.23:0/850694072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe278107a20 0x7fe27807cb90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe26c009fb0 tx=0x7fe26c011bf0 comp rx=0 tx=0).stop 2024-03-28T05:13:16.905 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.303+0000 7fe27f733700 1 -- 172.21.15.23:0/850694072 shutdown_connections 2024-03-28T05:13:16.905 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.303+0000 7fe27f733700 1 --2- 172.21.15.23:0/850694072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe278107a20 0x7fe27807cb90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:16.905 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.303+0000 7fe27f733700 1 -- 172.21.15.23:0/850694072 >> 172.21.15.23:0/850694072 conn(0x7fe278103040 msgr2=0x7fe278103bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:16.905 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.303+0000 7fe27f733700 1 -- 172.21.15.23:0/850694072 shutdown_connections 2024-03-28T05:13:16.905 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:16.303+0000 7fe27f733700 1 -- 172.21.15.23:0/850694072 wait complete. 2024-03-28T05:13:16.905 INFO:teuthology.orchestra.run.smithi023.stdout:Restarting the monitor... 2024-03-28T05:13:17.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:16 smithi023 systemd[1]: Stopping Ceph mon.a for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:13:17.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:17 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-a[31577]: 2024-03-28T05:13:17.178+0000 7f50ca8b7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:13:17.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:17 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-a[31577]: 2024-03-28T05:13:17.178+0000 7f50ca8b7700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-28T05:13:17.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:17 smithi023 podman[31825]: 2024-03-28 05:13:17.301027285 +0000 UTC m=+0.230118547 container died 07544db6c5c5b8291410f893b03ff0ef050b7cf253c2ee4f0d066bbd05d81713 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-a, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-28T05:13:17.824 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:17 smithi023 podman[31825]: 2024-03-28 05:13:17.644771352 +0000 UTC m=+0.573862612 container cleanup 07544db6c5c5b8291410f893b03ff0ef050b7cf253c2ee4f0d066bbd05d81713 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-a, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-28T05:13:17.824 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:17 smithi023 bash[31825]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-a 2024-03-28T05:13:18.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:17 smithi023 podman[31839]: 2024-03-28 05:13:17.829027703 +0000 UTC m=+0.525896092 container remove 07544db6c5c5b8291410f893b03ff0ef050b7cf253c2ee4f0d066bbd05d81713 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-a, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-28T05:13:18.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 systemd[1]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-28T05:13:18.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 systemd[1]: Stopped Ceph mon.a for bf37ebc4-ecc1-11ee-95d0-87774f69a715. 2024-03-28T05:13:18.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 systemd[1]: Starting Ceph mon.a for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:13:18.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 podman[31935]: 2024-03-28 05:13:18.179251484 +0000 UTC m=+0.017614198 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T05:13:18.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 podman[31935]: 2024-03-28 05:13:18.279820308 +0000 UTC m=+0.118182992 container create 6ba187531ba0f3ed1e1e055f2c7ee8cb929c8df2c5d692aa1be4443803d55012 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-a, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-28T05:13:18.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 podman[31935]: 2024-03-28 05:13:18.474215226 +0000 UTC m=+0.312577911 container init 6ba187531ba0f3ed1e1e055f2c7ee8cb929c8df2c5d692aa1be4443803d55012 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319) 2024-03-28T05:13:18.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 podman[31935]: 2024-03-28 05:13:18.477151292 +0000 UTC m=+0.315513970 container start 6ba187531ba0f3ed1e1e055f2c7ee8cb929c8df2c5d692aa1be4443803d55012 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-a, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-28T05:13:18.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: set uid:gid to 167:167 (ceph:ceph) 2024-03-28T05:13:18.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev), process ceph-mon, pid 2 2024-03-28T05:13:18.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: pidfile_write: ignore empty --pid-file 2024-03-28T05:13:18.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: load: jerasure load: lrc 2024-03-28T05:13:18.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: RocksDB version: 7.9.2 2024-03-28T05:13:18.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Git sha 0 2024-03-28T05:13:18.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-28T05:13:18.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: DB SUMMARY 2024-03-28T05:13:18.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: DB Session ID: 2W7AI9GWE0LIRVRSCBTV 2024-03-28T05:13:18.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: CURRENT file: CURRENT 2024-03-28T05:13:18.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: IDENTITY file: IDENTITY 2024-03-28T05:13:18.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-28T05:13:18.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-28T05:13:18.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 99480 ; 2024-03-28T05:13:18.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.error_if_exists: 0 2024-03-28T05:13:18.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.create_if_missing: 0 2024-03-28T05:13:18.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.paranoid_checks: 1 2024-03-28T05:13:18.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-28T05:13:18.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-28T05:13:18.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-28T05:13:18.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.env: 0x55ee053d4b20 2024-03-28T05:13:18.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.fs: PosixFileSystem 2024-03-28T05:13:18.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.info_log: 0x55ee05f1d920 2024-03-28T05:13:18.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_file_opening_threads: 16 2024-03-28T05:13:18.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.statistics: (nil) 2024-03-28T05:13:18.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.use_fsync: 0 2024-03-28T05:13:18.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_log_file_size: 0 2024-03-28T05:13:18.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-28T05:13:18.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-28T05:13:18.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.keep_log_file_num: 1000 2024-03-28T05:13:18.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.recycle_log_file_num: 0 2024-03-28T05:13:18.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.allow_fallocate: 1 2024-03-28T05:13:18.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.allow_mmap_reads: 0 2024-03-28T05:13:18.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.allow_mmap_writes: 0 2024-03-28T05:13:18.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.use_direct_reads: 0 2024-03-28T05:13:18.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-28T05:13:18.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.create_missing_column_families: 0 2024-03-28T05:13:18.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.db_log_dir: 2024-03-28T05:13:18.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.wal_dir: 2024-03-28T05:13:18.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-28T05:13:18.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-28T05:13:18.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-28T05:13:18.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-28T05:13:18.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-28T05:13:18.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-28T05:13:18.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.advise_random_on_open: 1 2024-03-28T05:13:18.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.db_write_buffer_size: 0 2024-03-28T05:13:18.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.write_buffer_manager: 0x55ee05ef01e0 2024-03-28T05:13:18.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-28T05:13:18.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-28T05:13:18.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-28T05:13:18.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.rate_limiter: (nil) 2024-03-28T05:13:18.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-28T05:13:18.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.wal_recovery_mode: 2 2024-03-28T05:13:18.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.enable_thread_tracking: 0 2024-03-28T05:13:18.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.enable_pipelined_write: 0 2024-03-28T05:13:18.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.unordered_write: 0 2024-03-28T05:13:18.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-28T05:13:18.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-28T05:13:18.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-28T05:13:18.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-28T05:13:18.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.row_cache: None 2024-03-28T05:13:18.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.wal_filter: None 2024-03-28T05:13:18.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-28T05:13:18.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.allow_ingest_behind: 0 2024-03-28T05:13:18.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.two_write_queues: 0 2024-03-28T05:13:18.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.manual_wal_flush: 0 2024-03-28T05:13:18.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.wal_compression: 0 2024-03-28T05:13:18.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.atomic_flush: 0 2024-03-28T05:13:18.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-28T05:13:18.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-28T05:13:18.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-28T05:13:18.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.log_readahead_size: 0 2024-03-28T05:13:18.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-28T05:13:18.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.best_efforts_recovery: 0 2024-03-28T05:13:18.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-28T05:13:18.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-28T05:13:18.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.allow_data_in_errors: 0 2024-03-28T05:13:18.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.db_host_id: __hostname__ 2024-03-28T05:13:18.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-28T05:13:18.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_background_jobs: 2 2024-03-28T05:13:18.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_background_compactions: -1 2024-03-28T05:13:18.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_subcompactions: 1 2024-03-28T05:13:18.534 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-28T05:13:18.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-28T05:13:18.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-28T05:13:18.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_total_wal_size: 0 2024-03-28T05:13:18.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-28T05:13:18.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-28T05:13:18.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-28T05:13:18.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-28T05:13:18.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_open_files: -1 2024-03-28T05:13:18.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.bytes_per_sync: 0 2024-03-28T05:13:18.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-28T05:13:18.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-28T05:13:18.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compaction_readahead_size: 0 2024-03-28T05:13:18.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_background_flushes: -1 2024-03-28T05:13:18.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Compression algorithms supported: 2024-03-28T05:13:18.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-28T05:13:18.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: kZSTD supported: 0 2024-03-28T05:13:18.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: kXpressCompression supported: 0 2024-03-28T05:13:18.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: kLZ4HCCompression supported: 1 2024-03-28T05:13:18.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: kZlibCompression supported: 1 2024-03-28T05:13:18.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: kSnappyCompression supported: 1 2024-03-28T05:13:18.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: kLZ4Compression supported: 1 2024-03-28T05:13:18.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: kBZip2Compression supported: 0 2024-03-28T05:13:18.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-28T05:13:18.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-28T05:13:18.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-28T05:13:18.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-28T05:13:18.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-28T05:13:18.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.merge_operator: 2024-03-28T05:13:18.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compaction_filter: None 2024-03-28T05:13:18.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compaction_filter_factory: None 2024-03-28T05:13:18.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.sst_partitioner_factory: None 2024-03-28T05:13:18.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-28T05:13:18.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-28T05:13:18.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ee05f1da20) 2024-03-28T05:13:18.537 INFO:journalctl@ceph.mon.a.smithi023.stdout: cache_index_and_filter_blocks: 1 2024-03-28T05:13:18.537 INFO:journalctl@ceph.mon.a.smithi023.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-28T05:13:18.537 INFO:journalctl@ceph.mon.a.smithi023.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-28T05:13:18.537 INFO:journalctl@ceph.mon.a.smithi023.stdout: pin_top_level_index_and_filter: 1 2024-03-28T05:13:18.537 INFO:journalctl@ceph.mon.a.smithi023.stdout: index_type: 0 2024-03-28T05:13:18.537 INFO:journalctl@ceph.mon.a.smithi023.stdout: data_block_index_type: 0 2024-03-28T05:13:18.537 INFO:journalctl@ceph.mon.a.smithi023.stdout: index_shortening: 1 2024-03-28T05:13:18.537 INFO:journalctl@ceph.mon.a.smithi023.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-28T05:13:18.537 INFO:journalctl@ceph.mon.a.smithi023.stdout: checksum: 4 2024-03-28T05:13:18.537 INFO:journalctl@ceph.mon.a.smithi023.stdout: no_block_cache: 0 2024-03-28T05:13:18.537 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache: 0x55ee05f23090 2024-03-28T05:13:18.537 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache_name: BinnedLRUCache 2024-03-28T05:13:18.537 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache_options: 2024-03-28T05:13:18.538 INFO:journalctl@ceph.mon.a.smithi023.stdout: capacity : 536870912 2024-03-28T05:13:18.538 INFO:journalctl@ceph.mon.a.smithi023.stdout: num_shard_bits : 4 2024-03-28T05:13:18.538 INFO:journalctl@ceph.mon.a.smithi023.stdout: strict_capacity_limit : 0 2024-03-28T05:13:18.538 INFO:journalctl@ceph.mon.a.smithi023.stdout: high_pri_pool_ratio: 0.000 2024-03-28T05:13:18.538 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache_compressed: (nil) 2024-03-28T05:13:18.538 INFO:journalctl@ceph.mon.a.smithi023.stdout: persistent_cache: (nil) 2024-03-28T05:13:18.538 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_size: 4096 2024-03-28T05:13:18.538 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_size_deviation: 10 2024-03-28T05:13:18.538 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_restart_interval: 16 2024-03-28T05:13:18.538 INFO:journalctl@ceph.mon.a.smithi023.stdout: index_block_restart_interval: 1 2024-03-28T05:13:18.538 INFO:journalctl@ceph.mon.a.smithi023.stdout: metadata_block_size: 4096 2024-03-28T05:13:18.538 INFO:journalctl@ceph.mon.a.smithi023.stdout: partition_filters: 0 2024-03-28T05:13:18.538 INFO:journalctl@ceph.mon.a.smithi023.stdout: use_delta_encoding: 1 2024-03-28T05:13:18.538 INFO:journalctl@ceph.mon.a.smithi023.stdout: filter_policy: bloomfilter 2024-03-28T05:13:18.538 INFO:journalctl@ceph.mon.a.smithi023.stdout: whole_key_filtering: 1 2024-03-28T05:13:18.538 INFO:journalctl@ceph.mon.a.smithi023.stdout: verify_compression: 0 2024-03-28T05:13:18.538 INFO:journalctl@ceph.mon.a.smithi023.stdout: read_amp_bytes_per_bit: 0 2024-03-28T05:13:18.539 INFO:journalctl@ceph.mon.a.smithi023.stdout: format_version: 5 2024-03-28T05:13:18.539 INFO:journalctl@ceph.mon.a.smithi023.stdout: enable_index_compression: 1 2024-03-28T05:13:18.539 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_align: 0 2024-03-28T05:13:18.539 INFO:journalctl@ceph.mon.a.smithi023.stdout: max_auto_readahead_size: 262144 2024-03-28T05:13:18.539 INFO:journalctl@ceph.mon.a.smithi023.stdout: prepopulate_block_cache: 0 2024-03-28T05:13:18.539 INFO:journalctl@ceph.mon.a.smithi023.stdout: initial_auto_readahead_size: 8192 2024-03-28T05:13:18.539 INFO:journalctl@ceph.mon.a.smithi023.stdout: num_file_reads_for_auto_readahead: 2 2024-03-28T05:13:18.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.write_buffer_size: 33554432 2024-03-28T05:13:18.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_write_buffer_number: 2 2024-03-28T05:13:18.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compression: NoCompression 2024-03-28T05:13:18.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.bottommost_compression: Disabled 2024-03-28T05:13:18.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.prefix_extractor: nullptr 2024-03-28T05:13:18.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-28T05:13:18.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.num_levels: 7 2024-03-28T05:13:18.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-28T05:13:18.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-28T05:13:18.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-28T05:13:18.539 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-28T05:13:18.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-28T05:13:18.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-28T05:13:18.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-28T05:13:18.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-28T05:13:18.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-28T05:13:18.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-28T05:13:18.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-28T05:13:18.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-28T05:13:18.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-28T05:13:18.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compression_opts.level: 32767 2024-03-28T05:13:18.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compression_opts.strategy: 0 2024-03-28T05:13:18.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-28T05:13:18.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-28T05:13:18.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-28T05:13:18.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-28T05:13:18.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compression_opts.enabled: false 2024-03-28T05:13:18.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-28T05:13:18.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-28T05:13:18.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-28T05:13:18.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-28T05:13:18.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.target_file_size_base: 67108864 2024-03-28T05:13:18.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-28T05:13:18.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-28T05:13:18.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-28T05:13:18.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-28T05:13:18.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-28T05:13:18.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-28T05:13:18.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-28T05:13:18.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-28T05:13:18.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-28T05:13:18.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-28T05:13:18.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-28T05:13:18.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-28T05:13:18.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-28T05:13:18.541 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-28T05:13:18.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.arena_block_size: 1048576 2024-03-28T05:13:18.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-28T05:13:18.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-28T05:13:18.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.disable_auto_compactions: 0 2024-03-28T05:13:18.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-28T05:13:18.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-28T05:13:18.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-28T05:13:18.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-28T05:13:18.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-28T05:13:18.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-28T05:13:18.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-28T05:13:18.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-28T05:13:18.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-28T05:13:18.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-28T05:13:18.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.table_properties_collectors: 2024-03-28T05:13:18.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.inplace_update_support: 0 2024-03-28T05:13:18.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-28T05:13:18.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-28T05:13:18.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-28T05:13:18.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-28T05:13:18.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.bloom_locality: 0 2024-03-28T05:13:18.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.max_successive_merges: 0 2024-03-28T05:13:18.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-28T05:13:18.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.paranoid_file_checks: 0 2024-03-28T05:13:18.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.force_consistency_checks: 1 2024-03-28T05:13:18.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.report_bg_io_stats: 0 2024-03-28T05:13:18.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.ttl: 2592000 2024-03-28T05:13:18.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-28T05:13:18.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-28T05:13:18.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-28T05:13:18.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.enable_blob_files: false 2024-03-28T05:13:18.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.min_blob_size: 0 2024-03-28T05:13:18.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.blob_file_size: 268435456 2024-03-28T05:13:18.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-28T05:13:18.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-28T05:13:18.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-28T05:13:18.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-28T05:13:18.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-28T05:13:18.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.blob_file_starting_level: 0 2024-03-28T05:13:18.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-28T05:13:18.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-28T05:13:18.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-28T05:13:18.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 386a4878-1bff-4997-b27f-c6e47c03a172 2024-03-28T05:13:18.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711602798506765, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-28T05:13:18.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-28T05:13:18.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711602798507805, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 94778, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 92766, "index_size": 294, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13640, "raw_average_key_size": 49, "raw_value_size": 85291, "raw_average_value_size": 312, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711602798, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "386a4878-1bff-4997-b27f-c6e47c03a172", "db_session_id": "2W7AI9GWE0LIRVRSCBTV", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-28T05:13:18.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711602798507889, "job": 1, "event": "recovery_finished"} 2024-03-28T05:13:18.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-28T05:13:18.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-28T05:13:18.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ee06016000 2024-03-28T05:13:18.544 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: DB pointer 0x55ee06000000 2024-03-28T05:13:18.545 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-28T05:13:18.545 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-28T05:13:18.545 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** DB Stats ** 2024-03-28T05:13:18.545 INFO:journalctl@ceph.mon.a.smithi023.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T05:13:18.545 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-28T05:13:18.545 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T05:13:18.545 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T05:13:18.545 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-28T05:13:18.545 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T05:13:18.545 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T05:13:18.545 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-28T05:13:18.545 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** Compaction Stats [default] ** 2024-03-28T05:13:18.545 INFO:journalctl@ceph.mon.a.smithi023.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T05:13:18.545 INFO:journalctl@ceph.mon.a.smithi023.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-28T05:13:18.545 INFO:journalctl@ceph.mon.a.smithi023.stdout: L0 2/0 94.39 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T05:13:18.545 INFO:journalctl@ceph.mon.a.smithi023.stdout: Sum 2/0 94.39 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T05:13:18.545 INFO:journalctl@ceph.mon.a.smithi023.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T05:13:18.546 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-28T05:13:18.546 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** Compaction Stats [default] ** 2024-03-28T05:13:18.546 INFO:journalctl@ceph.mon.a.smithi023.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T05:13:18.546 INFO:journalctl@ceph.mon.a.smithi023.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-28T05:13:18.546 INFO:journalctl@ceph.mon.a.smithi023.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 133.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T05:13:18.546 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-28T05:13:18.546 INFO:journalctl@ceph.mon.a.smithi023.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-28T05:13:18.546 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-28T05:13:18.546 INFO:journalctl@ceph.mon.a.smithi023.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T05:13:18.546 INFO:journalctl@ceph.mon.a.smithi023.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-28T05:13:18.546 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-28T05:13:18.546 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-28T05:13:18.546 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-28T05:13:18.546 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-28T05:13:18.546 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative compaction: 0.00 GB write, 15.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T05:13:18.546 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval compaction: 0.00 GB write, 15.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T05:13:18.547 INFO:journalctl@ceph.mon.a.smithi023.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-28T05:13:18.547 INFO:journalctl@ceph.mon.a.smithi023.stdout: Block cache BinnedLRUCache@0x55ee05f23090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-28T05:13:18.547 INFO:journalctl@ceph.mon.a.smithi023.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-28T05:13:18.547 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-28T05:13:18.547 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-28T05:13:18.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: starting mon.a rank 0 at public addrs [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] at bind addrs [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:13:18.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: mon.a@-1(???) e1 preinit fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:13:18.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: mon.a@-1(???).mds e1 new map 2024-03-28T05:13:18.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: mon.a@-1(???).mds e1 print_map 2024-03-28T05:13:18.547 INFO:journalctl@ceph.mon.a.smithi023.stdout: e1 2024-03-28T05:13:18.547 INFO:journalctl@ceph.mon.a.smithi023.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T05:13:18.547 INFO:journalctl@ceph.mon.a.smithi023.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T05:13:18.547 INFO:journalctl@ceph.mon.a.smithi023.stdout: legacy client fscid: -1 2024-03-28T05:13:18.547 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-28T05:13:18.547 INFO:journalctl@ceph.mon.a.smithi023.stdout: No filesystems configured 2024-03-28T05:13:18.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-28T05:13:18.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T05:13:18.547 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T05:13:18.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T05:13:18.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-28T05:13:18.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 bash[31935]: 6ba187531ba0f3ed1e1e055f2c7ee8cb929c8df2c5d692aa1be4443803d55012 2024-03-28T05:13:18.548 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 systemd[1]: Started Ceph mon.a for bf37ebc4-ecc1-11ee-95d0-87774f69a715. 2024-03-28T05:13:18.548 INFO:teuthology.orchestra.run.smithi023.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-28T05:13:18.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T05:13:18.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: monmap e1: 1 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T05:13:18.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: fsmap 2024-03-28T05:13:18.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: osdmap e1: 0 total, 0 up, 0 in 2024-03-28T05:13:18.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:18 smithi023 ceph-mon[31954]: mgrmap e1: no daemons active 2024-03-28T05:13:19.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.973+0000 7f4ddf085700 1 Processor -- start 2024-03-28T05:13:19.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.973+0000 7f4ddf085700 1 -- start start 2024-03-28T05:13:19.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.973+0000 7f4ddf085700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4dd8107a20 0x7f4dd8107e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:19.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.973+0000 7f4ddf085700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd8108410 con 0x7f4dd8107a20 2024-03-28T05:13:19.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.974+0000 7f4ddce21700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4dd8107a20 0x7f4dd8107e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:19.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.974+0000 7f4ddce21700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4dd8107a20 0x7f4dd8107e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57464/0 (socket says 172.21.15.23:57464) 2024-03-28T05:13:19.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.974+0000 7f4ddce21700 1 -- 172.21.15.23:0/4161270985 learned_addr learned my addr 172.21.15.23:0/4161270985 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:19.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.974+0000 7f4ddce21700 1 -- 172.21.15.23:0/4161270985 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd8108c20 con 0x7f4dd8107a20 2024-03-28T05:13:19.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.974+0000 7f4ddce21700 1 --2- 172.21.15.23:0/4161270985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4dd8107a20 0x7f4dd8107e40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc0098a0 tx=0x7f4dcc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c46531fc8dd8026e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:19.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.974+0000 7f4dd77fe700 1 -- 172.21.15.23:0/4161270985 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4dcc004160 con 0x7f4dd8107a20 2024-03-28T05:13:19.582 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.974+0000 7f4dd77fe700 1 -- 172.21.15.23:0/4161270985 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4dcc004300 con 0x7f4dd8107a20 2024-03-28T05:13:19.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.975+0000 7f4dd77fe700 1 -- 172.21.15.23:0/4161270985 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4dcc011b50 con 0x7f4dd8107a20 2024-03-28T05:13:19.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.975+0000 7f4ddf085700 1 -- 172.21.15.23:0/4161270985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4dd8107a20 msgr2=0x7f4dd8107e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:19.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.975+0000 7f4ddf085700 1 --2- 172.21.15.23:0/4161270985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4dd8107a20 0x7f4dd8107e40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc0098a0 tx=0x7f4dcc009bb0 comp rx=0 tx=0).stop 2024-03-28T05:13:19.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.975+0000 7f4ddf085700 1 -- 172.21.15.23:0/4161270985 shutdown_connections 2024-03-28T05:13:19.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.975+0000 7f4ddf085700 1 --2- 172.21.15.23:0/4161270985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4dd8107a20 0x7f4dd8107e40 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:19.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.975+0000 7f4ddf085700 1 -- 172.21.15.23:0/4161270985 >> 172.21.15.23:0/4161270985 conn(0x7f4dd81030a0 msgr2=0x7f4dd81054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:19.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.975+0000 7f4ddf085700 1 -- 172.21.15.23:0/4161270985 shutdown_connections 2024-03-28T05:13:19.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.975+0000 7f4ddf085700 1 -- 172.21.15.23:0/4161270985 wait complete. 2024-03-28T05:13:19.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.976+0000 7f4ddf085700 1 Processor -- start 2024-03-28T05:13:19.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.976+0000 7f4ddf085700 1 -- start start 2024-03-28T05:13:19.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.976+0000 7f4ddf085700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4dd8107a20 0x7f4dd819b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:19.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.976+0000 7f4ddf085700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd810bfd0 con 0x7f4dd8107a20 2024-03-28T05:13:19.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.976+0000 7f4ddce21700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4dd8107a20 0x7f4dd819b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:19.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.976+0000 7f4ddce21700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4dd8107a20 0x7f4dd819b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57466/0 (socket says 172.21.15.23:57466) 2024-03-28T05:13:19.583 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.976+0000 7f4ddce21700 1 -- 172.21.15.23:0/584294062 learned_addr learned my addr 172.21.15.23:0/584294062 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:19.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.976+0000 7f4ddce21700 1 -- 172.21.15.23:0/584294062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd819bef0 con 0x7f4dd8107a20 2024-03-28T05:13:19.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.977+0000 7f4ddce21700 1 --2- 172.21.15.23:0/584294062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4dd8107a20 0x7f4dd819b9b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc011670 tx=0x7f4dcc011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:19.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.977+0000 7f4dd5ffb700 1 -- 172.21.15.23:0/584294062 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4dcc0039c0 con 0x7f4dd8107a20 2024-03-28T05:13:19.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.977+0000 7f4dd5ffb700 1 -- 172.21.15.23:0/584294062 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4dcc011e50 con 0x7f4dd8107a20 2024-03-28T05:13:19.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.977+0000 7f4ddf085700 1 -- 172.21.15.23:0/584294062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd819c180 con 0x7f4dd8107a20 2024-03-28T05:13:19.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.977+0000 7f4dd5ffb700 1 -- 172.21.15.23:0/584294062 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4dcc01a5c0 con 0x7f4dd8107a20 2024-03-28T05:13:19.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.977+0000 7f4ddf085700 1 -- 172.21.15.23:0/584294062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd819ee70 con 0x7f4dd8107a20 2024-03-28T05:13:19.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.977+0000 7f4dd5ffb700 1 -- 172.21.15.23:0/584294062 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f4dcc01ac00 con 0x7f4dd8107a20 2024-03-28T05:13:19.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.977+0000 7f4dd5ffb700 1 -- 172.21.15.23:0/584294062 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4dcc019700 con 0x7f4dd8107a20 2024-03-28T05:13:19.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.978+0000 7f4ddf085700 1 -- 172.21.15.23:0/584294062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd80637c0 con 0x7f4dd8107a20 2024-03-28T05:13:19.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:18.980+0000 7f4dd5ffb700 1 -- 172.21.15.23:0/584294062 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f4dcc003b60 con 0x7f4dd8107a20 2024-03-28T05:13:19.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:19.050+0000 7f4ddf085700 1 -- 172.21.15.23:0/584294062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f4dd819f040 con 0x7f4dd8107a20 2024-03-28T05:13:19.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:19.052+0000 7f4dd5ffb700 1 -- 172.21.15.23:0/584294062 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f4dcc019900 con 0x7f4dd8107a20 2024-03-28T05:13:19.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:19.053+0000 7f4ddf085700 1 -- 172.21.15.23:0/584294062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4dd8107a20 msgr2=0x7f4dd819b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:19.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:19.053+0000 7f4ddf085700 1 --2- 172.21.15.23:0/584294062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4dd8107a20 0x7f4dd819b9b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc011670 tx=0x7f4dcc011b60 comp rx=0 tx=0).stop 2024-03-28T05:13:19.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:19.053+0000 7f4ddf085700 1 -- 172.21.15.23:0/584294062 shutdown_connections 2024-03-28T05:13:19.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:19.053+0000 7f4ddf085700 1 --2- 172.21.15.23:0/584294062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4dd8107a20 0x7f4dd819b9b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:19.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:19.053+0000 7f4ddf085700 1 -- 172.21.15.23:0/584294062 >> 172.21.15.23:0/584294062 conn(0x7f4dd81030a0 msgr2=0x7f4dd8103bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:19.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:19.053+0000 7f4ddf085700 1 -- 172.21.15.23:0/584294062 shutdown_connections 2024-03-28T05:13:19.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:19.053+0000 7f4ddf085700 1 -- 172.21.15.23:0/584294062 wait complete. 2024-03-28T05:13:19.585 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-28T05:13:19.600 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-28T05:13:19.600 INFO:teuthology.orchestra.run.smithi023.stdout:Creating mgr... 2024-03-28T05:13:19.600 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-28T05:13:19.601 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-28T05:13:19.837 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mgr.x 2024-03-28T05:13:19.837 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to reset failed state of unit ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mgr.x.service: Unit ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mgr.x.service not loaded. 2024-03-28T05:13:19.985 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715.target.wants/ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@.service. 2024-03-28T05:13:20.202 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:20 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/584294062' entity='client.admin' 2024-03-28T05:13:20.462 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T05:13:20.462 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-28T05:13:20.470 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T05:13:20.470 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-28T05:13:20.470 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-28T05:13:20.470 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to enable service . firewalld.service is not available 2024-03-28T05:13:20.480 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T05:13:20.480 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-28T05:13:20.488 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T05:13:20.488 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-28T05:13:20.488 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-28T05:13:20.488 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-28T05:13:20.488 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr to start... 2024-03-28T05:13:20.488 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr... 2024-03-28T05:13:20.527 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:20 smithi023 systemd[1]: Started Ceph mgr.x for bf37ebc4-ecc1-11ee-95d0-87774f69a715. 2024-03-28T05:13:21.277 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:21 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1616764087' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T05:13:21.278 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:20 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:20.958+0000 7f2f10a8d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T05:13:21.278 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:21 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:21.060+0000 7f2f10a8d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T05:13:21.953 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:21 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:21.721+0000 7f2f10a8d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T05:13:21.953 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:21 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:21.815+0000 7f2f10a8d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T05:13:21.953 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:21 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:21.957+0000 7f2f10a8d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T05:13:22.217 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:22 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:22.054+0000 7f2f10a8d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T05:13:22.217 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:22 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:22.138+0000 7f2f10a8d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T05:13:22.510 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:22 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:22.403+0000 7f2f10a8d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T05:13:22.510 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:22 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:22.514+0000 7f2f10a8d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T05:13:22.601 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T05:13:22.601 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-28T05:13:22.601 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "bf37ebc4-ecc1-11ee-95d0-87774f69a715", 2024-03-28T05:13:22.601 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T05:13:22.601 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T05:13:22.601 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T05:13:22.601 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T05:13:22.602 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T05:13:22.602 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T05:13:22.602 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T05:13:22.602 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-28T05:13:22.602 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-28T05:13:22.602 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T05:13:22.602 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T05:13:22.602 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-28T05:13:22.602 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-28T05:13:22.603 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T05:13:22.603 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T05:13:22.603 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T05:13:22.603 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T05:13:22.603 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T05:13:22.603 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T05:13:22.603 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T05:13:22.603 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T05:13:22.603 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T05:13:22.603 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T05:13:22.611 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T05:13:22.611 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T05:13:22.611 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T05:13:22.611 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T05:13:22.612 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T05:13:22.612 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T05:13:22.612 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T05:13:22.612 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T05:13:22.612 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T05:13:22.612 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T05:13:22.612 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T05:13:22.612 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T05:13:22.612 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T05:13:22.612 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T05:13:22.612 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T05:13:22.612 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T05:13:22.612 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T05:13:22.612 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T05:13:22.612 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T05:13:22.612 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T05:13:22.612 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T05:13:22.613 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T05:13:22.613 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T05:13:22.613 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T05:13:22.613 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T05:13:22.613 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T05:13:22.613 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-28T05:13:22.613 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T05:13:22.613 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T05:13:22.613 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T05:13:22.613 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T05:13:22.613 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T05:13:13.454119+0000", 2024-03-28T05:13:22.613 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T05:13:22.613 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T05:13:22.613 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T05:13:22.613 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-28T05:13:22.613 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.914+0000 7f987db69700 1 Processor -- start 2024-03-28T05:13:22.613 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.914+0000 7f987db69700 1 -- start start 2024-03-28T05:13:22.614 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.914+0000 7f987db69700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9878108450 0x7f987810a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:22.614 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.914+0000 7f987db69700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987807aed0 con 0x7f9878108450 2024-03-28T05:13:22.614 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.914+0000 7f98777fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9878108450 0x7f987810a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:22.614 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.914+0000 7f98777fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9878108450 0x7f987810a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57486/0 (socket says 172.21.15.23:57486) 2024-03-28T05:13:22.614 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.914+0000 7f98777fe700 1 -- 172.21.15.23:0/1829519591 learned_addr learned my addr 172.21.15.23:0/1829519591 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:22.614 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.915+0000 7f98777fe700 1 -- 172.21.15.23:0/1829519591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987810ad80 con 0x7f9878108450 2024-03-28T05:13:22.614 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.915+0000 7f98777fe700 1 --2- 172.21.15.23:0/1829519591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9878108450 0x7f987810a840 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f98680098a0 tx=0x7f9868009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e39199174983fce server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:22.614 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.915+0000 7f98767fc700 1 -- 172.21.15.23:0/1829519591 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9868004160 con 0x7f9878108450 2024-03-28T05:13:22.614 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.915+0000 7f98767fc700 1 -- 172.21.15.23:0/1829519591 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9868004300 con 0x7f9878108450 2024-03-28T05:13:22.614 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.915+0000 7f98767fc700 1 -- 172.21.15.23:0/1829519591 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9868011b50 con 0x7f9878108450 2024-03-28T05:13:22.614 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.916+0000 7f987db69700 1 -- 172.21.15.23:0/1829519591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9878108450 msgr2=0x7f987810a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:22.614 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.916+0000 7f987db69700 1 --2- 172.21.15.23:0/1829519591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9878108450 0x7f987810a840 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f98680098a0 tx=0x7f9868009bb0 comp rx=0 tx=0).stop 2024-03-28T05:13:22.614 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.916+0000 7f987db69700 1 -- 172.21.15.23:0/1829519591 shutdown_connections 2024-03-28T05:13:22.614 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.916+0000 7f987db69700 1 --2- 172.21.15.23:0/1829519591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9878108450 0x7f987810a840 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:22.614 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.916+0000 7f987db69700 1 -- 172.21.15.23:0/1829519591 >> 172.21.15.23:0/1829519591 conn(0x7f9878101ee0 msgr2=0x7f9878104300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:22.615 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.916+0000 7f987db69700 1 -- 172.21.15.23:0/1829519591 shutdown_connections 2024-03-28T05:13:22.615 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.916+0000 7f987db69700 1 -- 172.21.15.23:0/1829519591 wait complete. 2024-03-28T05:13:22.615 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.916+0000 7f987db69700 1 Processor -- start 2024-03-28T05:13:22.615 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.916+0000 7f987db69700 1 -- start start 2024-03-28T05:13:22.615 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.917+0000 7f987db69700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9878108450 0x7f987819ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:22.615 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.917+0000 7f987db69700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987810bdb0 con 0x7f9878108450 2024-03-28T05:13:22.615 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.917+0000 7f98777fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9878108450 0x7f987819ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:22.615 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.917+0000 7f98777fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9878108450 0x7f987819ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57494/0 (socket says 172.21.15.23:57494) 2024-03-28T05:13:22.615 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.917+0000 7f98777fe700 1 -- 172.21.15.23:0/1616764087 learned_addr learned my addr 172.21.15.23:0/1616764087 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:22.615 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.917+0000 7f98777fe700 1 -- 172.21.15.23:0/1616764087 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987819bf40 con 0x7f9878108450 2024-03-28T05:13:22.615 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.917+0000 7f98777fe700 1 --2- 172.21.15.23:0/1616764087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9878108450 0x7f987819ba00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9868011b60 tx=0x7f9868011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:22.615 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.917+0000 7f9874ff9700 1 -- 172.21.15.23:0/1616764087 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9868003970 con 0x7f9878108450 2024-03-28T05:13:22.615 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.917+0000 7f9874ff9700 1 -- 172.21.15.23:0/1616764087 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9868003b10 con 0x7f9878108450 2024-03-28T05:13:22.615 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.917+0000 7f987db69700 1 -- 172.21.15.23:0/1616764087 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f987819c1d0 con 0x7f9878108450 2024-03-28T05:13:22.615 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.917+0000 7f9874ff9700 1 -- 172.21.15.23:0/1616764087 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9868024610 con 0x7f9878108450 2024-03-28T05:13:22.615 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.918+0000 7f987db69700 1 -- 172.21.15.23:0/1616764087 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f987819eec0 con 0x7f9878108450 2024-03-28T05:13:22.616 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.918+0000 7f9874ff9700 1 -- 172.21.15.23:0/1616764087 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9868024ca0 con 0x7f9878108450 2024-03-28T05:13:22.616 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.918+0000 7f9874ff9700 1 -- 172.21.15.23:0/1616764087 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9868011e00 con 0x7f9878108450 2024-03-28T05:13:22.616 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.918+0000 7f987db69700 1 -- 172.21.15.23:0/1616764087 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9878069220 con 0x7f9878108450 2024-03-28T05:13:22.616 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.921+0000 7f9874ff9700 1 -- 172.21.15.23:0/1616764087 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9868043dd0 con 0x7f9878108450 2024-03-28T05:13:22.616 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.995+0000 7f987db69700 1 -- 172.21.15.23:0/1616764087 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f987819f470 con 0x7f9878108450 2024-03-28T05:13:22.616 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.995+0000 7f9874ff9700 1 -- 172.21.15.23:0/1616764087 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f9868032030 con 0x7f9878108450 2024-03-28T05:13:22.616 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.996+0000 7f987db69700 1 -- 172.21.15.23:0/1616764087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9878108450 msgr2=0x7f987819ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:22.616 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.996+0000 7f987db69700 1 --2- 172.21.15.23:0/1616764087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9878108450 0x7f987819ba00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9868011b60 tx=0x7f9868011c40 comp rx=0 tx=0).stop 2024-03-28T05:13:22.616 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.996+0000 7f987db69700 1 -- 172.21.15.23:0/1616764087 shutdown_connections 2024-03-28T05:13:22.616 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.996+0000 7f987db69700 1 --2- 172.21.15.23:0/1616764087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9878108450 0x7f987819ba00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:22.616 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.996+0000 7f987db69700 1 -- 172.21.15.23:0/1616764087 >> 172.21.15.23:0/1616764087 conn(0x7f9878101ee0 msgr2=0x7f9878102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:22.616 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.996+0000 7f987db69700 1 -- 172.21.15.23:0/1616764087 shutdown_connections 2024-03-28T05:13:22.616 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:20.996+0000 7f987db69700 1 -- 172.21.15.23:0/1616764087 wait complete. 2024-03-28T05:13:22.616 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (1/15)... 2024-03-28T05:13:22.777 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:22 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:22.602+0000 7f2f10a8d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T05:13:22.777 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:22 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:22.699+0000 7f2f10a8d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T05:13:23.277 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:22 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:22.878+0000 7f2f10a8d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T05:13:24.277 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:23 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:23.877+0000 7f2f10a8d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T05:13:24.277 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:24 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:24.011+0000 7f2f10a8d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T05:13:24.639 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:24 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:24.349+0000 7f2f10a8d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T05:13:24.639 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:24 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:24.441+0000 7f2f10a8d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T05:13:24.639 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:24 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:24.644+0000 7f2f10a8d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T05:13:25.027 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:24 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:24.833+0000 7f2f10a8d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T05:13:25.378 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:25 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:25.123+0000 7f2f10a8d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T05:13:25.378 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:25 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:25.383+0000 7f2f10a8d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T05:13:25.777 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:25 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:25.626+0000 7f2f10a8d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T05:13:26.777 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:26 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:26.450+0000 7f2f10a8d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T05:13:27.277 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:27 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:27.029+0000 7f2f10a8d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T05:13:27.277 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:27 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:27.132+0000 7f2f10a8d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T05:13:27.777 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:27 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:27.353+0000 7f2f10a8d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T05:13:27.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:27 smithi023 ceph-mon[31954]: Activating manager daemon x 2024-03-28T05:13:27.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:27 smithi023 ceph-mon[31954]: mgrmap e2: x(active, starting, since 0.00346878s) 2024-03-28T05:13:27.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:27 smithi023 ceph-mon[31954]: from='mgr.14100 172.21.15.23:0/917833595' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T05:13:27.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:27 smithi023 ceph-mon[31954]: from='mgr.14100 172.21.15.23:0/917833595' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T05:13:27.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:27 smithi023 ceph-mon[31954]: from='mgr.14100 172.21.15.23:0/917833595' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T05:13:27.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:27 smithi023 ceph-mon[31954]: from='mgr.14100 172.21.15.23:0/917833595' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T05:13:27.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:27 smithi023 ceph-mon[31954]: from='mgr.14100 172.21.15.23:0/917833595' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T05:13:27.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:27 smithi023 ceph-mon[31954]: Manager daemon x is now available 2024-03-28T05:13:27.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:27 smithi023 ceph-mon[31954]: from='mgr.14100 172.21.15.23:0/917833595' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T05:13:27.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:27 smithi023 ceph-mon[31954]: from='mgr.14100 172.21.15.23:0/917833595' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T05:13:27.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:27 smithi023 ceph-mon[31954]: from='mgr.14100 172.21.15.23:0/917833595' entity='mgr.x' 2024-03-28T05:13:27.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:27 smithi023 ceph-mon[31954]: from='mgr.14100 172.21.15.23:0/917833595' entity='mgr.x' 2024-03-28T05:13:27.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:27 smithi023 ceph-mon[31954]: from='mgr.14100 172.21.15.23:0/917833595' entity='mgr.x' 2024-03-28T05:13:28.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:28 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1611160188' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T05:13:28.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:28 smithi023 ceph-mon[31954]: mgrmap e3: x(active, since 1.00775s) 2024-03-28T05:13:29.011 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T05:13:29.011 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-28T05:13:29.011 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "bf37ebc4-ecc1-11ee-95d0-87774f69a715", 2024-03-28T05:13:29.011 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T05:13:29.011 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T05:13:29.011 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T05:13:29.011 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T05:13:29.011 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T05:13:29.011 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T05:13:29.012 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T05:13:29.012 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-28T05:13:29.012 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-28T05:13:29.012 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T05:13:29.012 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T05:13:29.012 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-28T05:13:29.012 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-28T05:13:29.012 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T05:13:29.012 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T05:13:29.012 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T05:13:29.012 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T05:13:29.012 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T05:13:29.012 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T05:13:29.012 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T05:13:29.012 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T05:13:29.012 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T05:13:29.012 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T05:13:29.020 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T05:13:29.020 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T05:13:29.020 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T05:13:29.020 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T05:13:29.020 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T05:13:29.020 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T05:13:29.020 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T05:13:29.020 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T05:13:29.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T05:13:29.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T05:13:29.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T05:13:29.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T05:13:29.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T05:13:29.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T05:13:29.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T05:13:29.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T05:13:29.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T05:13:29.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T05:13:29.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T05:13:29.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T05:13:29.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T05:13:29.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T05:13:29.021 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T05:13:29.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T05:13:29.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T05:13:29.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T05:13:29.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-28T05:13:29.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T05:13:29.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T05:13:29.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T05:13:29.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T05:13:29.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T05:13:13.454119+0000", 2024-03-28T05:13:29.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T05:13:29.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T05:13:29.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T05:13:29.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-28T05:13:29.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.950+0000 7fa44fe85700 1 Processor -- start 2024-03-28T05:13:29.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.950+0000 7fa44fe85700 1 -- start start 2024-03-28T05:13:29.022 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.950+0000 7fa44fe85700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa448107ac0 0x7fa448107ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:29.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.950+0000 7fa44fe85700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4481084b0 con 0x7fa448107ac0 2024-03-28T05:13:29.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.950+0000 7fa44dc21700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa448107ac0 0x7fa448107ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:29.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.950+0000 7fa44dc21700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa448107ac0 0x7fa448107ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43588/0 (socket says 172.21.15.23:43588) 2024-03-28T05:13:29.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.950+0000 7fa44dc21700 1 -- 172.21.15.23:0/2543376604 learned_addr learned my addr 172.21.15.23:0/2543376604 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:29.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.950+0000 7fa44dc21700 1 -- 172.21.15.23:0/2543376604 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa448108c70 con 0x7fa448107ac0 2024-03-28T05:13:29.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.950+0000 7fa44dc21700 1 --2- 172.21.15.23:0/2543376604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa448107ac0 0x7fa448107ee0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa4440098a0 tx=0x7fa444009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ef66ce3acf59e53 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:29.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.951+0000 7fa44cc1f700 1 -- 172.21.15.23:0/2543376604 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa444004160 con 0x7fa448107ac0 2024-03-28T05:13:29.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.951+0000 7fa44cc1f700 1 -- 172.21.15.23:0/2543376604 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa444004300 con 0x7fa448107ac0 2024-03-28T05:13:29.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.951+0000 7fa44cc1f700 1 -- 172.21.15.23:0/2543376604 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa444011b50 con 0x7fa448107ac0 2024-03-28T05:13:29.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.951+0000 7fa44fe85700 1 -- 172.21.15.23:0/2543376604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa448107ac0 msgr2=0x7fa448107ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:29.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.951+0000 7fa44fe85700 1 --2- 172.21.15.23:0/2543376604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa448107ac0 0x7fa448107ee0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa4440098a0 tx=0x7fa444009bb0 comp rx=0 tx=0).stop 2024-03-28T05:13:29.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.951+0000 7fa44fe85700 1 -- 172.21.15.23:0/2543376604 shutdown_connections 2024-03-28T05:13:29.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.951+0000 7fa44fe85700 1 --2- 172.21.15.23:0/2543376604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa448107ac0 0x7fa448107ee0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:29.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.951+0000 7fa44fe85700 1 -- 172.21.15.23:0/2543376604 >> 172.21.15.23:0/2543376604 conn(0x7fa4481030c0 msgr2=0x7fa4481054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:29.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.951+0000 7fa44fe85700 1 -- 172.21.15.23:0/2543376604 shutdown_connections 2024-03-28T05:13:29.023 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.951+0000 7fa44fe85700 1 -- 172.21.15.23:0/2543376604 wait complete. 2024-03-28T05:13:29.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.952+0000 7fa44fe85700 1 Processor -- start 2024-03-28T05:13:29.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.952+0000 7fa44fe85700 1 -- start start 2024-03-28T05:13:29.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.952+0000 7fa44fe85700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa448107ac0 0x7fa44819ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:29.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.952+0000 7fa44fe85700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44810c020 con 0x7fa448107ac0 2024-03-28T05:13:29.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.952+0000 7fa44dc21700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa448107ac0 0x7fa44819ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:29.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.952+0000 7fa44dc21700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa448107ac0 0x7fa44819ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43604/0 (socket says 172.21.15.23:43604) 2024-03-28T05:13:29.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.952+0000 7fa44dc21700 1 -- 172.21.15.23:0/1611160188 learned_addr learned my addr 172.21.15.23:0/1611160188 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:29.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.952+0000 7fa44dc21700 1 -- 172.21.15.23:0/1611160188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44819bf70 con 0x7fa448107ac0 2024-03-28T05:13:29.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.952+0000 7fa44dc21700 1 --2- 172.21.15.23:0/1611160188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa448107ac0 0x7fa44819ba30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa444011850 tx=0x7fa444011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:29.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.953+0000 7fa43affd700 1 -- 172.21.15.23:0/1611160188 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa444003af0 con 0x7fa448107ac0 2024-03-28T05:13:29.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.953+0000 7fa43affd700 1 -- 172.21.15.23:0/1611160188 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa444020070 con 0x7fa448107ac0 2024-03-28T05:13:29.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.953+0000 7fa43affd700 1 -- 172.21.15.23:0/1611160188 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa4440195a0 con 0x7fa448107ac0 2024-03-28T05:13:29.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.953+0000 7fa44fe85700 1 -- 172.21.15.23:0/1611160188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa44819c200 con 0x7fa448107ac0 2024-03-28T05:13:29.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.953+0000 7fa44fe85700 1 -- 172.21.15.23:0/1611160188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa44819eef0 con 0x7fa448107ac0 2024-03-28T05:13:29.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.953+0000 7fa43affd700 1 -- 172.21.15.23:0/1611160188 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 2) v1 ==== 45862+0+0 (secure 0 0 0) 0x7fa4440198a0 con 0x7fa448107ac0 2024-03-28T05:13:29.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.953+0000 7fa44fe85700 1 -- 172.21.15.23:0/1611160188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa448069220 con 0x7fa448107ac0 2024-03-28T05:13:29.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.954+0000 7fa43affd700 1 -- 172.21.15.23:0/1611160188 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa444011e00 con 0x7fa448107ac0 2024-03-28T05:13:29.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:27.956+0000 7fa43affd700 1 -- 172.21.15.23:0/1611160188 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fa444003c90 con 0x7fa448107ac0 2024-03-28T05:13:29.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:28.029+0000 7fa44fe85700 1 -- 172.21.15.23:0/1611160188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa44819f0c0 con 0x7fa448107ac0 2024-03-28T05:13:29.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:28.029+0000 7fa43affd700 1 -- 172.21.15.23:0/1611160188 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fa44819f0c0 con 0x7fa448107ac0 2024-03-28T05:13:29.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:28.030+0000 7fa44fe85700 1 -- 172.21.15.23:0/1611160188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa448107ac0 msgr2=0x7fa44819ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:29.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:28.030+0000 7fa44fe85700 1 --2- 172.21.15.23:0/1611160188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa448107ac0 0x7fa44819ba30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa444011850 tx=0x7fa444011c10 comp rx=0 tx=0).stop 2024-03-28T05:13:29.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:28.030+0000 7fa44fe85700 1 -- 172.21.15.23:0/1611160188 shutdown_connections 2024-03-28T05:13:29.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:28.030+0000 7fa44fe85700 1 --2- 172.21.15.23:0/1611160188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa448107ac0 0x7fa44819ba30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:29.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:28.030+0000 7fa44fe85700 1 -- 172.21.15.23:0/1611160188 >> 172.21.15.23:0/1611160188 conn(0x7fa4481030c0 msgr2=0x7fa448103ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:29.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:28.030+0000 7fa44fe85700 1 -- 172.21.15.23:0/1611160188 shutdown_connections 2024-03-28T05:13:29.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:28.030+0000 7fa44fe85700 1 -- 172.21.15.23:0/1611160188 wait complete. 2024-03-28T05:13:29.025 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (2/15)... 2024-03-28T05:13:30.777 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:30 smithi023 ceph-mon[31954]: mgrmap e4: x(active, since 2s) 2024-03-28T05:13:32.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:31 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/460550753' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T05:13:32.300 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T05:13:32.300 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-28T05:13:32.301 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "bf37ebc4-ecc1-11ee-95d0-87774f69a715", 2024-03-28T05:13:32.301 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T05:13:32.301 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T05:13:32.301 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T05:13:32.301 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T05:13:32.301 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T05:13:32.301 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T05:13:32.301 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T05:13:32.301 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-28T05:13:32.301 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-28T05:13:32.301 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T05:13:32.301 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T05:13:32.301 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-28T05:13:32.301 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-03-28T05:13:32.301 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T05:13:32.301 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T05:13:32.302 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T05:13:32.302 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T05:13:32.302 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T05:13:32.302 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T05:13:32.302 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T05:13:32.302 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T05:13:32.302 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T05:13:32.302 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T05:13:32.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T05:13:32.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T05:13:32.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T05:13:32.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T05:13:32.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T05:13:32.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T05:13:32.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T05:13:32.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T05:13:32.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T05:13:32.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T05:13:32.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T05:13:32.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T05:13:32.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T05:13:32.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T05:13:32.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T05:13:32.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T05:13:32.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T05:13:32.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T05:13:32.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T05:13:32.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T05:13:32.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T05:13:32.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T05:13:32.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T05:13:32.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T05:13:32.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T05:13:32.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T05:13:32.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-28T05:13:32.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T05:13:32.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T05:13:32.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T05:13:32.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T05:13:32.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T05:13:13.454119+0000", 2024-03-28T05:13:32.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T05:13:32.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T05:13:32.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T05:13:32.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-28T05:13:32.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.464+0000 7f590c9f4700 1 Processor -- start 2024-03-28T05:13:32.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.464+0000 7f590c9f4700 1 -- start start 2024-03-28T05:13:32.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.464+0000 7f590c9f4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5908108510 0x7f590810a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:32.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.464+0000 7f590c9f4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590807adf0 con 0x7f5908108510 2024-03-28T05:13:32.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.464+0000 7f590659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5908108510 0x7f590810a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:32.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.464+0000 7f590659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5908108510 0x7f590810a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43606/0 (socket says 172.21.15.23:43606) 2024-03-28T05:13:32.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.464+0000 7f590659c700 1 -- 172.21.15.23:0/2680099993 learned_addr learned my addr 172.21.15.23:0/2680099993 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:32.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.465+0000 7f590659c700 1 -- 172.21.15.23:0/2680099993 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f590810ae40 con 0x7f5908108510 2024-03-28T05:13:32.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.465+0000 7f590659c700 1 --2- 172.21.15.23:0/2680099993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5908108510 0x7f590810a900 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f58f80098a0 tx=0x7f58f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5521d43beb636b44 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:32.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.465+0000 7f590559a700 1 -- 172.21.15.23:0/2680099993 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58f8004160 con 0x7f5908108510 2024-03-28T05:13:32.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.465+0000 7f590559a700 1 -- 172.21.15.23:0/2680099993 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58f8004300 con 0x7f5908108510 2024-03-28T05:13:32.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.465+0000 7f590559a700 1 -- 172.21.15.23:0/2680099993 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58f8011b50 con 0x7f5908108510 2024-03-28T05:13:32.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.465+0000 7f590c9f4700 1 -- 172.21.15.23:0/2680099993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5908108510 msgr2=0x7f590810a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:32.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.465+0000 7f590c9f4700 1 --2- 172.21.15.23:0/2680099993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5908108510 0x7f590810a900 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f58f80098a0 tx=0x7f58f8009bb0 comp rx=0 tx=0).stop 2024-03-28T05:13:32.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.466+0000 7f590c9f4700 1 -- 172.21.15.23:0/2680099993 shutdown_connections 2024-03-28T05:13:32.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.466+0000 7f590c9f4700 1 --2- 172.21.15.23:0/2680099993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5908108510 0x7f590810a900 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:32.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.466+0000 7f590c9f4700 1 -- 172.21.15.23:0/2680099993 >> 172.21.15.23:0/2680099993 conn(0x7f5908101f40 msgr2=0x7f5908104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:32.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.466+0000 7f590c9f4700 1 -- 172.21.15.23:0/2680099993 shutdown_connections 2024-03-28T05:13:32.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.466+0000 7f590c9f4700 1 -- 172.21.15.23:0/2680099993 wait complete. 2024-03-28T05:13:32.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.466+0000 7f590c9f4700 1 Processor -- start 2024-03-28T05:13:32.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.466+0000 7f590c9f4700 1 -- start start 2024-03-28T05:13:32.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.467+0000 7f590c9f4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5908108510 0x7f590819b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:32.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.467+0000 7f590c9f4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590810bda0 con 0x7f5908108510 2024-03-28T05:13:32.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.467+0000 7f590659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5908108510 0x7f590819b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:32.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.467+0000 7f590659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5908108510 0x7f590819b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43614/0 (socket says 172.21.15.23:43614) 2024-03-28T05:13:32.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.467+0000 7f590659c700 1 -- 172.21.15.23:0/460550753 learned_addr learned my addr 172.21.15.23:0/460550753 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:32.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.467+0000 7f590659c700 1 -- 172.21.15.23:0/460550753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f590819bec0 con 0x7f5908108510 2024-03-28T05:13:32.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.467+0000 7f590659c700 1 --2- 172.21.15.23:0/460550753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5908108510 0x7f590819b980 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f58f8009fb0 tx=0x7f58f8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:32.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.467+0000 7f58f77fe700 1 -- 172.21.15.23:0/460550753 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58f80038b0 con 0x7f5908108510 2024-03-28T05:13:32.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.467+0000 7f590c9f4700 1 -- 172.21.15.23:0/460550753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f590819c150 con 0x7f5908108510 2024-03-28T05:13:32.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.467+0000 7f58f77fe700 1 -- 172.21.15.23:0/460550753 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58f8011e50 con 0x7f5908108510 2024-03-28T05:13:32.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.467+0000 7f58f77fe700 1 -- 172.21.15.23:0/460550753 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58f8019420 con 0x7f5908108510 2024-03-28T05:13:32.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.467+0000 7f590c9f4700 1 -- 172.21.15.23:0/460550753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f590819ee40 con 0x7f5908108510 2024-03-28T05:13:32.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.468+0000 7f58f77fe700 1 -- 172.21.15.23:0/460550753 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f58f8019740 con 0x7f5908108510 2024-03-28T05:13:32.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.468+0000 7f590c9f4700 1 -- 172.21.15.23:0/460550753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5908069220 con 0x7f5908108510 2024-03-28T05:13:32.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.468+0000 7f58f77fe700 1 --2- 172.21.15.23:0/460550753 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f58f0038e80 0x7f58f003b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:32.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.469+0000 7f58f77fe700 1 -- 172.21.15.23:0/460550753 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f58f804d020 con 0x7f5908108510 2024-03-28T05:13:32.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.469+0000 7f5905d9b700 1 --2- 172.21.15.23:0/460550753 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f58f0038e80 0x7f58f003b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:32.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.469+0000 7f5905d9b700 1 --2- 172.21.15.23:0/460550753 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f58f0038e80 0x7f58f003b340 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f58fc006fd0 tx=0x7f58fc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:32.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.474+0000 7f58f77fe700 1 -- 172.21.15.23:0/460550753 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f58f800ba30 con 0x7f5908108510 2024-03-28T05:13:32.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.760+0000 7f590c9f4700 1 -- 172.21.15.23:0/460550753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f590819f010 con 0x7f5908108510 2024-03-28T05:13:32.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.760+0000 7f58f77fe700 1 -- 172.21.15.23:0/460550753 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f590819f010 con 0x7f5908108510 2024-03-28T05:13:32.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.763+0000 7f590c9f4700 1 -- 172.21.15.23:0/460550753 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f58f0038e80 msgr2=0x7f58f003b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:32.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.763+0000 7f590c9f4700 1 --2- 172.21.15.23:0/460550753 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f58f0038e80 0x7f58f003b340 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f58fc006fd0 tx=0x7f58fc006e40 comp rx=0 tx=0).stop 2024-03-28T05:13:32.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.763+0000 7f590c9f4700 1 -- 172.21.15.23:0/460550753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5908108510 msgr2=0x7f590819b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:32.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.763+0000 7f590c9f4700 1 --2- 172.21.15.23:0/460550753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5908108510 0x7f590819b980 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f58f8009fb0 tx=0x7f58f8011bf0 comp rx=0 tx=0).stop 2024-03-28T05:13:32.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.763+0000 7f590c9f4700 1 -- 172.21.15.23:0/460550753 shutdown_connections 2024-03-28T05:13:32.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.763+0000 7f590c9f4700 1 --2- 172.21.15.23:0/460550753 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f58f0038e80 0x7f58f003b340 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:32.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.763+0000 7f590c9f4700 1 --2- 172.21.15.23:0/460550753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5908108510 0x7f590819b980 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:32.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.763+0000 7f590c9f4700 1 -- 172.21.15.23:0/460550753 >> 172.21.15.23:0/460550753 conn(0x7f5908101f40 msgr2=0x7f5908102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:32.317 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.763+0000 7f590c9f4700 1 -- 172.21.15.23:0/460550753 shutdown_connections 2024-03-28T05:13:32.317 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:31.764+0000 7f590c9f4700 1 -- 172.21.15.23:0/460550753 wait complete. 2024-03-28T05:13:32.317 INFO:teuthology.orchestra.run.smithi023.stdout:mgr is available 2024-03-28T05:13:33.277 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:32 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/828143337' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-28T05:13:33.629 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T05:13:33.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [global] 2024-03-28T05:13:33.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout fsid = bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:13:33.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-28T05:13:33.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-28T05:13:33.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-28T05:13:33.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-28T05:13:33.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T05:13:33.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-28T05:13:33.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-28T05:13:33.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T05:13:33.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [osd] 2024-03-28T05:13:33.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-28T05:13:33.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-28T05:13:33.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-28T05:13:33.631 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-28T05:13:33.631 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-28T05:13:33.631 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.714+0000 7f6a5a58f700 1 Processor -- start 2024-03-28T05:13:33.631 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.714+0000 7f6a5a58f700 1 -- start start 2024-03-28T05:13:33.631 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.714+0000 7f6a5a58f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a54107ac0 0x7f6a54107ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:33.631 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.714+0000 7f6a5a58f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a541084b0 con 0x7f6a54107ac0 2024-03-28T05:13:33.631 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.714+0000 7f6a53fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a54107ac0 0x7f6a54107ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:33.631 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.714+0000 7f6a53fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a54107ac0 0x7f6a54107ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43624/0 (socket says 172.21.15.23:43624) 2024-03-28T05:13:33.631 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.714+0000 7f6a53fff700 1 -- 172.21.15.23:0/1416540419 learned_addr learned my addr 172.21.15.23:0/1416540419 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:33.631 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.715+0000 7f6a53fff700 1 -- 172.21.15.23:0/1416540419 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a54108c70 con 0x7f6a54107ac0 2024-03-28T05:13:33.631 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.715+0000 7f6a53fff700 1 --2- 172.21.15.23:0/1416540419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a54107ac0 0x7f6a54107ee0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6a440098a0 tx=0x7f6a44009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98cee3660a30ab4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:33.631 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.715+0000 7f6a52ffd700 1 -- 172.21.15.23:0/1416540419 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a44004160 con 0x7f6a54107ac0 2024-03-28T05:13:33.631 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.715+0000 7f6a52ffd700 1 -- 172.21.15.23:0/1416540419 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a44004300 con 0x7f6a54107ac0 2024-03-28T05:13:33.631 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.715+0000 7f6a52ffd700 1 -- 172.21.15.23:0/1416540419 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a44011b50 con 0x7f6a54107ac0 2024-03-28T05:13:33.631 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.716+0000 7f6a5a58f700 1 -- 172.21.15.23:0/1416540419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a54107ac0 msgr2=0x7f6a54107ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:33.631 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.716+0000 7f6a5a58f700 1 --2- 172.21.15.23:0/1416540419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a54107ac0 0x7f6a54107ee0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6a440098a0 tx=0x7f6a44009bb0 comp rx=0 tx=0).stop 2024-03-28T05:13:33.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.716+0000 7f6a5a58f700 1 -- 172.21.15.23:0/1416540419 shutdown_connections 2024-03-28T05:13:33.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.716+0000 7f6a5a58f700 1 --2- 172.21.15.23:0/1416540419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a54107ac0 0x7f6a54107ee0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:33.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.716+0000 7f6a5a58f700 1 -- 172.21.15.23:0/1416540419 >> 172.21.15.23:0/1416540419 conn(0x7f6a541030c0 msgr2=0x7f6a541054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:33.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.716+0000 7f6a5a58f700 1 -- 172.21.15.23:0/1416540419 shutdown_connections 2024-03-28T05:13:33.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.716+0000 7f6a5a58f700 1 -- 172.21.15.23:0/1416540419 wait complete. 2024-03-28T05:13:33.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.716+0000 7f6a5a58f700 1 Processor -- start 2024-03-28T05:13:33.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.716+0000 7f6a5a58f700 1 -- start start 2024-03-28T05:13:33.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.717+0000 7f6a5a58f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a54107ac0 0x7f6a5419ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:33.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.717+0000 7f6a5a58f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5410c020 con 0x7f6a54107ac0 2024-03-28T05:13:33.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.717+0000 7f6a53fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a54107ac0 0x7f6a5419ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:33.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.717+0000 7f6a53fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a54107ac0 0x7f6a5419ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43634/0 (socket says 172.21.15.23:43634) 2024-03-28T05:13:33.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.717+0000 7f6a53fff700 1 -- 172.21.15.23:0/828143337 learned_addr learned my addr 172.21.15.23:0/828143337 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:33.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.717+0000 7f6a53fff700 1 -- 172.21.15.23:0/828143337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a5419bf70 con 0x7f6a54107ac0 2024-03-28T05:13:33.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.717+0000 7f6a53fff700 1 --2- 172.21.15.23:0/828143337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a54107ac0 0x7f6a5419ba30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6a44011850 tx=0x7f6a44011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:33.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.717+0000 7f6a517fa700 1 -- 172.21.15.23:0/828143337 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a44003af0 con 0x7f6a54107ac0 2024-03-28T05:13:33.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.717+0000 7f6a517fa700 1 -- 172.21.15.23:0/828143337 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a44020070 con 0x7f6a54107ac0 2024-03-28T05:13:33.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.717+0000 7f6a517fa700 1 -- 172.21.15.23:0/828143337 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a440195a0 con 0x7f6a54107ac0 2024-03-28T05:13:33.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.717+0000 7f6a5a58f700 1 -- 172.21.15.23:0/828143337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a5419c200 con 0x7f6a54107ac0 2024-03-28T05:13:33.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.717+0000 7f6a5a58f700 1 -- 172.21.15.23:0/828143337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a5419eef0 con 0x7f6a54107ac0 2024-03-28T05:13:33.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.718+0000 7f6a517fa700 1 -- 172.21.15.23:0/828143337 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f6a44019850 con 0x7f6a54107ac0 2024-03-28T05:13:33.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.718+0000 7f6a5a58f700 1 -- 172.21.15.23:0/828143337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a54069220 con 0x7f6a54107ac0 2024-03-28T05:13:33.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.718+0000 7f6a517fa700 1 --2- 172.21.15.23:0/828143337 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f6a3c038f20 0x7f6a3c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:33.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.718+0000 7f6a517fa700 1 -- 172.21.15.23:0/828143337 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6a44011e00 con 0x7f6a54107ac0 2024-03-28T05:13:33.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.718+0000 7f6a537fe700 1 --2- 172.21.15.23:0/828143337 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f6a3c038f20 0x7f6a3c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:33.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.719+0000 7f6a537fe700 1 --2- 172.21.15.23:0/828143337 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f6a3c038f20 0x7f6a3c03b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6a48006fd0 tx=0x7f6a48006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:33.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.725+0000 7f6a517fa700 1 -- 172.21.15.23:0/828143337 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6a44003c90 con 0x7f6a54107ac0 2024-03-28T05:13:33.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.927+0000 7f6a5a58f700 1 -- 172.21.15.23:0/828143337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f6a5419f0c0 con 0x7f6a54107ac0 2024-03-28T05:13:33.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.928+0000 7f6a517fa700 1 -- 172.21.15.23:0/828143337 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f6a5419f0c0 con 0x7f6a54107ac0 2024-03-28T05:13:33.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.930+0000 7f6a5a58f700 1 -- 172.21.15.23:0/828143337 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f6a3c038f20 msgr2=0x7f6a3c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:33.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.930+0000 7f6a5a58f700 1 --2- 172.21.15.23:0/828143337 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f6a3c038f20 0x7f6a3c03b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6a48006fd0 tx=0x7f6a48006e40 comp rx=0 tx=0).stop 2024-03-28T05:13:33.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.930+0000 7f6a5a58f700 1 -- 172.21.15.23:0/828143337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a54107ac0 msgr2=0x7f6a5419ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:33.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.930+0000 7f6a5a58f700 1 --2- 172.21.15.23:0/828143337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a54107ac0 0x7f6a5419ba30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6a44011850 tx=0x7f6a44011c10 comp rx=0 tx=0).stop 2024-03-28T05:13:33.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.930+0000 7f6a5a58f700 1 -- 172.21.15.23:0/828143337 shutdown_connections 2024-03-28T05:13:33.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.930+0000 7f6a5a58f700 1 --2- 172.21.15.23:0/828143337 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f6a3c038f20 0x7f6a3c03b3e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:33.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.930+0000 7f6a5a58f700 1 --2- 172.21.15.23:0/828143337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a54107ac0 0x7f6a5419ba30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:33.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.930+0000 7f6a5a58f700 1 -- 172.21.15.23:0/828143337 >> 172.21.15.23:0/828143337 conn(0x7f6a541030c0 msgr2=0x7f6a54103ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:33.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.930+0000 7f6a5a58f700 1 -- 172.21.15.23:0/828143337 shutdown_connections 2024-03-28T05:13:33.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:32.930+0000 7f6a5a58f700 1 -- 172.21.15.23:0/828143337 wait complete. 2024-03-28T05:13:33.634 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling cephadm module... 2024-03-28T05:13:35.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:35 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/230770666' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-28T05:13:35.778 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:35 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: ignoring --setuser ceph since I am not root 2024-03-28T05:13:35.778 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:35 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: ignoring --setgroup ceph since I am not root 2024-03-28T05:13:36.277 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:36 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:36.149+0000 7f4bc4c3c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T05:13:36.278 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:36 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:36.245+0000 7f4bc4c3c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T05:13:36.899 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:36 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/230770666' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-28T05:13:36.900 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:36 smithi023 ceph-mon[31954]: mgrmap e5: x(active, since 8s) 2024-03-28T05:13:37.114 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.251+0000 7f9b37325700 1 Processor -- start 2024-03-28T05:13:37.114 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.251+0000 7f9b37325700 1 -- start start 2024-03-28T05:13:37.114 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.251+0000 7f9b37325700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b30108520 0x7f9b3010a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:37.114 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.251+0000 7f9b37325700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3007ae70 con 0x7f9b30108520 2024-03-28T05:13:37.114 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.252+0000 7f9b350c1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b30108520 0x7f9b3010a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:37.114 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.252+0000 7f9b350c1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b30108520 0x7f9b3010a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42186/0 (socket says 172.21.15.23:42186) 2024-03-28T05:13:37.114 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.252+0000 7f9b350c1700 1 -- 172.21.15.23:0/1196344417 learned_addr learned my addr 172.21.15.23:0/1196344417 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:37.114 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.252+0000 7f9b350c1700 1 -- 172.21.15.23:0/1196344417 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3010ae50 con 0x7f9b30108520 2024-03-28T05:13:37.115 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.252+0000 7f9b350c1700 1 --2- 172.21.15.23:0/1196344417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b30108520 0x7f9b3010a910 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9b200098a0 tx=0x7f9b20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df5e7586e77583a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:37.115 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.252+0000 7f9b27fff700 1 -- 172.21.15.23:0/1196344417 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9b20004160 con 0x7f9b30108520 2024-03-28T05:13:37.115 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.252+0000 7f9b27fff700 1 -- 172.21.15.23:0/1196344417 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b20004300 con 0x7f9b30108520 2024-03-28T05:13:37.115 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.252+0000 7f9b27fff700 1 -- 172.21.15.23:0/1196344417 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9b20011b50 con 0x7f9b30108520 2024-03-28T05:13:37.115 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.253+0000 7f9b37325700 1 -- 172.21.15.23:0/1196344417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b30108520 msgr2=0x7f9b3010a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:37.115 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.253+0000 7f9b37325700 1 --2- 172.21.15.23:0/1196344417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b30108520 0x7f9b3010a910 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9b200098a0 tx=0x7f9b20009bb0 comp rx=0 tx=0).stop 2024-03-28T05:13:37.115 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.253+0000 7f9b37325700 1 -- 172.21.15.23:0/1196344417 shutdown_connections 2024-03-28T05:13:37.115 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.253+0000 7f9b37325700 1 --2- 172.21.15.23:0/1196344417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b30108520 0x7f9b3010a910 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:37.115 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.253+0000 7f9b37325700 1 -- 172.21.15.23:0/1196344417 >> 172.21.15.23:0/1196344417 conn(0x7f9b30102010 msgr2=0x7f9b30104430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:37.115 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.253+0000 7f9b37325700 1 -- 172.21.15.23:0/1196344417 shutdown_connections 2024-03-28T05:13:37.115 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.253+0000 7f9b37325700 1 -- 172.21.15.23:0/1196344417 wait complete. 2024-03-28T05:13:37.115 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.253+0000 7f9b37325700 1 Processor -- start 2024-03-28T05:13:37.116 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.254+0000 7f9b37325700 1 -- start start 2024-03-28T05:13:37.116 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.254+0000 7f9b37325700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b30108520 0x7f9b3019b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:37.116 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.254+0000 7f9b37325700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3010bdb0 con 0x7f9b30108520 2024-03-28T05:13:37.116 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.254+0000 7f9b350c1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b30108520 0x7f9b3019b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:37.116 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.254+0000 7f9b350c1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b30108520 0x7f9b3019b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42192/0 (socket says 172.21.15.23:42192) 2024-03-28T05:13:37.116 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.254+0000 7f9b350c1700 1 -- 172.21.15.23:0/230770666 learned_addr learned my addr 172.21.15.23:0/230770666 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:37.116 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.254+0000 7f9b350c1700 1 -- 172.21.15.23:0/230770666 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3019bf20 con 0x7f9b30108520 2024-03-28T05:13:37.116 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.254+0000 7f9b350c1700 1 --2- 172.21.15.23:0/230770666 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b30108520 0x7f9b3019b9e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9b20009fb0 tx=0x7f9b20011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:37.116 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.254+0000 7f9b267fc700 1 -- 172.21.15.23:0/230770666 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9b200038b0 con 0x7f9b30108520 2024-03-28T05:13:37.116 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.254+0000 7f9b267fc700 1 -- 172.21.15.23:0/230770666 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b20011e50 con 0x7f9b30108520 2024-03-28T05:13:37.116 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.254+0000 7f9b267fc700 1 -- 172.21.15.23:0/230770666 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9b20019420 con 0x7f9b30108520 2024-03-28T05:13:37.116 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.254+0000 7f9b37325700 1 -- 172.21.15.23:0/230770666 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b3019c1b0 con 0x7f9b30108520 2024-03-28T05:13:37.117 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.254+0000 7f9b37325700 1 -- 172.21.15.23:0/230770666 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b3019eea0 con 0x7f9b30108520 2024-03-28T05:13:37.117 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.256+0000 7f9b267fc700 1 -- 172.21.15.23:0/230770666 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f9b20003a50 con 0x7f9b30108520 2024-03-28T05:13:37.117 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.256+0000 7f9b37325700 1 -- 172.21.15.23:0/230770666 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b300637c0 con 0x7f9b30108520 2024-03-28T05:13:37.117 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.256+0000 7f9b267fc700 1 --2- 172.21.15.23:0/230770666 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f9b1c038ed0 0x7f9b1c03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:37.117 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.256+0000 7f9b267fc700 1 -- 172.21.15.23:0/230770666 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9b2004d040 con 0x7f9b30108520 2024-03-28T05:13:37.117 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.256+0000 7f9b348c0700 1 --2- 172.21.15.23:0/230770666 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f9b1c038ed0 0x7f9b1c03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:37.117 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.256+0000 7f9b348c0700 1 --2- 172.21.15.23:0/230770666 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f9b1c038ed0 0x7f9b1c03b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c006fd0 tx=0x7f9b2c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:37.117 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.261+0000 7f9b267fc700 1 -- 172.21.15.23:0/230770666 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9b2004ca60 con 0x7f9b30108520 2024-03-28T05:13:37.117 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.529+0000 7f9b37325700 1 -- 172.21.15.23:0/230770666 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f9b3019f070 con 0x7f9b30108520 2024-03-28T05:13:37.117 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.581+0000 7f9b267fc700 1 -- 172.21.15.23:0/230770666 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f9b3019f070 con 0x7f9b30108520 2024-03-28T05:13:37.117 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.583+0000 7f9b267fc700 1 -- 172.21.15.23:0/230770666 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f9b20023390 con 0x7f9b30108520 2024-03-28T05:13:37.117 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.584+0000 7f9b37325700 1 -- 172.21.15.23:0/230770666 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f9b1c038ed0 msgr2=0x7f9b1c03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:37.118 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.584+0000 7f9b37325700 1 --2- 172.21.15.23:0/230770666 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f9b1c038ed0 0x7f9b1c03b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c006fd0 tx=0x7f9b2c006e40 comp rx=0 tx=0).stop 2024-03-28T05:13:37.118 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.584+0000 7f9b37325700 1 -- 172.21.15.23:0/230770666 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b30108520 msgr2=0x7f9b3019b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:37.118 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.584+0000 7f9b37325700 1 --2- 172.21.15.23:0/230770666 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b30108520 0x7f9b3019b9e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9b20009fb0 tx=0x7f9b20011bf0 comp rx=0 tx=0).stop 2024-03-28T05:13:37.118 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.584+0000 7f9b37325700 1 -- 172.21.15.23:0/230770666 shutdown_connections 2024-03-28T05:13:37.118 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.584+0000 7f9b37325700 1 --2- 172.21.15.23:0/230770666 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f9b1c038ed0 0x7f9b1c03b390 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:37.118 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.584+0000 7f9b37325700 1 --2- 172.21.15.23:0/230770666 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b30108520 0x7f9b3019b9e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:37.118 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.584+0000 7f9b37325700 1 -- 172.21.15.23:0/230770666 >> 172.21.15.23:0/230770666 conn(0x7f9b30102010 msgr2=0x7f9b30102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:37.118 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.584+0000 7f9b37325700 1 -- 172.21.15.23:0/230770666 shutdown_connections 2024-03-28T05:13:37.118 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:35.584+0000 7f9b37325700 1 -- 172.21.15.23:0/230770666 wait complete. 2024-03-28T05:13:37.237 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:36 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:36.904+0000 7f4bc4c3c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T05:13:37.237 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:36 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:36.998+0000 7f4bc4c3c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T05:13:37.237 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:37 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:37.148+0000 7f4bc4c3c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T05:13:37.508 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:37 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:37.247+0000 7f4bc4c3c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T05:13:37.508 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:37 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:37.332+0000 7f4bc4c3c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T05:13:37.777 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:37 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:37.606+0000 7f4bc4c3c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T05:13:37.778 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:37 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:37.718+0000 7f4bc4c3c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T05:13:38.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:37 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3281770422' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-28T05:13:38.088 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:37 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:37.806+0000 7f4bc4c3c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T05:13:38.089 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:37 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:37.914+0000 7f4bc4c3c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T05:13:38.089 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:38 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:38.092+0000 7f4bc4c3c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T05:13:38.322 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-28T05:13:38.322 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-28T05:13:38.322 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T05:13:38.322 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-28T05:13:38.322 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-28T05:13:38.323 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-28T05:13:38.323 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.531+0000 7f3fff887700 1 Processor -- start 2024-03-28T05:13:38.323 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.531+0000 7f3fff887700 1 -- start start 2024-03-28T05:13:38.323 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.532+0000 7f3fff887700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ff807ce70 0x7f3ff807b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:38.323 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.532+0000 7f3fff887700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff807b8f0 con 0x7f3ff807ce70 2024-03-28T05:13:38.323 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.532+0000 7f3ffd623700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ff807ce70 0x7f3ff807b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:38.323 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.532+0000 7f3ffd623700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ff807ce70 0x7f3ff807b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42222/0 (socket says 172.21.15.23:42222) 2024-03-28T05:13:38.323 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.532+0000 7f3ffd623700 1 -- 172.21.15.23:0/838234783 learned_addr learned my addr 172.21.15.23:0/838234783 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:38.323 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.532+0000 7f3ffd623700 1 -- 172.21.15.23:0/838234783 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff807ba70 con 0x7f3ff807ce70 2024-03-28T05:13:38.323 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.532+0000 7f3ffd623700 1 --2- 172.21.15.23:0/838234783 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ff807ce70 0x7f3ff807b320 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3fec0098a0 tx=0x7f3fec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38ed07e2bf8e2e4f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:38.323 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.532+0000 7f3febfff700 1 -- 172.21.15.23:0/838234783 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3fec004160 con 0x7f3ff807ce70 2024-03-28T05:13:38.323 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.533+0000 7f3febfff700 1 -- 172.21.15.23:0/838234783 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3fec004300 con 0x7f3ff807ce70 2024-03-28T05:13:38.323 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.533+0000 7f3febfff700 1 -- 172.21.15.23:0/838234783 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3fec011b50 con 0x7f3ff807ce70 2024-03-28T05:13:38.323 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.533+0000 7f3fff887700 1 -- 172.21.15.23:0/838234783 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ff807ce70 msgr2=0x7f3ff807b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:38.323 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.533+0000 7f3fff887700 1 --2- 172.21.15.23:0/838234783 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ff807ce70 0x7f3ff807b320 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3fec0098a0 tx=0x7f3fec009bb0 comp rx=0 tx=0).stop 2024-03-28T05:13:38.323 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.533+0000 7f3fff887700 1 -- 172.21.15.23:0/838234783 shutdown_connections 2024-03-28T05:13:38.324 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.533+0000 7f3fff887700 1 --2- 172.21.15.23:0/838234783 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ff807ce70 0x7f3ff807b320 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:38.324 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.533+0000 7f3fff887700 1 -- 172.21.15.23:0/838234783 >> 172.21.15.23:0/838234783 conn(0x7f3ff8103090 msgr2=0x7f3ff81054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:38.324 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.533+0000 7f3fff887700 1 -- 172.21.15.23:0/838234783 shutdown_connections 2024-03-28T05:13:38.324 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.533+0000 7f3fff887700 1 -- 172.21.15.23:0/838234783 wait complete. 2024-03-28T05:13:38.324 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.534+0000 7f3fff887700 1 Processor -- start 2024-03-28T05:13:38.324 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.534+0000 7f3fff887700 1 -- start start 2024-03-28T05:13:38.324 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.534+0000 7f3fff887700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ff807ce70 0x7f3ff819db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:38.324 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.534+0000 7f3fff887700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff810e000 con 0x7f3ff807ce70 2024-03-28T05:13:38.324 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.534+0000 7f3ffd623700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ff807ce70 0x7f3ff819db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:38.324 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.534+0000 7f3ffd623700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ff807ce70 0x7f3ff819db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42226/0 (socket says 172.21.15.23:42226) 2024-03-28T05:13:38.324 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.534+0000 7f3ffd623700 1 -- 172.21.15.23:0/3281770422 learned_addr learned my addr 172.21.15.23:0/3281770422 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:38.324 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.535+0000 7f3ffd623700 1 -- 172.21.15.23:0/3281770422 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff819e0d0 con 0x7f3ff807ce70 2024-03-28T05:13:38.324 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.535+0000 7f3ffd623700 1 --2- 172.21.15.23:0/3281770422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ff807ce70 0x7f3ff819db90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3fec011b60 tx=0x7f3fec011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:38.324 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.535+0000 7f3fea7fc700 1 -- 172.21.15.23:0/3281770422 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3fec003b20 con 0x7f3ff807ce70 2024-03-28T05:13:38.324 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.535+0000 7f3fff887700 1 -- 172.21.15.23:0/3281770422 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ff819e360 con 0x7f3ff807ce70 2024-03-28T05:13:38.325 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.535+0000 7f3fea7fc700 1 -- 172.21.15.23:0/3281770422 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3fec003cc0 con 0x7f3ff807ce70 2024-03-28T05:13:38.325 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.535+0000 7f3fea7fc700 1 -- 172.21.15.23:0/3281770422 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3fec024840 con 0x7f3ff807ce70 2024-03-28T05:13:38.325 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.535+0000 7f3fff887700 1 -- 172.21.15.23:0/3281770422 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ff81a1050 con 0x7f3ff807ce70 2024-03-28T05:13:38.325 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.536+0000 7f3fea7fc700 1 -- 172.21.15.23:0/3281770422 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f3fec022020 con 0x7f3ff807ce70 2024-03-28T05:13:38.325 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.536+0000 7f3fff887700 1 -- 172.21.15.23:0/3281770422 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ff8069220 con 0x7f3ff807ce70 2024-03-28T05:13:38.325 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.536+0000 7f3fea7fc700 1 --2- 172.21.15.23:0/3281770422 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f3fe4038f20 0x7f3fe403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:38.325 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.536+0000 7f3fea7fc700 1 -- 172.21.15.23:0/3281770422 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3fec011e00 con 0x7f3ff807ce70 2024-03-28T05:13:38.325 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.536+0000 7f3ffce22700 1 -- 172.21.15.23:0/3281770422 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f3fe4038f20 msgr2=0x7f3fe403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1735221490 2024-03-28T05:13:38.325 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.536+0000 7f3ffce22700 1 --2- 172.21.15.23:0/3281770422 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f3fe4038f20 0x7f3fe403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T05:13:38.325 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.541+0000 7f3fea7fc700 1 -- 172.21.15.23:0/3281770422 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3fec019c70 con 0x7f3ff807ce70 2024-03-28T05:13:38.325 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.737+0000 7f3ffce22700 1 -- 172.21.15.23:0/3281770422 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f3fe4038f20 msgr2=0x7f3fe403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1735221490 2024-03-28T05:13:38.325 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.737+0000 7f3ffce22700 1 --2- 172.21.15.23:0/3281770422 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f3fe4038f20 0x7f3fe403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T05:13:38.325 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.808+0000 7f3fff887700 1 -- 172.21.15.23:0/3281770422 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f3ff81a1220 con 0x7f3ff807ce70 2024-03-28T05:13:38.325 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.808+0000 7f3fea7fc700 1 -- 172.21.15.23:0/3281770422 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f3ff81a1220 con 0x7f3ff807ce70 2024-03-28T05:13:38.325 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.812+0000 7f3fff887700 1 -- 172.21.15.23:0/3281770422 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f3fe4038f20 msgr2=0x7f3fe403b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T05:13:38.325 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.812+0000 7f3fff887700 1 --2- 172.21.15.23:0/3281770422 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f3fe4038f20 0x7f3fe403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:38.326 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.812+0000 7f3fff887700 1 -- 172.21.15.23:0/3281770422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ff807ce70 msgr2=0x7f3ff819db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:38.326 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.812+0000 7f3fff887700 1 --2- 172.21.15.23:0/3281770422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ff807ce70 0x7f3ff819db90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3fec011b60 tx=0x7f3fec011c40 comp rx=0 tx=0).stop 2024-03-28T05:13:38.326 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.812+0000 7f3fff887700 1 -- 172.21.15.23:0/3281770422 shutdown_connections 2024-03-28T05:13:38.326 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.812+0000 7f3fff887700 1 --2- 172.21.15.23:0/3281770422 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7f3fe4038f20 0x7f3fe403b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:38.326 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.812+0000 7f3fff887700 1 --2- 172.21.15.23:0/3281770422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ff807ce70 0x7f3ff819db90 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:38.326 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.812+0000 7f3fff887700 1 -- 172.21.15.23:0/3281770422 >> 172.21.15.23:0/3281770422 conn(0x7f3ff8103090 msgr2=0x7f3ff8103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:38.326 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.812+0000 7f3fff887700 1 -- 172.21.15.23:0/3281770422 shutdown_connections 2024-03-28T05:13:38.326 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:37.812+0000 7f3fff887700 1 -- 172.21.15.23:0/3281770422 wait complete. 2024-03-28T05:13:38.326 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for the mgr to restart... 2024-03-28T05:13:38.326 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr epoch 5... 2024-03-28T05:13:39.528 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:39 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:39.099+0000 7f4bc4c3c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T05:13:39.528 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:39 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:39.232+0000 7f4bc4c3c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T05:13:39.855 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:39 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:39.568+0000 7f4bc4c3c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T05:13:39.856 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:39 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:39.659+0000 7f4bc4c3c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T05:13:39.856 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:39 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:39.860+0000 7f4bc4c3c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T05:13:40.134 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:40 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:40.049+0000 7f4bc4c3c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T05:13:40.527 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:40 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:40.339+0000 7f4bc4c3c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T05:13:41.027 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:40 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:40.599+0000 7f4bc4c3c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T05:13:41.028 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:40 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:40.841+0000 7f4bc4c3c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T05:13:42.027 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:41 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:41.667+0000 7f4bc4c3c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T05:13:42.527 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:42 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:42.245+0000 7f4bc4c3c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T05:13:42.528 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:42 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:42.345+0000 7f4bc4c3c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T05:13:43.028 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:42 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:13:42.566+0000 7f4bc4c3c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T05:13:43.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:42 smithi023 ceph-mon[31954]: Active manager daemon x restarted 2024-03-28T05:13:43.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:42 smithi023 ceph-mon[31954]: Activating manager daemon x 2024-03-28T05:13:43.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:42 smithi023 ceph-mon[31954]: osdmap e2: 0 total, 0 up, 0 in 2024-03-28T05:13:43.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:42 smithi023 ceph-mon[31954]: mgrmap e6: x(active, starting, since 0.00352297s) 2024-03-28T05:13:43.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:42 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T05:13:43.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:42 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T05:13:43.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:42 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T05:13:43.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:42 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T05:13:43.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:42 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T05:13:43.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:42 smithi023 ceph-mon[31954]: Manager daemon x is now available 2024-03-28T05:13:43.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:42 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' 2024-03-28T05:13:43.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:42 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' 2024-03-28T05:13:43.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:42 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:13:43.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:42 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:13:44.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:43 smithi023 ceph-mon[31954]: Found migration_current of "None". Setting to last migration. 2024-03-28T05:13:44.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:43 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:13:44.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:43 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T05:13:44.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:43 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T05:13:44.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:43 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' 2024-03-28T05:13:44.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:43 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' 2024-03-28T05:13:44.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:43 smithi023 ceph-mon[31954]: mgrmap e7: x(active, since 1.01006s) 2024-03-28T05:13:44.040 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-28T05:13:44.040 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-28T05:13:44.040 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-28T05:13:44.040 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-28T05:13:44.040 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.731+0000 7efc046df700 1 Processor -- start 2024-03-28T05:13:44.040 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.731+0000 7efc046df700 1 -- start start 2024-03-28T05:13:44.040 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.731+0000 7efc046df700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbfc1084f0 0x7efbfc10a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:44.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.731+0000 7efc046df700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc07ae10 con 0x7efbfc1084f0 2024-03-28T05:13:44.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.731+0000 7efc0247b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbfc1084f0 0x7efbfc10a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:44.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.731+0000 7efc0247b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbfc1084f0 0x7efbfc10a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42240/0 (socket says 172.21.15.23:42240) 2024-03-28T05:13:44.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.731+0000 7efc0247b700 1 -- 172.21.15.23:0/1364027788 learned_addr learned my addr 172.21.15.23:0/1364027788 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:44.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.732+0000 7efc0247b700 1 -- 172.21.15.23:0/1364027788 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbfc10ae20 con 0x7efbfc1084f0 2024-03-28T05:13:44.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.732+0000 7efc0247b700 1 --2- 172.21.15.23:0/1364027788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbfc1084f0 0x7efbfc10a8e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efbec0098a0 tx=0x7efbec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29c3e4ead0c4d96c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:44.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.732+0000 7efc01479700 1 -- 172.21.15.23:0/1364027788 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efbec004160 con 0x7efbfc1084f0 2024-03-28T05:13:44.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.732+0000 7efc01479700 1 -- 172.21.15.23:0/1364027788 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efbec004300 con 0x7efbfc1084f0 2024-03-28T05:13:44.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.732+0000 7efc01479700 1 -- 172.21.15.23:0/1364027788 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efbec011b50 con 0x7efbfc1084f0 2024-03-28T05:13:44.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.733+0000 7efc046df700 1 -- 172.21.15.23:0/1364027788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbfc1084f0 msgr2=0x7efbfc10a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:44.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.733+0000 7efc046df700 1 --2- 172.21.15.23:0/1364027788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbfc1084f0 0x7efbfc10a8e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efbec0098a0 tx=0x7efbec009bb0 comp rx=0 tx=0).stop 2024-03-28T05:13:44.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.733+0000 7efc046df700 1 -- 172.21.15.23:0/1364027788 shutdown_connections 2024-03-28T05:13:44.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.733+0000 7efc046df700 1 --2- 172.21.15.23:0/1364027788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbfc1084f0 0x7efbfc10a8e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:44.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.733+0000 7efc046df700 1 -- 172.21.15.23:0/1364027788 >> 172.21.15.23:0/1364027788 conn(0x7efbfc101f40 msgr2=0x7efbfc104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:44.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.733+0000 7efc046df700 1 -- 172.21.15.23:0/1364027788 shutdown_connections 2024-03-28T05:13:44.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.733+0000 7efc046df700 1 -- 172.21.15.23:0/1364027788 wait complete. 2024-03-28T05:13:44.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.734+0000 7efc046df700 1 Processor -- start 2024-03-28T05:13:44.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.734+0000 7efc046df700 1 -- start start 2024-03-28T05:13:44.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.734+0000 7efc046df700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbfc1084f0 0x7efbfc19b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:44.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.734+0000 7efc046df700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc10bd90 con 0x7efbfc1084f0 2024-03-28T05:13:44.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.734+0000 7efc0247b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbfc1084f0 0x7efbfc19b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:44.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.734+0000 7efc0247b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbfc1084f0 0x7efbfc19b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42246/0 (socket says 172.21.15.23:42246) 2024-03-28T05:13:44.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.734+0000 7efc0247b700 1 -- 172.21.15.23:0/1409273189 learned_addr learned my addr 172.21.15.23:0/1409273189 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:44.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.735+0000 7efc0247b700 1 -- 172.21.15.23:0/1409273189 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbfc19bed0 con 0x7efbfc1084f0 2024-03-28T05:13:44.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.735+0000 7efc0247b700 1 --2- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbfc1084f0 0x7efbfc19b990 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efbec009fb0 tx=0x7efbec011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:44.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.735+0000 7efbf37fe700 1 -- 172.21.15.23:0/1409273189 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efbec0038b0 con 0x7efbfc1084f0 2024-03-28T05:13:44.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.735+0000 7efbf37fe700 1 -- 172.21.15.23:0/1409273189 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efbec011e50 con 0x7efbfc1084f0 2024-03-28T05:13:44.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.735+0000 7efc046df700 1 -- 172.21.15.23:0/1409273189 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbfc19c160 con 0x7efbfc1084f0 2024-03-28T05:13:44.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.735+0000 7efbf37fe700 1 -- 172.21.15.23:0/1409273189 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efbec019420 con 0x7efbfc1084f0 2024-03-28T05:13:44.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.735+0000 7efc046df700 1 -- 172.21.15.23:0/1409273189 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbfc19ee50 con 0x7efbfc1084f0 2024-03-28T05:13:44.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.736+0000 7efbf37fe700 1 -- 172.21.15.23:0/1409273189 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7efbec019730 con 0x7efbfc1084f0 2024-03-28T05:13:44.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.736+0000 7efbf37fe700 1 --2- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7efbe8038f20 0x7efbe803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:44.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.736+0000 7efbf37fe700 1 -- 172.21.15.23:0/1409273189 --> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7efbe803baf0 con 0x7efbe8038f20 2024-03-28T05:13:44.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.736+0000 7efc01c7a700 1 -- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7efbe8038f20 msgr2=0x7efbe803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1735221490 2024-03-28T05:13:44.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.736+0000 7efbf37fe700 1 -- 172.21.15.23:0/1409273189 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7efbec04d060 con 0x7efbfc1084f0 2024-03-28T05:13:44.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.736+0000 7efc01c7a700 1 --2- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7efbe8038f20 0x7efbe803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T05:13:44.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.937+0000 7efc01c7a700 1 -- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7efbe8038f20 msgr2=0x7efbe803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1735221490 2024-03-28T05:13:44.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:38.937+0000 7efc01c7a700 1 --2- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7efbe8038f20 0x7efbe803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T05:13:44.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:39.338+0000 7efc01c7a700 1 -- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7efbe8038f20 msgr2=0x7efbe803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1735221490 2024-03-28T05:13:44.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:39.338+0000 7efc01c7a700 1 --2- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7efbe8038f20 0x7efbe803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T05:13:44.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:40.139+0000 7efc01c7a700 1 -- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7efbe8038f20 msgr2=0x7efbe803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1735221490 2024-03-28T05:13:44.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:40.139+0000 7efc01c7a700 1 --2- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7efbe8038f20 0x7efbe803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T05:13:44.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:41.740+0000 7efc01c7a700 1 -- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7efbe8038f20 msgr2=0x7efbe803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1735221490 2024-03-28T05:13:44.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:41.740+0000 7efc01c7a700 1 --2- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7efbe8038f20 0x7efbe803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T05:13:44.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:42.581+0000 7efbf37fe700 1 -- 172.21.15.23:0/1409273189 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mgrmap(e 6) v1 ==== 45873+0+0 (secure 0 0 0) 0x7efbec003a50 con 0x7efbfc1084f0 2024-03-28T05:13:44.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:42.581+0000 7efbf37fe700 1 -- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7efbe8038f20 msgr2=0x7efbe803b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T05:13:44.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:42.581+0000 7efbf37fe700 1 --2- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7efbe8038f20 0x7efbe803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:44.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:43.587+0000 7efbf37fe700 1 -- 172.21.15.23:0/1409273189 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7efbec02d7c0 con 0x7efbfc1084f0 2024-03-28T05:13:44.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:43.587+0000 7efbf37fe700 1 --2- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7efbe803c810 0x7efbe803ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:44.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:43.587+0000 7efbf37fe700 1 -- 172.21.15.23:0/1409273189 --> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7efbec012ea0 con 0x7efbe803c810 2024-03-28T05:13:44.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:43.587+0000 7efc01c7a700 1 --2- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7efbe803c810 0x7efbe803ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:44.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:43.588+0000 7efc01c7a700 1 --2- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7efbe803c810 0x7efbe803ec00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7efbf8003a10 tx=0x7efbf80092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:44.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:43.589+0000 7efbf37fe700 1 -- 172.21.15.23:0/1409273189 <== mgr.14118 v2:172.21.15.23:6800/1530849123 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7efbec012ea0 con 0x7efbe803c810 2024-03-28T05:13:44.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:43.595+0000 7efc046df700 1 -- 172.21.15.23:0/1409273189 --> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7efbfc069220 con 0x7efbe803c810 2024-03-28T05:13:44.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:43.595+0000 7efbf37fe700 1 -- 172.21.15.23:0/1409273189 <== mgr.14118 v2:172.21.15.23:6800/1530849123 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7efbfc069220 con 0x7efbe803c810 2024-03-28T05:13:44.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:43.596+0000 7efc046df700 1 -- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7efbe803c810 msgr2=0x7efbe803ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:44.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:43.596+0000 7efc046df700 1 --2- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7efbe803c810 0x7efbe803ec00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7efbf8003a10 tx=0x7efbf80092b0 comp rx=0 tx=0).stop 2024-03-28T05:13:44.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:43.596+0000 7efc046df700 1 -- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbfc1084f0 msgr2=0x7efbfc19b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:44.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:43.596+0000 7efc046df700 1 --2- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbfc1084f0 0x7efbfc19b990 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efbec009fb0 tx=0x7efbec011bf0 comp rx=0 tx=0).stop 2024-03-28T05:13:44.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:43.596+0000 7efc046df700 1 -- 172.21.15.23:0/1409273189 shutdown_connections 2024-03-28T05:13:44.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:43.596+0000 7efc046df700 1 --2- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7efbe803c810 0x7efbe803ec00 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:44.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:43.596+0000 7efc046df700 1 --2- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbfc1084f0 0x7efbfc19b990 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:44.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:43.596+0000 7efc046df700 1 --2- 172.21.15.23:0/1409273189 >> [v2:172.21.15.23:6800/1735221490,v1:172.21.15.23:6801/1735221490] conn(0x7efbe8038f20 0x7efbe803b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:44.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:43.596+0000 7efc046df700 1 -- 172.21.15.23:0/1409273189 >> 172.21.15.23:0/1409273189 conn(0x7efbfc101f40 msgr2=0x7efbfc102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:44.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:43.596+0000 7efc046df700 1 -- 172.21.15.23:0/1409273189 shutdown_connections 2024-03-28T05:13:44.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:43.596+0000 7efc046df700 1 -- 172.21.15.23:0/1409273189 wait complete. 2024-03-28T05:13:44.045 INFO:teuthology.orchestra.run.smithi023.stdout:mgr epoch 5 is available 2024-03-28T05:13:44.045 INFO:teuthology.orchestra.run.smithi023.stdout:Setting orchestrator backend to cephadm... 2024-03-28T05:13:44.964 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:44 smithi023 ceph-mon[31954]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-28T05:13:44.964 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:44 smithi023 ceph-mon[31954]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-28T05:13:44.964 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:44 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:13:45.181 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.432+0000 7f2854779700 1 Processor -- start 2024-03-28T05:13:45.181 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.432+0000 7f2854779700 1 -- start start 2024-03-28T05:13:45.181 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.432+0000 7f2854779700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f284c1068d0 0x7f284c106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:45.181 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.432+0000 7f2854779700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284c1072c0 con 0x7f284c1068d0 2024-03-28T05:13:45.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.433+0000 7f2852515700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f284c1068d0 0x7f284c106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:45.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.433+0000 7f2852515700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f284c1068d0 0x7f284c106cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43462/0 (socket says 172.21.15.23:43462) 2024-03-28T05:13:45.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.433+0000 7f2852515700 1 -- 172.21.15.23:0/549908680 learned_addr learned my addr 172.21.15.23:0/549908680 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:45.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.433+0000 7f2852515700 1 -- 172.21.15.23:0/549908680 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f284c107ad0 con 0x7f284c1068d0 2024-03-28T05:13:45.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.433+0000 7f2852515700 1 --2- 172.21.15.23:0/549908680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f284c1068d0 0x7f284c106cf0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f284400d180 tx=0x7f284400d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d2fd9fe0c62af12 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:45.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.433+0000 7f2851513700 1 -- 172.21.15.23:0/549908680 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2844019070 con 0x7f284c1068d0 2024-03-28T05:13:45.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.433+0000 7f2851513700 1 -- 172.21.15.23:0/549908680 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f284400de20 con 0x7f284c1068d0 2024-03-28T05:13:45.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.433+0000 7f2851513700 1 -- 172.21.15.23:0/549908680 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2844009b90 con 0x7f284c1068d0 2024-03-28T05:13:45.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.434+0000 7f2854779700 1 -- 172.21.15.23:0/549908680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f284c1068d0 msgr2=0x7f284c106cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:45.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.434+0000 7f2854779700 1 --2- 172.21.15.23:0/549908680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f284c1068d0 0x7f284c106cf0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f284400d180 tx=0x7f284400d490 comp rx=0 tx=0).stop 2024-03-28T05:13:45.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.434+0000 7f2854779700 1 -- 172.21.15.23:0/549908680 shutdown_connections 2024-03-28T05:13:45.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.434+0000 7f2854779700 1 --2- 172.21.15.23:0/549908680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f284c1068d0 0x7f284c106cf0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:45.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.434+0000 7f2854779700 1 -- 172.21.15.23:0/549908680 >> 172.21.15.23:0/549908680 conn(0x7f284c101f10 msgr2=0x7f284c104330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:45.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.434+0000 7f2854779700 1 -- 172.21.15.23:0/549908680 shutdown_connections 2024-03-28T05:13:45.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.434+0000 7f2854779700 1 -- 172.21.15.23:0/549908680 wait complete. 2024-03-28T05:13:45.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.434+0000 7f2854779700 1 Processor -- start 2024-03-28T05:13:45.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.434+0000 7f2854779700 1 -- start start 2024-03-28T05:13:45.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.435+0000 7f2854779700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f284c1068d0 0x7f284c1997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:45.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.435+0000 7f2854779700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284c079ff0 con 0x7f284c1068d0 2024-03-28T05:13:45.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.435+0000 7f2852515700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f284c1068d0 0x7f284c1997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:45.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.435+0000 7f2852515700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f284c1068d0 0x7f284c1997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43478/0 (socket says 172.21.15.23:43478) 2024-03-28T05:13:45.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.435+0000 7f2852515700 1 -- 172.21.15.23:0/3856050616 learned_addr learned my addr 172.21.15.23:0/3856050616 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:45.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.435+0000 7f2852515700 1 -- 172.21.15.23:0/3856050616 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f284c199cf0 con 0x7f284c1068d0 2024-03-28T05:13:45.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.435+0000 7f2852515700 1 --2- 172.21.15.23:0/3856050616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f284c1068d0 0x7f284c1997b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f284400f010 tx=0x7f284400dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:45.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.435+0000 7f284b7fe700 1 -- 172.21.15.23:0/3856050616 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2844019050 con 0x7f284c1068d0 2024-03-28T05:13:45.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.435+0000 7f284b7fe700 1 -- 172.21.15.23:0/3856050616 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2844025070 con 0x7f284c1068d0 2024-03-28T05:13:45.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.435+0000 7f2854779700 1 -- 172.21.15.23:0/3856050616 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f284c199f80 con 0x7f284c1068d0 2024-03-28T05:13:45.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.435+0000 7f284b7fe700 1 -- 172.21.15.23:0/3856050616 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2844004570 con 0x7f284c1068d0 2024-03-28T05:13:45.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.435+0000 7f2854779700 1 -- 172.21.15.23:0/3856050616 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f284c19cc70 con 0x7f284c1068d0 2024-03-28T05:13:45.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.436+0000 7f284b7fe700 1 -- 172.21.15.23:0/3856050616 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7f2844020cb0 con 0x7f284c1068d0 2024-03-28T05:13:45.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.436+0000 7f2854779700 1 -- 172.21.15.23:0/3856050616 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f284c0637c0 con 0x7f284c1068d0 2024-03-28T05:13:45.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.436+0000 7f284b7fe700 1 --2- 172.21.15.23:0/3856050616 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f2834038e00 0x7f283403b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:45.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.436+0000 7f284b7fe700 1 -- 172.21.15.23:0/3856050616 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2844011050 con 0x7f284c1068d0 2024-03-28T05:13:45.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.437+0000 7f2851d14700 1 --2- 172.21.15.23:0/3856050616 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f2834038e00 0x7f283403b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:45.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.437+0000 7f2851d14700 1 --2- 172.21.15.23:0/3856050616 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f2834038e00 0x7f283403b2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2840006fd0 tx=0x7f2840006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:45.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.442+0000 7f284b7fe700 1 -- 172.21.15.23:0/3856050616 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2844020aa0 con 0x7f284c1068d0 2024-03-28T05:13:45.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.632+0000 7f284b7fe700 1 -- 172.21.15.23:0/3856050616 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f2844020580 con 0x7f284c1068d0 2024-03-28T05:13:45.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.669+0000 7f2854779700 1 -- 172.21.15.23:0/3856050616 --> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f284c0625d0 con 0x7f2834038e00 2024-03-28T05:13:45.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.674+0000 7f284b7fe700 1 -- 172.21.15.23:0/3856050616 <== mgr.14118 v2:172.21.15.23:6800/1530849123 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f284c0625d0 con 0x7f2834038e00 2024-03-28T05:13:45.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.677+0000 7f2854779700 1 -- 172.21.15.23:0/3856050616 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f2834038e00 msgr2=0x7f283403b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:45.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.677+0000 7f2854779700 1 --2- 172.21.15.23:0/3856050616 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f2834038e00 0x7f283403b2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2840006fd0 tx=0x7f2840006e40 comp rx=0 tx=0).stop 2024-03-28T05:13:45.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.677+0000 7f2854779700 1 -- 172.21.15.23:0/3856050616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f284c1068d0 msgr2=0x7f284c1997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:45.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.677+0000 7f2854779700 1 --2- 172.21.15.23:0/3856050616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f284c1068d0 0x7f284c1997b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f284400f010 tx=0x7f284400dda0 comp rx=0 tx=0).stop 2024-03-28T05:13:45.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.677+0000 7f2854779700 1 -- 172.21.15.23:0/3856050616 shutdown_connections 2024-03-28T05:13:45.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.677+0000 7f2854779700 1 --2- 172.21.15.23:0/3856050616 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f2834038e00 0x7f283403b2c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:45.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.677+0000 7f2854779700 1 --2- 172.21.15.23:0/3856050616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f284c1068d0 0x7f284c1997b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:45.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.677+0000 7f2854779700 1 -- 172.21.15.23:0/3856050616 >> 172.21.15.23:0/3856050616 conn(0x7f284c101f10 msgr2=0x7f284c18fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:45.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.677+0000 7f2854779700 1 -- 172.21.15.23:0/3856050616 shutdown_connections 2024-03-28T05:13:45.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:44.677+0000 7f2854779700 1 -- 172.21.15.23:0/3856050616 wait complete. 2024-03-28T05:13:46.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:45 smithi023 ceph-mon[31954]: [28/Mar/2024:05:13:44] ENGINE Bus STARTING 2024-03-28T05:13:46.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:45 smithi023 ceph-mon[31954]: [28/Mar/2024:05:13:44] ENGINE Serving on http://172.21.15.23:8765 2024-03-28T05:13:46.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:45 smithi023 ceph-mon[31954]: [28/Mar/2024:05:13:44] ENGINE Serving on https://172.21.15.23:7150 2024-03-28T05:13:46.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:45 smithi023 ceph-mon[31954]: [28/Mar/2024:05:13:44] ENGINE Bus STARTED 2024-03-28T05:13:46.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:45 smithi023 ceph-mon[31954]: mgrmap e8: x(active, since 2s) 2024-03-28T05:13:46.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:45 smithi023 ceph-mon[31954]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:13:46.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:45 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' 2024-03-28T05:13:46.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:45 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:13:46.341 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-28T05:13:46.346 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.548+0000 7f1a2fb7c700 1 Processor -- start 2024-03-28T05:13:46.346 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.548+0000 7f1a2fb7c700 1 -- start start 2024-03-28T05:13:46.346 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.548+0000 7f1a2fb7c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a281084b0 0x7f1a2810a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:46.346 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.548+0000 7f1a2fb7c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2807ae70 con 0x7f1a281084b0 2024-03-28T05:13:46.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.549+0000 7f1a2d918700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a281084b0 0x7f1a2810a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:46.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.549+0000 7f1a2d918700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a281084b0 0x7f1a2810a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43494/0 (socket says 172.21.15.23:43494) 2024-03-28T05:13:46.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.549+0000 7f1a2d918700 1 -- 172.21.15.23:0/4236685054 learned_addr learned my addr 172.21.15.23:0/4236685054 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:46.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.549+0000 7f1a2d918700 1 -- 172.21.15.23:0/4236685054 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2810ade0 con 0x7f1a281084b0 2024-03-28T05:13:46.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.549+0000 7f1a2d918700 1 --2- 172.21.15.23:0/4236685054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a281084b0 0x7f1a2810a8a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1a180098a0 tx=0x7f1a18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db7d1c9577ce3558 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:46.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.549+0000 7f1a2c916700 1 -- 172.21.15.23:0/4236685054 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a18004160 con 0x7f1a281084b0 2024-03-28T05:13:46.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.549+0000 7f1a2c916700 1 -- 172.21.15.23:0/4236685054 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a18004300 con 0x7f1a281084b0 2024-03-28T05:13:46.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.549+0000 7f1a2c916700 1 -- 172.21.15.23:0/4236685054 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a18011b50 con 0x7f1a281084b0 2024-03-28T05:13:46.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.550+0000 7f1a2fb7c700 1 -- 172.21.15.23:0/4236685054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a281084b0 msgr2=0x7f1a2810a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:46.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.550+0000 7f1a2fb7c700 1 --2- 172.21.15.23:0/4236685054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a281084b0 0x7f1a2810a8a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1a180098a0 tx=0x7f1a18009bb0 comp rx=0 tx=0).stop 2024-03-28T05:13:46.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.550+0000 7f1a2fb7c700 1 -- 172.21.15.23:0/4236685054 shutdown_connections 2024-03-28T05:13:46.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.550+0000 7f1a2fb7c700 1 --2- 172.21.15.23:0/4236685054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a281084b0 0x7f1a2810a8a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:46.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.550+0000 7f1a2fb7c700 1 -- 172.21.15.23:0/4236685054 >> 172.21.15.23:0/4236685054 conn(0x7f1a28101ee0 msgr2=0x7f1a28104300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:46.347 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.550+0000 7f1a2fb7c700 1 -- 172.21.15.23:0/4236685054 shutdown_connections 2024-03-28T05:13:46.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.550+0000 7f1a2fb7c700 1 -- 172.21.15.23:0/4236685054 wait complete. 2024-03-28T05:13:46.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.551+0000 7f1a2fb7c700 1 Processor -- start 2024-03-28T05:13:46.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.551+0000 7f1a2fb7c700 1 -- start start 2024-03-28T05:13:46.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.551+0000 7f1a2fb7c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a281084b0 0x7f1a2819ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:46.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.551+0000 7f1a2fb7c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2810bdb0 con 0x7f1a281084b0 2024-03-28T05:13:46.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.551+0000 7f1a2d918700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a281084b0 0x7f1a2819ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:46.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.551+0000 7f1a2d918700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a281084b0 0x7f1a2819ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43500/0 (socket says 172.21.15.23:43500) 2024-03-28T05:13:46.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.551+0000 7f1a2d918700 1 -- 172.21.15.23:0/4149951844 learned_addr learned my addr 172.21.15.23:0/4149951844 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:46.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.551+0000 7f1a2d918700 1 -- 172.21.15.23:0/4149951844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2819bf40 con 0x7f1a281084b0 2024-03-28T05:13:46.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.552+0000 7f1a2d918700 1 --2- 172.21.15.23:0/4149951844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a281084b0 0x7f1a2819ba00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1a18009fb0 tx=0x7f1a18011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:46.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.552+0000 7f1a1effd700 1 -- 172.21.15.23:0/4149951844 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a180038b0 con 0x7f1a281084b0 2024-03-28T05:13:46.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.552+0000 7f1a1effd700 1 -- 172.21.15.23:0/4149951844 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a18011e50 con 0x7f1a281084b0 2024-03-28T05:13:46.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.552+0000 7f1a2fb7c700 1 -- 172.21.15.23:0/4149951844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a2819c1d0 con 0x7f1a281084b0 2024-03-28T05:13:46.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.552+0000 7f1a1effd700 1 -- 172.21.15.23:0/4149951844 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a18019420 con 0x7f1a281084b0 2024-03-28T05:13:46.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.552+0000 7f1a2fb7c700 1 -- 172.21.15.23:0/4149951844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a2819eec0 con 0x7f1a281084b0 2024-03-28T05:13:46.348 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.553+0000 7f1a1effd700 1 -- 172.21.15.23:0/4149951844 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f1a180196d0 con 0x7f1a281084b0 2024-03-28T05:13:46.349 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.553+0000 7f1a1effd700 1 --2- 172.21.15.23:0/4149951844 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f1a14038f20 0x7f1a1403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:46.349 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.553+0000 7f1a1effd700 1 -- 172.21.15.23:0/4149951844 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1a1804e100 con 0x7f1a281084b0 2024-03-28T05:13:46.349 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.553+0000 7f1a2d117700 1 --2- 172.21.15.23:0/4149951844 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f1a14038f20 0x7f1a1403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:46.349 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.553+0000 7f1a2fb7c700 1 -- 172.21.15.23:0/4149951844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a28069220 con 0x7f1a281084b0 2024-03-28T05:13:46.349 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.554+0000 7f1a2d117700 1 --2- 172.21.15.23:0/4149951844 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f1a14038f20 0x7f1a1403b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1a24006fd0 tx=0x7f1a24006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:46.349 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.559+0000 7f1a1effd700 1 -- 172.21.15.23:0/4149951844 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1a1800ba30 con 0x7f1a281084b0 2024-03-28T05:13:46.349 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.770+0000 7f1a2fb7c700 1 -- 172.21.15.23:0/4149951844 --> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a281031f0 con 0x7f1a14038f20 2024-03-28T05:13:46.349 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.770+0000 7f1a1effd700 1 -- 172.21.15.23:0/4149951844 <== mgr.14118 v2:172.21.15.23:6800/1530849123 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f1a281031f0 con 0x7f1a14038f20 2024-03-28T05:13:46.349 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.772+0000 7f1a2fb7c700 1 -- 172.21.15.23:0/4149951844 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f1a14038f20 msgr2=0x7f1a1403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:46.349 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.772+0000 7f1a2fb7c700 1 --2- 172.21.15.23:0/4149951844 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f1a14038f20 0x7f1a1403b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1a24006fd0 tx=0x7f1a24006e40 comp rx=0 tx=0).stop 2024-03-28T05:13:46.349 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.772+0000 7f1a2fb7c700 1 -- 172.21.15.23:0/4149951844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a281084b0 msgr2=0x7f1a2819ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:46.349 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.772+0000 7f1a2fb7c700 1 --2- 172.21.15.23:0/4149951844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a281084b0 0x7f1a2819ba00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1a18009fb0 tx=0x7f1a18011bf0 comp rx=0 tx=0).stop 2024-03-28T05:13:46.349 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.772+0000 7f1a2fb7c700 1 -- 172.21.15.23:0/4149951844 shutdown_connections 2024-03-28T05:13:46.349 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.772+0000 7f1a2fb7c700 1 --2- 172.21.15.23:0/4149951844 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f1a14038f20 0x7f1a1403b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:46.349 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.772+0000 7f1a2fb7c700 1 --2- 172.21.15.23:0/4149951844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a281084b0 0x7f1a2819ba00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:46.350 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.772+0000 7f1a2fb7c700 1 -- 172.21.15.23:0/4149951844 >> 172.21.15.23:0/4149951844 conn(0x7f1a28101ee0 msgr2=0x7f1a28102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:46.350 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.773+0000 7f1a2fb7c700 1 -- 172.21.15.23:0/4149951844 shutdown_connections 2024-03-28T05:13:46.350 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:45.773+0000 7f1a2fb7c700 1 -- 172.21.15.23:0/4149951844 wait complete. 2024-03-28T05:13:46.350 INFO:teuthology.orchestra.run.smithi023.stdout:Generating ssh key... 2024-03-28T05:13:46.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:46 smithi023 ceph-mon[31954]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:13:47.278 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: Generating public/private rsa key pair. 2024-03-28T05:13:47.278 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: Your identification has been saved in /tmp/tmp89a3eyr7/key. 2024-03-28T05:13:47.278 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: Your public key has been saved in /tmp/tmp89a3eyr7/key.pub. 2024-03-28T05:13:47.278 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: The key fingerprint is: 2024-03-28T05:13:47.278 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: SHA256:S6ClhdcizzQsiOnxulp7QnV9B8cddTzrsBlDMfVBIJI ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:13:47.278 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: The key's randomart image is: 2024-03-28T05:13:47.279 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: +---[RSA 3072]----+ 2024-03-28T05:13:47.279 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: | ..o =*Bo| 2024-03-28T05:13:47.279 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: | o . o . Eo +.o.=| 2024-03-28T05:13:47.279 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: |o.. + X.. o. +| 2024-03-28T05:13:47.279 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: |. o .%.+. . .+ . | 2024-03-28T05:13:47.279 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: | . oo.o S. . B | 2024-03-28T05:13:47.279 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: | o . . o . | 2024-03-28T05:13:47.279 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: | o. . | 2024-03-28T05:13:47.279 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: | .o.. | 2024-03-28T05:13:47.279 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: |o..o | 2024-03-28T05:13:47.279 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:13:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: +----[SHA256]-----+ 2024-03-28T05:13:47.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.769+0000 7f4b71256700 1 Processor -- start 2024-03-28T05:13:47.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.769+0000 7f4b71256700 1 -- start start 2024-03-28T05:13:47.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.769+0000 7f4b71256700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c105ce0 0x7f4b6c106100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:47.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.769+0000 7f4b71256700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c1066d0 con 0x7f4b6c105ce0 2024-03-28T05:13:47.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.769+0000 7f4b6ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c105ce0 0x7f4b6c106100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:47.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.769+0000 7f4b6ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c105ce0 0x7f4b6c106100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43504/0 (socket says 172.21.15.23:43504) 2024-03-28T05:13:47.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.769+0000 7f4b6ad9d700 1 -- 172.21.15.23:0/54595589 learned_addr learned my addr 172.21.15.23:0/54595589 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:47.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.769+0000 7f4b6ad9d700 1 -- 172.21.15.23:0/54595589 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6c108c70 con 0x7f4b6c105ce0 2024-03-28T05:13:47.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.770+0000 7f4b6ad9d700 1 --2- 172.21.15.23:0/54595589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c105ce0 0x7f4b6c106100 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c0098a0 tx=0x7f4b5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548591f8c8528ad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:47.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.770+0000 7f4b69d9b700 1 -- 172.21.15.23:0/54595589 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b5c004160 con 0x7f4b6c105ce0 2024-03-28T05:13:47.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.770+0000 7f4b69d9b700 1 -- 172.21.15.23:0/54595589 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b5c004300 con 0x7f4b6c105ce0 2024-03-28T05:13:47.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.770+0000 7f4b69d9b700 1 -- 172.21.15.23:0/54595589 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b5c011b50 con 0x7f4b6c105ce0 2024-03-28T05:13:47.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.770+0000 7f4b71256700 1 -- 172.21.15.23:0/54595589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c105ce0 msgr2=0x7f4b6c106100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:47.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.770+0000 7f4b71256700 1 --2- 172.21.15.23:0/54595589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c105ce0 0x7f4b6c106100 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c0098a0 tx=0x7f4b5c009bb0 comp rx=0 tx=0).stop 2024-03-28T05:13:47.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.770+0000 7f4b71256700 1 -- 172.21.15.23:0/54595589 shutdown_connections 2024-03-28T05:13:47.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.770+0000 7f4b71256700 1 --2- 172.21.15.23:0/54595589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c105ce0 0x7f4b6c106100 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:47.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.770+0000 7f4b71256700 1 -- 172.21.15.23:0/54595589 >> 172.21.15.23:0/54595589 conn(0x7f4b6c101690 msgr2=0x7f4b6c103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:47.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.770+0000 7f4b71256700 1 -- 172.21.15.23:0/54595589 shutdown_connections 2024-03-28T05:13:47.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.770+0000 7f4b71256700 1 -- 172.21.15.23:0/54595589 wait complete. 2024-03-28T05:13:47.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.771+0000 7f4b71256700 1 Processor -- start 2024-03-28T05:13:47.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.771+0000 7f4b71256700 1 -- start start 2024-03-28T05:13:47.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.772+0000 7f4b71256700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c105ce0 0x7f4b6c19ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:47.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.772+0000 7f4b71256700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c10c020 con 0x7f4b6c105ce0 2024-03-28T05:13:47.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.772+0000 7f4b6ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c105ce0 0x7f4b6c19ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:47.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.772+0000 7f4b6ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c105ce0 0x7f4b6c19ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43516/0 (socket says 172.21.15.23:43516) 2024-03-28T05:13:47.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.772+0000 7f4b6ad9d700 1 -- 172.21.15.23:0/3140875296 learned_addr learned my addr 172.21.15.23:0/3140875296 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:47.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.772+0000 7f4b6ad9d700 1 -- 172.21.15.23:0/3140875296 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6c19bf60 con 0x7f4b6c105ce0 2024-03-28T05:13:47.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.772+0000 7f4b6ad9d700 1 --2- 172.21.15.23:0/3140875296 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c105ce0 0x7f4b6c19ba20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c011850 tx=0x7f4b5c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:47.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.772+0000 7f4b53fff700 1 -- 172.21.15.23:0/3140875296 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b5c003af0 con 0x7f4b6c105ce0 2024-03-28T05:13:47.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.772+0000 7f4b53fff700 1 -- 172.21.15.23:0/3140875296 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b5c020070 con 0x7f4b6c105ce0 2024-03-28T05:13:47.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.772+0000 7f4b71256700 1 -- 172.21.15.23:0/3140875296 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b6c19c1f0 con 0x7f4b6c105ce0 2024-03-28T05:13:47.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.772+0000 7f4b53fff700 1 -- 172.21.15.23:0/3140875296 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b5c0195a0 con 0x7f4b6c105ce0 2024-03-28T05:13:47.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.772+0000 7f4b71256700 1 -- 172.21.15.23:0/3140875296 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b6c19eee0 con 0x7f4b6c105ce0 2024-03-28T05:13:47.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.773+0000 7f4b53fff700 1 -- 172.21.15.23:0/3140875296 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f4b5c0198a0 con 0x7f4b6c105ce0 2024-03-28T05:13:47.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.773+0000 7f4b71256700 1 -- 172.21.15.23:0/3140875296 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b6c069220 con 0x7f4b6c105ce0 2024-03-28T05:13:47.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.773+0000 7f4b53fff700 1 --2- 172.21.15.23:0/3140875296 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f4b54038f20 0x7f4b5403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:47.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.773+0000 7f4b53fff700 1 -- 172.21.15.23:0/3140875296 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4b5c011e00 con 0x7f4b6c105ce0 2024-03-28T05:13:47.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.774+0000 7f4b6a59c700 1 --2- 172.21.15.23:0/3140875296 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f4b54038f20 0x7f4b5403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:47.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.774+0000 7f4b6a59c700 1 --2- 172.21.15.23:0/3140875296 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f4b54038f20 0x7f4b5403b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4b60006fd0 tx=0x7f4b60006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:47.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.778+0000 7f4b53fff700 1 -- 172.21.15.23:0/3140875296 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4b5c023370 con 0x7f4b6c105ce0 2024-03-28T05:13:47.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:46.985+0000 7f4b71256700 1 -- 172.21.15.23:0/3140875296 --> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b6c1028d0 con 0x7f4b54038f20 2024-03-28T05:13:47.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:47.098+0000 7f4b53fff700 1 -- 172.21.15.23:0/3140875296 <== mgr.14118 v2:172.21.15.23:6800/1530849123 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4b6c1028d0 con 0x7f4b54038f20 2024-03-28T05:13:47.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:47.100+0000 7f4b71256700 1 -- 172.21.15.23:0/3140875296 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f4b54038f20 msgr2=0x7f4b5403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:47.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:47.100+0000 7f4b71256700 1 --2- 172.21.15.23:0/3140875296 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f4b54038f20 0x7f4b5403b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4b60006fd0 tx=0x7f4b60006e40 comp rx=0 tx=0).stop 2024-03-28T05:13:47.665 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:47.100+0000 7f4b71256700 1 -- 172.21.15.23:0/3140875296 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c105ce0 msgr2=0x7f4b6c19ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:47.665 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:47.100+0000 7f4b71256700 1 --2- 172.21.15.23:0/3140875296 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c105ce0 0x7f4b6c19ba20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c011850 tx=0x7f4b5c011c10 comp rx=0 tx=0).stop 2024-03-28T05:13:47.665 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:47.100+0000 7f4b71256700 1 -- 172.21.15.23:0/3140875296 shutdown_connections 2024-03-28T05:13:47.665 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:47.100+0000 7f4b71256700 1 --2- 172.21.15.23:0/3140875296 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f4b54038f20 0x7f4b5403b3e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:47.665 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:47.100+0000 7f4b71256700 1 --2- 172.21.15.23:0/3140875296 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c105ce0 0x7f4b6c19ba20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:47.665 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:47.100+0000 7f4b71256700 1 -- 172.21.15.23:0/3140875296 >> 172.21.15.23:0/3140875296 conn(0x7f4b6c101690 msgr2=0x7f4b6c1021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:47.665 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:47.100+0000 7f4b71256700 1 -- 172.21.15.23:0/3140875296 shutdown_connections 2024-03-28T05:13:47.665 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:47.100+0000 7f4b71256700 1 -- 172.21.15.23:0/3140875296 wait complete. 2024-03-28T05:13:48.277 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:48 smithi023 ceph-mon[31954]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:13:48.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:48 smithi023 ceph-mon[31954]: Generating ssh key... 2024-03-28T05:13:48.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:48 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' 2024-03-28T05:13:48.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:48 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' 2024-03-28T05:13:48.784 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJm9JJyDDWwQrMhwNXtEGvp00PVOSTVjxw5bl5xgpTZFd2JUIegKutwN1l5Zozcw2gNVppK/hukUK49IYN1qfpC464mZUje438hcotyzRJvKDov2FU/7AAtVA7VcIWzjicD04dDIFk7I637dusTKotawi1inz2RalnaQBlsThkvqCkujx3iZ0zWqyVDvXpOdnWe7PlmdqTKdvZWe+oqLmU8tqaheIkFoEHUg/KkbLV9jJnR62lC9JSGhX3Tyz+tYSti7i5N8iNpcywRF3NSrnEXNV9xuULcLCwtbfn6vYxsMdblZggu+es14YB8MidwDqhvgQPQ4t5rM4s8QGsBGtJV6mp3EJtLymsvPCjX2w7iNYSRBJLhRdz9uFJIsUVUxPlP9COvTsEp3fIfVGLzoM7wZi6MbfOzUaDBqcdhqG2gJd+fNMOcuueILfkpD7uw6Q8KuRvnJcM1+5QSpT+C78wu+/OqNxEDcX8ZNSKODPXpxPcqne/mkJS8zLatNQv4dM= ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:13:48.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.084+0000 7faae42fc700 1 Processor -- start 2024-03-28T05:13:48.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.084+0000 7faae42fc700 1 -- start start 2024-03-28T05:13:48.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.084+0000 7faae42fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc1084b0 0x7faadc10a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:48.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.084+0000 7faae42fc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc07ae50 con 0x7faadc1084b0 2024-03-28T05:13:48.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.084+0000 7faae2098700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc1084b0 0x7faadc10a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:48.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.085+0000 7faae2098700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc1084b0 0x7faadc10a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43524/0 (socket says 172.21.15.23:43524) 2024-03-28T05:13:48.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.085+0000 7faae2098700 1 -- 172.21.15.23:0/2913831945 learned_addr learned my addr 172.21.15.23:0/2913831945 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:48.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.085+0000 7faae2098700 1 -- 172.21.15.23:0/2913831945 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faadc10ade0 con 0x7faadc1084b0 2024-03-28T05:13:48.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.085+0000 7faae2098700 1 --2- 172.21.15.23:0/2913831945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc1084b0 0x7faadc10a8a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7faad80098a0 tx=0x7faad8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb11386182d79178 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:48.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.085+0000 7faae1096700 1 -- 172.21.15.23:0/2913831945 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faad8004160 con 0x7faadc1084b0 2024-03-28T05:13:48.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.085+0000 7faae1096700 1 -- 172.21.15.23:0/2913831945 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faad8004300 con 0x7faadc1084b0 2024-03-28T05:13:48.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.085+0000 7faae1096700 1 -- 172.21.15.23:0/2913831945 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faad8011b50 con 0x7faadc1084b0 2024-03-28T05:13:48.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.086+0000 7faae42fc700 1 -- 172.21.15.23:0/2913831945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc1084b0 msgr2=0x7faadc10a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:48.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.086+0000 7faae42fc700 1 --2- 172.21.15.23:0/2913831945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc1084b0 0x7faadc10a8a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7faad80098a0 tx=0x7faad8009bb0 comp rx=0 tx=0).stop 2024-03-28T05:13:48.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.086+0000 7faae42fc700 1 -- 172.21.15.23:0/2913831945 shutdown_connections 2024-03-28T05:13:48.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.086+0000 7faae42fc700 1 --2- 172.21.15.23:0/2913831945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc1084b0 0x7faadc10a8a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:48.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.086+0000 7faae42fc700 1 -- 172.21.15.23:0/2913831945 >> 172.21.15.23:0/2913831945 conn(0x7faadc101f40 msgr2=0x7faadc104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:48.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.086+0000 7faae42fc700 1 -- 172.21.15.23:0/2913831945 shutdown_connections 2024-03-28T05:13:48.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.086+0000 7faae42fc700 1 -- 172.21.15.23:0/2913831945 wait complete. 2024-03-28T05:13:48.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.087+0000 7faae42fc700 1 Processor -- start 2024-03-28T05:13:48.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.087+0000 7faae42fc700 1 -- start start 2024-03-28T05:13:48.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.087+0000 7faae42fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc1084b0 0x7faadc19b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:48.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.087+0000 7faae42fc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc10bda0 con 0x7faadc1084b0 2024-03-28T05:13:48.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.087+0000 7faae2098700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc1084b0 0x7faadc19b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:48.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.087+0000 7faae2098700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc1084b0 0x7faadc19b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43526/0 (socket says 172.21.15.23:43526) 2024-03-28T05:13:48.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.087+0000 7faae2098700 1 -- 172.21.15.23:0/2818028546 learned_addr learned my addr 172.21.15.23:0/2818028546 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:48.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.087+0000 7faae2098700 1 -- 172.21.15.23:0/2818028546 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faadc19bee0 con 0x7faadc1084b0 2024-03-28T05:13:48.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.087+0000 7faae2098700 1 --2- 172.21.15.23:0/2818028546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc1084b0 0x7faadc19b9a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7faad8011b60 tx=0x7faad8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:48.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.087+0000 7faad37fe700 1 -- 172.21.15.23:0/2818028546 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faad8003970 con 0x7faadc1084b0 2024-03-28T05:13:48.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.088+0000 7faad37fe700 1 -- 172.21.15.23:0/2818028546 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faad8003b10 con 0x7faadc1084b0 2024-03-28T05:13:48.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.088+0000 7faae42fc700 1 -- 172.21.15.23:0/2818028546 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faadc19c170 con 0x7faadc1084b0 2024-03-28T05:13:48.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.088+0000 7faad37fe700 1 -- 172.21.15.23:0/2818028546 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faad8024610 con 0x7faadc1084b0 2024-03-28T05:13:48.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.088+0000 7faae42fc700 1 -- 172.21.15.23:0/2818028546 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faadc19ee60 con 0x7faadc1084b0 2024-03-28T05:13:48.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.088+0000 7faad37fe700 1 -- 172.21.15.23:0/2818028546 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7faad80248c0 con 0x7faadc1084b0 2024-03-28T05:13:48.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.089+0000 7faae42fc700 1 -- 172.21.15.23:0/2818028546 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faadc069220 con 0x7faadc1084b0 2024-03-28T05:13:48.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.089+0000 7faad37fe700 1 --2- 172.21.15.23:0/2818028546 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7faac8038f70 0x7faac803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:48.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.089+0000 7faad37fe700 1 -- 172.21.15.23:0/2818028546 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7faad8011e00 con 0x7faadc1084b0 2024-03-28T05:13:48.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.089+0000 7faae1897700 1 --2- 172.21.15.23:0/2818028546 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7faac8038f70 0x7faac803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:48.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.089+0000 7faae1897700 1 --2- 172.21.15.23:0/2818028546 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7faac8038f70 0x7faac803b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7faacc006fd0 tx=0x7faacc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:48.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.094+0000 7faad37fe700 1 -- 172.21.15.23:0/2818028546 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faad800ba00 con 0x7faadc1084b0 2024-03-28T05:13:48.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.300+0000 7faae42fc700 1 -- 172.21.15.23:0/2818028546 --> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7faadc1031d0 con 0x7faac8038f70 2024-03-28T05:13:48.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.301+0000 7faad37fe700 1 -- 172.21.15.23:0/2818028546 <== mgr.14118 v2:172.21.15.23:6800/1530849123 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7faadc1031d0 con 0x7faac8038f70 2024-03-28T05:13:48.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.303+0000 7faae42fc700 1 -- 172.21.15.23:0/2818028546 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7faac8038f70 msgr2=0x7faac803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:48.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.303+0000 7faae42fc700 1 --2- 172.21.15.23:0/2818028546 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7faac8038f70 0x7faac803b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7faacc006fd0 tx=0x7faacc006e40 comp rx=0 tx=0).stop 2024-03-28T05:13:48.788 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.303+0000 7faae42fc700 1 -- 172.21.15.23:0/2818028546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc1084b0 msgr2=0x7faadc19b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:48.788 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.303+0000 7faae42fc700 1 --2- 172.21.15.23:0/2818028546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc1084b0 0x7faadc19b9a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7faad8011b60 tx=0x7faad8011c40 comp rx=0 tx=0).stop 2024-03-28T05:13:48.788 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.303+0000 7faae42fc700 1 -- 172.21.15.23:0/2818028546 shutdown_connections 2024-03-28T05:13:48.788 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.303+0000 7faae42fc700 1 --2- 172.21.15.23:0/2818028546 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7faac8038f70 0x7faac803b430 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:48.788 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.303+0000 7faae42fc700 1 --2- 172.21.15.23:0/2818028546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faadc1084b0 0x7faadc19b9a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:48.788 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.303+0000 7faae42fc700 1 -- 172.21.15.23:0/2818028546 >> 172.21.15.23:0/2818028546 conn(0x7faadc101f40 msgr2=0x7faadc102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:48.788 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.303+0000 7faae42fc700 1 -- 172.21.15.23:0/2818028546 shutdown_connections 2024-03-28T05:13:48.788 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:48.303+0000 7faae42fc700 1 -- 172.21.15.23:0/2818028546 wait complete. 2024-03-28T05:13:48.788 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-28T05:13:48.788 INFO:teuthology.orchestra.run.smithi023.stdout:Adding key to root@localhost authorized_keys... 2024-03-28T05:13:48.788 INFO:teuthology.orchestra.run.smithi023.stdout:Adding host smithi023... 2024-03-28T05:13:49.428 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:49 smithi023 ceph-mon[31954]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:13:50.277 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:50 smithi023 ceph-mon[31954]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi023", "addr": "172.21.15.23", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:13:51.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:51 smithi023 ceph-mon[31954]: Deploying cephadm binary to smithi023 2024-03-28T05:13:54.326 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Added host 'smithi023' with addr '172.21.15.23' 2024-03-28T05:13:54.326 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.213+0000 7f1a10d8e700 1 Processor -- start 2024-03-28T05:13:54.326 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.213+0000 7f1a10d8e700 1 -- start start 2024-03-28T05:13:54.326 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.213+0000 7f1a10d8e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a0c1084e0 0x7f1a0c10a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:54.326 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.213+0000 7f1a10d8e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c07ae70 con 0x7f1a0c1084e0 2024-03-28T05:13:54.326 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.213+0000 7f1a0a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a0c1084e0 0x7f1a0c10a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:54.327 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.213+0000 7f1a0a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a0c1084e0 0x7f1a0c10a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43530/0 (socket says 172.21.15.23:43530) 2024-03-28T05:13:54.327 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.213+0000 7f1a0a59c700 1 -- 172.21.15.23:0/4139751001 learned_addr learned my addr 172.21.15.23:0/4139751001 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:54.327 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.214+0000 7f1a0a59c700 1 -- 172.21.15.23:0/4139751001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a0c10ae10 con 0x7f1a0c1084e0 2024-03-28T05:13:54.327 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.214+0000 7f1a0a59c700 1 --2- 172.21.15.23:0/4139751001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a0c1084e0 0x7f1a0c10a8d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f19fc0098a0 tx=0x7f19fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c72c920c61a1b842 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:54.327 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.214+0000 7f1a0959a700 1 -- 172.21.15.23:0/4139751001 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19fc004160 con 0x7f1a0c1084e0 2024-03-28T05:13:54.327 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.214+0000 7f1a0959a700 1 -- 172.21.15.23:0/4139751001 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f19fc004300 con 0x7f1a0c1084e0 2024-03-28T05:13:54.327 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.214+0000 7f1a0959a700 1 -- 172.21.15.23:0/4139751001 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19fc011b50 con 0x7f1a0c1084e0 2024-03-28T05:13:54.327 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.215+0000 7f1a10d8e700 1 -- 172.21.15.23:0/4139751001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a0c1084e0 msgr2=0x7f1a0c10a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:54.327 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.215+0000 7f1a10d8e700 1 --2- 172.21.15.23:0/4139751001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a0c1084e0 0x7f1a0c10a8d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f19fc0098a0 tx=0x7f19fc009bb0 comp rx=0 tx=0).stop 2024-03-28T05:13:54.327 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.215+0000 7f1a10d8e700 1 -- 172.21.15.23:0/4139751001 shutdown_connections 2024-03-28T05:13:54.327 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.215+0000 7f1a10d8e700 1 --2- 172.21.15.23:0/4139751001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a0c1084e0 0x7f1a0c10a8d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:54.327 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.215+0000 7f1a10d8e700 1 -- 172.21.15.23:0/4139751001 >> 172.21.15.23:0/4139751001 conn(0x7f1a0c101f70 msgr2=0x7f1a0c104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:54.328 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.215+0000 7f1a10d8e700 1 -- 172.21.15.23:0/4139751001 shutdown_connections 2024-03-28T05:13:54.328 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.215+0000 7f1a10d8e700 1 -- 172.21.15.23:0/4139751001 wait complete. 2024-03-28T05:13:54.328 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.215+0000 7f1a10d8e700 1 Processor -- start 2024-03-28T05:13:54.328 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.215+0000 7f1a10d8e700 1 -- start start 2024-03-28T05:13:54.328 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.216+0000 7f1a10d8e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a0c1084e0 0x7f1a0c19ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:54.328 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.216+0000 7f1a10d8e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c10bde0 con 0x7f1a0c1084e0 2024-03-28T05:13:54.328 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.216+0000 7f1a0a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a0c1084e0 0x7f1a0c19ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:54.328 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.216+0000 7f1a0a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a0c1084e0 0x7f1a0c19ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43542/0 (socket says 172.21.15.23:43542) 2024-03-28T05:13:54.328 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.216+0000 7f1a0a59c700 1 -- 172.21.15.23:0/1301713924 learned_addr learned my addr 172.21.15.23:0/1301713924 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:54.328 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.216+0000 7f1a0a59c700 1 -- 172.21.15.23:0/1301713924 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a0c19bf50 con 0x7f1a0c1084e0 2024-03-28T05:13:54.328 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.216+0000 7f1a0a59c700 1 --2- 172.21.15.23:0/1301713924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a0c1084e0 0x7f1a0c19ba10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f19fc009fb0 tx=0x7f19fc011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:54.328 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.216+0000 7f19fb7fe700 1 -- 172.21.15.23:0/1301713924 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19fc0038b0 con 0x7f1a0c1084e0 2024-03-28T05:13:54.329 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.216+0000 7f19fb7fe700 1 -- 172.21.15.23:0/1301713924 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f19fc011e50 con 0x7f1a0c1084e0 2024-03-28T05:13:54.329 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.216+0000 7f19fb7fe700 1 -- 172.21.15.23:0/1301713924 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19fc019420 con 0x7f1a0c1084e0 2024-03-28T05:13:54.329 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.216+0000 7f1a10d8e700 1 -- 172.21.15.23:0/1301713924 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a0c19c1e0 con 0x7f1a0c1084e0 2024-03-28T05:13:54.329 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.216+0000 7f1a10d8e700 1 -- 172.21.15.23:0/1301713924 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a0c19eed0 con 0x7f1a0c1084e0 2024-03-28T05:13:54.329 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.217+0000 7f19fb7fe700 1 -- 172.21.15.23:0/1301713924 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f19fc0196d0 con 0x7f1a0c1084e0 2024-03-28T05:13:54.329 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.217+0000 7f1a10d8e700 1 -- 172.21.15.23:0/1301713924 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a0c0637c0 con 0x7f1a0c1084e0 2024-03-28T05:13:54.329 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.218+0000 7f19fb7fe700 1 --2- 172.21.15.23:0/1301713924 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f19f4038f20 0x7f19f403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:54.329 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.218+0000 7f19fb7fe700 1 -- 172.21.15.23:0/1301713924 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f19fc04d0f0 con 0x7f1a0c1084e0 2024-03-28T05:13:54.329 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.218+0000 7f1a09d9b700 1 --2- 172.21.15.23:0/1301713924 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f19f4038f20 0x7f19f403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:54.329 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.218+0000 7f1a09d9b700 1 --2- 172.21.15.23:0/1301713924 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f19f4038f20 0x7f19f403b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1a00006fd0 tx=0x7f1a00006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:54.329 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.223+0000 7f19fb7fe700 1 -- 172.21.15.23:0/1301713924 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f19fc003a50 con 0x7f1a0c1084e0 2024-03-28T05:13:54.330 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:49.431+0000 7f1a10d8e700 1 -- 172.21.15.23:0/1301713924 --> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi023", "addr": "172.21.15.23", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a0c1031e0 con 0x7f19f4038f20 2024-03-28T05:13:54.330 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:53.781+0000 7f19fb7fe700 1 -- 172.21.15.23:0/1301713924 <== mgr.14118 v2:172.21.15.23:6800/1530849123 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f1a0c1031e0 con 0x7f19f4038f20 2024-03-28T05:13:54.330 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:53.784+0000 7f1a10d8e700 1 -- 172.21.15.23:0/1301713924 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f19f4038f20 msgr2=0x7f19f403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:54.330 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:53.784+0000 7f1a10d8e700 1 --2- 172.21.15.23:0/1301713924 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f19f4038f20 0x7f19f403b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1a00006fd0 tx=0x7f1a00006e40 comp rx=0 tx=0).stop 2024-03-28T05:13:54.330 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:53.784+0000 7f1a10d8e700 1 -- 172.21.15.23:0/1301713924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a0c1084e0 msgr2=0x7f1a0c19ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:54.330 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:53.784+0000 7f1a10d8e700 1 --2- 172.21.15.23:0/1301713924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a0c1084e0 0x7f1a0c19ba10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f19fc009fb0 tx=0x7f19fc011bf0 comp rx=0 tx=0).stop 2024-03-28T05:13:54.330 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:53.784+0000 7f1a10d8e700 1 -- 172.21.15.23:0/1301713924 shutdown_connections 2024-03-28T05:13:54.330 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:53.784+0000 7f1a10d8e700 1 --2- 172.21.15.23:0/1301713924 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f19f4038f20 0x7f19f403b3e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:54.330 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:53.784+0000 7f1a10d8e700 1 --2- 172.21.15.23:0/1301713924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a0c1084e0 0x7f1a0c19ba10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:54.330 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:53.784+0000 7f1a10d8e700 1 -- 172.21.15.23:0/1301713924 >> 172.21.15.23:0/1301713924 conn(0x7f1a0c101f70 msgr2=0x7f1a0c102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:54.330 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:53.784+0000 7f1a10d8e700 1 -- 172.21.15.23:0/1301713924 shutdown_connections 2024-03-28T05:13:54.330 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:53.784+0000 7f1a10d8e700 1 -- 172.21.15.23:0/1301713924 wait complete. 2024-03-28T05:13:54.330 INFO:teuthology.orchestra.run.smithi023.stdout:Deploying unmanaged mon service... 2024-03-28T05:13:55.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:54 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' 2024-03-28T05:13:55.126 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:54 smithi023 ceph-mon[31954]: Added host smithi023 2024-03-28T05:13:55.127 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:54 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:13:56.277 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:56 smithi023 ceph-mon[31954]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:13:56.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:56 smithi023 ceph-mon[31954]: Saving service mon spec with placement count:5 2024-03-28T05:13:56.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:56 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' 2024-03-28T05:13:56.424 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-28T05:13:56.424 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.794+0000 7f6bdde38700 1 Processor -- start 2024-03-28T05:13:56.424 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.794+0000 7f6bdde38700 1 -- start start 2024-03-28T05:13:56.424 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.794+0000 7f6bdde38700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bd8107e90 0x7f6bd81082b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:56.424 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.794+0000 7f6bdde38700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd8108880 con 0x7f6bd8107e90 2024-03-28T05:13:56.424 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.794+0000 7f6bd77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bd8107e90 0x7f6bd81082b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:56.424 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.794+0000 7f6bd77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bd8107e90 0x7f6bd81082b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54028/0 (socket says 172.21.15.23:54028) 2024-03-28T05:13:56.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.794+0000 7f6bd77fe700 1 -- 172.21.15.23:0/691969956 learned_addr learned my addr 172.21.15.23:0/691969956 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:56.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.794+0000 7f6bd77fe700 1 -- 172.21.15.23:0/691969956 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd8108a00 con 0x7f6bd8107e90 2024-03-28T05:13:56.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.795+0000 7f6bd77fe700 1 --2- 172.21.15.23:0/691969956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bd8107e90 0x7f6bd81082b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6bc800bb40 tx=0x7f6bc800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f88368591481fcc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:56.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.795+0000 7f6bd67fc700 1 -- 172.21.15.23:0/691969956 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6bc800dbe0 con 0x7f6bd8107e90 2024-03-28T05:13:56.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.795+0000 7f6bd67fc700 1 -- 172.21.15.23:0/691969956 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6bc80044f0 con 0x7f6bd8107e90 2024-03-28T05:13:56.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.795+0000 7f6bd67fc700 1 -- 172.21.15.23:0/691969956 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6bc8010470 con 0x7f6bd8107e90 2024-03-28T05:13:56.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.795+0000 7f6bdde38700 1 -- 172.21.15.23:0/691969956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bd8107e90 msgr2=0x7f6bd81082b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:56.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.795+0000 7f6bdde38700 1 --2- 172.21.15.23:0/691969956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bd8107e90 0x7f6bd81082b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6bc800bb40 tx=0x7f6bc800be50 comp rx=0 tx=0).stop 2024-03-28T05:13:56.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.795+0000 7f6bdde38700 1 -- 172.21.15.23:0/691969956 shutdown_connections 2024-03-28T05:13:56.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.795+0000 7f6bdde38700 1 --2- 172.21.15.23:0/691969956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bd8107e90 0x7f6bd81082b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:56.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.795+0000 7f6bdde38700 1 -- 172.21.15.23:0/691969956 >> 172.21.15.23:0/691969956 conn(0x7f6bd8101720 msgr2=0x7f6bd8103b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:56.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.795+0000 7f6bdde38700 1 -- 172.21.15.23:0/691969956 shutdown_connections 2024-03-28T05:13:56.425 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.795+0000 7f6bdde38700 1 -- 172.21.15.23:0/691969956 wait complete. 2024-03-28T05:13:56.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.796+0000 7f6bdde38700 1 Processor -- start 2024-03-28T05:13:56.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.796+0000 7f6bdde38700 1 -- start start 2024-03-28T05:13:56.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.796+0000 7f6bdde38700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bd8107e90 0x7f6bd8199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:56.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.796+0000 7f6bdde38700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd8105ac0 con 0x7f6bd8107e90 2024-03-28T05:13:56.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.796+0000 7f6bd77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bd8107e90 0x7f6bd8199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:56.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.796+0000 7f6bd77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bd8107e90 0x7f6bd8199820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54038/0 (socket says 172.21.15.23:54038) 2024-03-28T05:13:56.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.796+0000 7f6bd77fe700 1 -- 172.21.15.23:0/2645782712 learned_addr learned my addr 172.21.15.23:0/2645782712 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:56.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.797+0000 7f6bd77fe700 1 -- 172.21.15.23:0/2645782712 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd8199d60 con 0x7f6bd8107e90 2024-03-28T05:13:56.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.797+0000 7f6bd77fe700 1 --2- 172.21.15.23:0/2645782712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bd8107e90 0x7f6bd8199820 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6bc800b340 tx=0x7f6bc80040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:56.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.797+0000 7f6bd4ff9700 1 -- 172.21.15.23:0/2645782712 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6bc8027070 con 0x7f6bd8107e90 2024-03-28T05:13:56.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.797+0000 7f6bd4ff9700 1 -- 172.21.15.23:0/2645782712 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6bc80104e0 con 0x7f6bd8107e90 2024-03-28T05:13:56.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.797+0000 7f6bdde38700 1 -- 172.21.15.23:0/2645782712 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd8199ff0 con 0x7f6bd8107e90 2024-03-28T05:13:56.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.797+0000 7f6bd4ff9700 1 -- 172.21.15.23:0/2645782712 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6bc8022c40 con 0x7f6bd8107e90 2024-03-28T05:13:56.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.797+0000 7f6bdde38700 1 -- 172.21.15.23:0/2645782712 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd819cce0 con 0x7f6bd8107e90 2024-03-28T05:13:56.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.798+0000 7f6bd4ff9700 1 -- 172.21.15.23:0/2645782712 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f6bc802a070 con 0x7f6bd8107e90 2024-03-28T05:13:56.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.798+0000 7f6bd4ff9700 1 --2- 172.21.15.23:0/2645782712 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f6bc0038f20 0x7f6bc003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:56.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.798+0000 7f6bd4ff9700 1 -- 172.21.15.23:0/2645782712 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6bc801f070 con 0x7f6bd8107e90 2024-03-28T05:13:56.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.798+0000 7f6bdde38700 1 -- 172.21.15.23:0/2645782712 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd80637c0 con 0x7f6bd8107e90 2024-03-28T05:13:56.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.798+0000 7f6bd6ffd700 1 --2- 172.21.15.23:0/2645782712 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f6bc0038f20 0x7f6bc003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:56.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.798+0000 7f6bd6ffd700 1 --2- 172.21.15.23:0/2645782712 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f6bc0038f20 0x7f6bc003b3e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc006fd0 tx=0x7f6bcc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:56.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:54.804+0000 7f6bd4ff9700 1 -- 172.21.15.23:0/2645782712 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6bc8031030 con 0x7f6bd8107e90 2024-03-28T05:13:56.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:55.036+0000 7f6bdde38700 1 -- 172.21.15.23:0/2645782712 --> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f6bd80625d0 con 0x7f6bc0038f20 2024-03-28T05:13:56.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:55.046+0000 7f6bd4ff9700 1 -- 172.21.15.23:0/2645782712 <== mgr.14118 v2:172.21.15.23:6800/1530849123 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6bd80625d0 con 0x7f6bc0038f20 2024-03-28T05:13:56.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:55.048+0000 7f6bdde38700 1 -- 172.21.15.23:0/2645782712 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f6bc0038f20 msgr2=0x7f6bc003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:56.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:55.048+0000 7f6bdde38700 1 --2- 172.21.15.23:0/2645782712 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f6bc0038f20 0x7f6bc003b3e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc006fd0 tx=0x7f6bcc006e40 comp rx=0 tx=0).stop 2024-03-28T05:13:56.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:55.048+0000 7f6bdde38700 1 -- 172.21.15.23:0/2645782712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bd8107e90 msgr2=0x7f6bd8199820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:56.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:55.048+0000 7f6bdde38700 1 --2- 172.21.15.23:0/2645782712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bd8107e90 0x7f6bd8199820 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6bc800b340 tx=0x7f6bc80040b0 comp rx=0 tx=0).stop 2024-03-28T05:13:56.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:55.048+0000 7f6bdde38700 1 -- 172.21.15.23:0/2645782712 shutdown_connections 2024-03-28T05:13:56.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:55.048+0000 7f6bdde38700 1 --2- 172.21.15.23:0/2645782712 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f6bc0038f20 0x7f6bc003b3e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:56.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:55.048+0000 7f6bdde38700 1 --2- 172.21.15.23:0/2645782712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bd8107e90 0x7f6bd8199820 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:56.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:55.048+0000 7f6bdde38700 1 -- 172.21.15.23:0/2645782712 >> 172.21.15.23:0/2645782712 conn(0x7f6bd8101720 msgr2=0x7f6bd818fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:56.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:55.048+0000 7f6bdde38700 1 -- 172.21.15.23:0/2645782712 shutdown_connections 2024-03-28T05:13:56.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:55.048+0000 7f6bdde38700 1 -- 172.21.15.23:0/2645782712 wait complete. 2024-03-28T05:13:56.428 INFO:teuthology.orchestra.run.smithi023.stdout:Deploying unmanaged mgr service... 2024-03-28T05:13:57.396 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:57 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' 2024-03-28T05:13:57.396 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:57 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' 2024-03-28T05:13:57.689 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-28T05:13:57.689 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.867+0000 7f0ed7b68700 1 Processor -- start 2024-03-28T05:13:57.689 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.867+0000 7f0ed7b68700 1 -- start start 2024-03-28T05:13:57.689 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.867+0000 7f0ed7b68700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed007cea0 0x7f0ed007b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:57.689 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.867+0000 7f0ed7b68700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed007b8d0 con 0x7f0ed007cea0 2024-03-28T05:13:57.689 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.868+0000 7f0ed5904700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed007cea0 0x7f0ed007b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:57.689 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.868+0000 7f0ed5904700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed007cea0 0x7f0ed007b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54044/0 (socket says 172.21.15.23:54044) 2024-03-28T05:13:57.690 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.868+0000 7f0ed5904700 1 -- 172.21.15.23:0/837651751 learned_addr learned my addr 172.21.15.23:0/837651751 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:57.690 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.868+0000 7f0ed5904700 1 -- 172.21.15.23:0/837651751 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed007ba50 con 0x7f0ed007cea0 2024-03-28T05:13:57.690 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.868+0000 7f0ed5904700 1 --2- 172.21.15.23:0/837651751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed007cea0 0x7f0ed007b300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc0098a0 tx=0x7f0ecc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1066bc9b506aba4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:57.690 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.868+0000 7f0ed4902700 1 -- 172.21.15.23:0/837651751 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ecc004160 con 0x7f0ed007cea0 2024-03-28T05:13:57.690 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.868+0000 7f0ed4902700 1 -- 172.21.15.23:0/837651751 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ecc004300 con 0x7f0ed007cea0 2024-03-28T05:13:57.690 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.868+0000 7f0ed4902700 1 -- 172.21.15.23:0/837651751 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ecc011b20 con 0x7f0ed007cea0 2024-03-28T05:13:57.690 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.869+0000 7f0ed7b68700 1 -- 172.21.15.23:0/837651751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed007cea0 msgr2=0x7f0ed007b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:57.690 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.869+0000 7f0ed7b68700 1 --2- 172.21.15.23:0/837651751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed007cea0 0x7f0ed007b300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc0098a0 tx=0x7f0ecc009bb0 comp rx=0 tx=0).stop 2024-03-28T05:13:57.690 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.869+0000 7f0ed7b68700 1 -- 172.21.15.23:0/837651751 shutdown_connections 2024-03-28T05:13:57.690 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.869+0000 7f0ed7b68700 1 --2- 172.21.15.23:0/837651751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed007cea0 0x7f0ed007b300 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:57.690 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.869+0000 7f0ed7b68700 1 -- 172.21.15.23:0/837651751 >> 172.21.15.23:0/837651751 conn(0x7f0ed0103040 msgr2=0x7f0ed0105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:57.690 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.869+0000 7f0ed7b68700 1 -- 172.21.15.23:0/837651751 shutdown_connections 2024-03-28T05:13:57.690 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.869+0000 7f0ed7b68700 1 -- 172.21.15.23:0/837651751 wait complete. 2024-03-28T05:13:57.690 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.870+0000 7f0ed7b68700 1 Processor -- start 2024-03-28T05:13:57.690 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.870+0000 7f0ed7b68700 1 -- start start 2024-03-28T05:13:57.691 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.870+0000 7f0ed7b68700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed007cea0 0x7f0ed019db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:57.691 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.870+0000 7f0ed7b68700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed010e060 con 0x7f0ed007cea0 2024-03-28T05:13:57.691 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.870+0000 7f0ed5904700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed007cea0 0x7f0ed019db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:57.691 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.870+0000 7f0ed5904700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed007cea0 0x7f0ed019db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54060/0 (socket says 172.21.15.23:54060) 2024-03-28T05:13:57.691 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.870+0000 7f0ed5904700 1 -- 172.21.15.23:0/1959259913 learned_addr learned my addr 172.21.15.23:0/1959259913 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:57.691 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.870+0000 7f0ed5904700 1 -- 172.21.15.23:0/1959259913 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed019e0c0 con 0x7f0ed007cea0 2024-03-28T05:13:57.691 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.871+0000 7f0ed5904700 1 --2- 172.21.15.23:0/1959259913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed007cea0 0x7f0ed019db80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc011b60 tx=0x7f0ecc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:57.691 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.871+0000 7f0ec2ffd700 1 -- 172.21.15.23:0/1959259913 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ecc003b60 con 0x7f0ed007cea0 2024-03-28T05:13:57.691 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.871+0000 7f0ec2ffd700 1 -- 172.21.15.23:0/1959259913 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ecc01a910 con 0x7f0ed007cea0 2024-03-28T05:13:57.691 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.871+0000 7f0ed7b68700 1 -- 172.21.15.23:0/1959259913 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed019e350 con 0x7f0ed007cea0 2024-03-28T05:13:57.691 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.871+0000 7f0ec2ffd700 1 -- 172.21.15.23:0/1959259913 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ecc024930 con 0x7f0ed007cea0 2024-03-28T05:13:57.691 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.871+0000 7f0ed7b68700 1 -- 172.21.15.23:0/1959259913 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed019e7b0 con 0x7f0ed007cea0 2024-03-28T05:13:57.691 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.872+0000 7f0ed7b68700 1 -- 172.21.15.23:0/1959259913 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed00637c0 con 0x7f0ed007cea0 2024-03-28T05:13:57.691 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.872+0000 7f0ec2ffd700 1 -- 172.21.15.23:0/1959259913 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f0ecc022020 con 0x7f0ed007cea0 2024-03-28T05:13:57.691 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.872+0000 7f0ec2ffd700 1 --2- 172.21.15.23:0/1959259913 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f0ebc038f70 0x7f0ebc03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:57.691 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.872+0000 7f0ec2ffd700 1 -- 172.21.15.23:0/1959259913 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0ecc04d000 con 0x7f0ed007cea0 2024-03-28T05:13:57.692 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.872+0000 7f0ed5103700 1 --2- 172.21.15.23:0/1959259913 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f0ebc038f70 0x7f0ebc03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:57.692 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.873+0000 7f0ed5103700 1 --2- 172.21.15.23:0/1959259913 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f0ebc038f70 0x7f0ebc03b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4006fd0 tx=0x7f0ec4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:57.692 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:56.877+0000 7f0ec2ffd700 1 -- 172.21.15.23:0/1959259913 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0ecc02d770 con 0x7f0ed007cea0 2024-03-28T05:13:57.692 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:57.098+0000 7f0ed7b68700 1 -- 172.21.15.23:0/1959259913 --> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f0ed01042c0 con 0x7f0ebc038f70 2024-03-28T05:13:57.692 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:57.103+0000 7f0ec2ffd700 1 -- 172.21.15.23:0/1959259913 <== mgr.14118 v2:172.21.15.23:6800/1530849123 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f0ed01042c0 con 0x7f0ebc038f70 2024-03-28T05:13:57.692 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:57.105+0000 7f0ed7b68700 1 -- 172.21.15.23:0/1959259913 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f0ebc038f70 msgr2=0x7f0ebc03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:57.692 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:57.105+0000 7f0ed7b68700 1 --2- 172.21.15.23:0/1959259913 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f0ebc038f70 0x7f0ebc03b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4006fd0 tx=0x7f0ec4006e40 comp rx=0 tx=0).stop 2024-03-28T05:13:57.692 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:57.105+0000 7f0ed7b68700 1 -- 172.21.15.23:0/1959259913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed007cea0 msgr2=0x7f0ed019db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:57.692 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:57.105+0000 7f0ed7b68700 1 --2- 172.21.15.23:0/1959259913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed007cea0 0x7f0ed019db80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc011b60 tx=0x7f0ecc011c40 comp rx=0 tx=0).stop 2024-03-28T05:13:57.692 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:57.105+0000 7f0ed7b68700 1 -- 172.21.15.23:0/1959259913 shutdown_connections 2024-03-28T05:13:57.692 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:57.105+0000 7f0ed7b68700 1 --2- 172.21.15.23:0/1959259913 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f0ebc038f70 0x7f0ebc03b430 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:57.692 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:57.105+0000 7f0ed7b68700 1 --2- 172.21.15.23:0/1959259913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ed007cea0 0x7f0ed019db80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:57.692 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:57.105+0000 7f0ed7b68700 1 -- 172.21.15.23:0/1959259913 >> 172.21.15.23:0/1959259913 conn(0x7f0ed0103040 msgr2=0x7f0ed0103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:57.692 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:57.106+0000 7f0ed7b68700 1 -- 172.21.15.23:0/1959259913 shutdown_connections 2024-03-28T05:13:57.692 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:57.106+0000 7f0ed7b68700 1 -- 172.21.15.23:0/1959259913 wait complete. 2024-03-28T05:13:58.261 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:58 smithi023 ceph-mon[31954]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:13:58.261 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:58 smithi023 ceph-mon[31954]: Saving service mgr spec with placement count:2 2024-03-28T05:13:58.261 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:58 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' 2024-03-28T05:13:58.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.071+0000 7ff6ae923700 1 Processor -- start 2024-03-28T05:13:58.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.071+0000 7ff6ae923700 1 -- start start 2024-03-28T05:13:58.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.071+0000 7ff6ae923700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6a8109c40 0x7ff6a810a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:58.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.071+0000 7ff6ae923700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a810a630 con 0x7ff6a8109c40 2024-03-28T05:13:58.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.072+0000 7ff6a7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6a8109c40 0x7ff6a810a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:58.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.072+0000 7ff6a7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6a8109c40 0x7ff6a810a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54074/0 (socket says 172.21.15.23:54074) 2024-03-28T05:13:58.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.072+0000 7ff6a7fff700 1 -- 172.21.15.23:0/2968371641 learned_addr learned my addr 172.21.15.23:0/2968371641 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:58.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.072+0000 7ff6a7fff700 1 -- 172.21.15.23:0/2968371641 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a810ada0 con 0x7ff6a8109c40 2024-03-28T05:13:58.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.072+0000 7ff6a7fff700 1 --2- 172.21.15.23:0/2968371641 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6a8109c40 0x7ff6a810a060 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff6980098a0 tx=0x7ff698009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95736714411dae45 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:58.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.072+0000 7ff6a6ffd700 1 -- 172.21.15.23:0/2968371641 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff698004160 con 0x7ff6a8109c40 2024-03-28T05:13:58.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.072+0000 7ff6a6ffd700 1 -- 172.21.15.23:0/2968371641 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff698004300 con 0x7ff6a8109c40 2024-03-28T05:13:58.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.072+0000 7ff6a6ffd700 1 -- 172.21.15.23:0/2968371641 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff698011b20 con 0x7ff6a8109c40 2024-03-28T05:13:58.973 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.073+0000 7ff6ae923700 1 -- 172.21.15.23:0/2968371641 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6a8109c40 msgr2=0x7ff6a810a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:58.973 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.073+0000 7ff6ae923700 1 --2- 172.21.15.23:0/2968371641 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6a8109c40 0x7ff6a810a060 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff6980098a0 tx=0x7ff698009bb0 comp rx=0 tx=0).stop 2024-03-28T05:13:58.973 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.073+0000 7ff6ae923700 1 -- 172.21.15.23:0/2968371641 shutdown_connections 2024-03-28T05:13:58.973 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.073+0000 7ff6ae923700 1 --2- 172.21.15.23:0/2968371641 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6a8109c40 0x7ff6a810a060 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:58.973 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.073+0000 7ff6ae923700 1 -- 172.21.15.23:0/2968371641 >> 172.21.15.23:0/2968371641 conn(0x7ff6a81054e0 msgr2=0x7ff6a81078d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:58.973 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.073+0000 7ff6ae923700 1 -- 172.21.15.23:0/2968371641 shutdown_connections 2024-03-28T05:13:58.973 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.073+0000 7ff6ae923700 1 -- 172.21.15.23:0/2968371641 wait complete. 2024-03-28T05:13:58.973 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.074+0000 7ff6ae923700 1 Processor -- start 2024-03-28T05:13:58.973 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.074+0000 7ff6ae923700 1 -- start start 2024-03-28T05:13:58.973 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.074+0000 7ff6ae923700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6a8109c40 0x7ff6a819db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:58.973 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.074+0000 7ff6ae923700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a810e150 con 0x7ff6a8109c40 2024-03-28T05:13:58.973 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.074+0000 7ff6a7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6a8109c40 0x7ff6a819db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:58.973 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.074+0000 7ff6a7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6a8109c40 0x7ff6a819db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54078/0 (socket says 172.21.15.23:54078) 2024-03-28T05:13:58.973 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.074+0000 7ff6a7fff700 1 -- 172.21.15.23:0/1366553982 learned_addr learned my addr 172.21.15.23:0/1366553982 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:13:58.973 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.074+0000 7ff6a7fff700 1 -- 172.21.15.23:0/1366553982 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a819e050 con 0x7ff6a8109c40 2024-03-28T05:13:58.974 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.075+0000 7ff6a7fff700 1 --2- 172.21.15.23:0/1366553982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6a8109c40 0x7ff6a819db10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff6980117e0 tx=0x7ff698011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:58.974 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.075+0000 7ff6a57fa700 1 -- 172.21.15.23:0/1366553982 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff698003bf0 con 0x7ff6a8109c40 2024-03-28T05:13:58.974 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.075+0000 7ff6ae923700 1 -- 172.21.15.23:0/1366553982 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6a819e2e0 con 0x7ff6a8109c40 2024-03-28T05:13:58.974 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.075+0000 7ff6a57fa700 1 -- 172.21.15.23:0/1366553982 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff69801a910 con 0x7ff6a8109c40 2024-03-28T05:13:58.974 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.075+0000 7ff6a57fa700 1 -- 172.21.15.23:0/1366553982 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff6980249a0 con 0x7ff6a8109c40 2024-03-28T05:13:58.974 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.075+0000 7ff6ae923700 1 -- 172.21.15.23:0/1366553982 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6a819e740 con 0x7ff6a8109c40 2024-03-28T05:13:58.974 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.076+0000 7ff6ae923700 1 -- 172.21.15.23:0/1366553982 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6a8069220 con 0x7ff6a8109c40 2024-03-28T05:13:58.974 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.076+0000 7ff6a57fa700 1 -- 172.21.15.23:0/1366553982 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7ff698022020 con 0x7ff6a8109c40 2024-03-28T05:13:58.974 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.076+0000 7ff6a57fa700 1 --2- 172.21.15.23:0/1366553982 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7ff690038f20 0x7ff69003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:13:58.974 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.076+0000 7ff6a57fa700 1 -- 172.21.15.23:0/1366553982 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff69804d170 con 0x7ff6a8109c40 2024-03-28T05:13:58.974 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.077+0000 7ff6a77fe700 1 --2- 172.21.15.23:0/1366553982 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7ff690038f20 0x7ff69003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:13:58.974 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.077+0000 7ff6a77fe700 1 --2- 172.21.15.23:0/1366553982 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7ff690038f20 0x7ff69003b3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff69c006fd0 tx=0x7ff69c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:13:58.974 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.082+0000 7ff6a57fa700 1 -- 172.21.15.23:0/1366553982 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff698011e20 con 0x7ff6a8109c40 2024-03-28T05:13:58.974 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.286+0000 7ff6ae923700 1 -- 172.21.15.23:0/1366553982 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7ff6a81a0fd0 con 0x7ff6a8109c40 2024-03-28T05:13:58.975 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.288+0000 7ff6a57fa700 1 -- 172.21.15.23:0/1366553982 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7ff69800ba00 con 0x7ff6a8109c40 2024-03-28T05:13:58.975 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.291+0000 7ff6ae923700 1 -- 172.21.15.23:0/1366553982 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7ff690038f20 msgr2=0x7ff69003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:58.975 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.291+0000 7ff6ae923700 1 --2- 172.21.15.23:0/1366553982 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7ff690038f20 0x7ff69003b3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff69c006fd0 tx=0x7ff69c006e40 comp rx=0 tx=0).stop 2024-03-28T05:13:58.975 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.291+0000 7ff6ae923700 1 -- 172.21.15.23:0/1366553982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6a8109c40 msgr2=0x7ff6a819db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:13:58.975 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.291+0000 7ff6ae923700 1 --2- 172.21.15.23:0/1366553982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6a8109c40 0x7ff6a819db10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff6980117e0 tx=0x7ff698011c10 comp rx=0 tx=0).stop 2024-03-28T05:13:58.975 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.291+0000 7ff6ae923700 1 -- 172.21.15.23:0/1366553982 shutdown_connections 2024-03-28T05:13:58.975 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.291+0000 7ff6ae923700 1 --2- 172.21.15.23:0/1366553982 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7ff690038f20 0x7ff69003b3e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:58.975 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.291+0000 7ff6ae923700 1 --2- 172.21.15.23:0/1366553982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6a8109c40 0x7ff6a819db10 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:13:58.975 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.291+0000 7ff6ae923700 1 -- 172.21.15.23:0/1366553982 >> 172.21.15.23:0/1366553982 conn(0x7ff6a81054e0 msgr2=0x7ff6a8107030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:13:58.975 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.291+0000 7ff6ae923700 1 -- 172.21.15.23:0/1366553982 shutdown_connections 2024-03-28T05:13:58.975 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:58.291+0000 7ff6ae923700 1 -- 172.21.15.23:0/1366553982 wait complete. 2024-03-28T05:13:59.602 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:13:59 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1366553982' entity='client.admin' 2024-03-28T05:14:00.488 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.711+0000 7f9785b9e700 1 Processor -- start 2024-03-28T05:14:00.488 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.711+0000 7f9785b9e700 1 -- start start 2024-03-28T05:14:00.488 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.711+0000 7f9785b9e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9780107a20 0x7f9780107e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:00.489 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.711+0000 7f9785b9e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9780108410 con 0x7f9780107a20 2024-03-28T05:14:00.489 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.711+0000 7f977f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9780107a20 0x7f9780107e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:00.489 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.711+0000 7f977f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9780107a20 0x7f9780107e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54086/0 (socket says 172.21.15.23:54086) 2024-03-28T05:14:00.489 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.711+0000 7f977f7fe700 1 -- 172.21.15.23:0/3719780004 learned_addr learned my addr 172.21.15.23:0/3719780004 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:00.489 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.711+0000 7f977f7fe700 1 -- 172.21.15.23:0/3719780004 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9780108c20 con 0x7f9780107a20 2024-03-28T05:14:00.489 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.712+0000 7f977f7fe700 1 --2- 172.21.15.23:0/3719780004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9780107a20 0x7f9780107e40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f97700098a0 tx=0x7f9770009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259ed14325007d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:00.489 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.712+0000 7f977e7fc700 1 -- 172.21.15.23:0/3719780004 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9770004160 con 0x7f9780107a20 2024-03-28T05:14:00.489 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.712+0000 7f977e7fc700 1 -- 172.21.15.23:0/3719780004 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9770004300 con 0x7f9780107a20 2024-03-28T05:14:00.489 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.712+0000 7f977e7fc700 1 -- 172.21.15.23:0/3719780004 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9770011b20 con 0x7f9780107a20 2024-03-28T05:14:00.489 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.712+0000 7f9785b9e700 1 -- 172.21.15.23:0/3719780004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9780107a20 msgr2=0x7f9780107e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:00.489 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.712+0000 7f9785b9e700 1 --2- 172.21.15.23:0/3719780004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9780107a20 0x7f9780107e40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f97700098a0 tx=0x7f9770009bb0 comp rx=0 tx=0).stop 2024-03-28T05:14:00.489 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.712+0000 7f9785b9e700 1 -- 172.21.15.23:0/3719780004 shutdown_connections 2024-03-28T05:14:00.489 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.712+0000 7f9785b9e700 1 --2- 172.21.15.23:0/3719780004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9780107a20 0x7f9780107e40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:00.489 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.712+0000 7f9785b9e700 1 -- 172.21.15.23:0/3719780004 >> 172.21.15.23:0/3719780004 conn(0x7f97801030a0 msgr2=0x7f97801054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:00.489 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.712+0000 7f9785b9e700 1 -- 172.21.15.23:0/3719780004 shutdown_connections 2024-03-28T05:14:00.489 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.712+0000 7f9785b9e700 1 -- 172.21.15.23:0/3719780004 wait complete. 2024-03-28T05:14:00.490 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.713+0000 7f9785b9e700 1 Processor -- start 2024-03-28T05:14:00.490 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.713+0000 7f9785b9e700 1 -- start start 2024-03-28T05:14:00.490 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.713+0000 7f9785b9e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9780107a20 0x7f978019b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:00.490 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.713+0000 7f9785b9e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978010bfd0 con 0x7f9780107a20 2024-03-28T05:14:00.490 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.713+0000 7f977f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9780107a20 0x7f978019b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:00.490 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.713+0000 7f977f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9780107a20 0x7f978019b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54100/0 (socket says 172.21.15.23:54100) 2024-03-28T05:14:00.490 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.713+0000 7f977f7fe700 1 -- 172.21.15.23:0/178202819 learned_addr learned my addr 172.21.15.23:0/178202819 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:00.490 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.714+0000 7f977f7fe700 1 -- 172.21.15.23:0/178202819 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978019be90 con 0x7f9780107a20 2024-03-28T05:14:00.490 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.714+0000 7f977f7fe700 1 --2- 172.21.15.23:0/178202819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9780107a20 0x7f978019b950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f97700117e0 tx=0x7f9770011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:00.490 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.714+0000 7f977cff9700 1 -- 172.21.15.23:0/178202819 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9770003bf0 con 0x7f9780107a20 2024-03-28T05:14:00.490 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.714+0000 7f977cff9700 1 -- 172.21.15.23:0/178202819 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f977001a910 con 0x7f9780107a20 2024-03-28T05:14:00.490 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.714+0000 7f9785b9e700 1 -- 172.21.15.23:0/178202819 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978019c120 con 0x7f9780107a20 2024-03-28T05:14:00.490 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.714+0000 7f977cff9700 1 -- 172.21.15.23:0/178202819 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97700249a0 con 0x7f9780107a20 2024-03-28T05:14:00.490 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.714+0000 7f9785b9e700 1 -- 172.21.15.23:0/178202819 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978019c580 con 0x7f9780107a20 2024-03-28T05:14:00.491 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.715+0000 7f977cff9700 1 -- 172.21.15.23:0/178202819 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f9770022020 con 0x7f9780107a20 2024-03-28T05:14:00.491 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.715+0000 7f9785b9e700 1 -- 172.21.15.23:0/178202819 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97800637c0 con 0x7f9780107a20 2024-03-28T05:14:00.491 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.715+0000 7f977cff9700 1 --2- 172.21.15.23:0/178202819 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f9768038ed0 0x7f976803b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:00.491 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.715+0000 7f977cff9700 1 -- 172.21.15.23:0/178202819 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f977004d170 con 0x7f9780107a20 2024-03-28T05:14:00.491 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.715+0000 7f977effd700 1 --2- 172.21.15.23:0/178202819 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f9768038ed0 0x7f976803b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:00.491 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.716+0000 7f977effd700 1 --2- 172.21.15.23:0/178202819 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f9768038ed0 0x7f976803b390 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9774006fd0 tx=0x7f9774006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:00.491 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.720+0000 7f977cff9700 1 -- 172.21.15.23:0/178202819 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9770011e20 con 0x7f9780107a20 2024-03-28T05:14:00.491 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.941+0000 7f9785b9e700 1 -- 172.21.15.23:0/178202819 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f978019ef50 con 0x7f9780107a20 2024-03-28T05:14:00.491 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.945+0000 7f977cff9700 1 -- 172.21.15.23:0/178202819 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f977000b9d0 con 0x7f9780107a20 2024-03-28T05:14:00.491 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.948+0000 7f9785b9e700 1 -- 172.21.15.23:0/178202819 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f9768038ed0 msgr2=0x7f976803b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:00.491 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.948+0000 7f9785b9e700 1 --2- 172.21.15.23:0/178202819 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f9768038ed0 0x7f976803b390 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9774006fd0 tx=0x7f9774006e40 comp rx=0 tx=0).stop 2024-03-28T05:14:00.491 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.948+0000 7f9785b9e700 1 -- 172.21.15.23:0/178202819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9780107a20 msgr2=0x7f978019b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:00.491 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.948+0000 7f9785b9e700 1 --2- 172.21.15.23:0/178202819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9780107a20 0x7f978019b950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f97700117e0 tx=0x7f9770011c10 comp rx=0 tx=0).stop 2024-03-28T05:14:00.491 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.948+0000 7f9785b9e700 1 -- 172.21.15.23:0/178202819 shutdown_connections 2024-03-28T05:14:00.491 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.948+0000 7f9785b9e700 1 --2- 172.21.15.23:0/178202819 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f9768038ed0 0x7f976803b390 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:00.492 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.948+0000 7f9785b9e700 1 --2- 172.21.15.23:0/178202819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9780107a20 0x7f978019b950 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:00.492 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.948+0000 7f9785b9e700 1 -- 172.21.15.23:0/178202819 >> 172.21.15.23:0/178202819 conn(0x7f97801030a0 msgr2=0x7f9780103b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:00.492 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.948+0000 7f9785b9e700 1 -- 172.21.15.23:0/178202819 shutdown_connections 2024-03-28T05:14:00.492 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:13:59.948+0000 7f9785b9e700 1 -- 172.21.15.23:0/178202819 wait complete. 2024-03-28T05:14:00.492 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling the dashboard module... 2024-03-28T05:14:00.609 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:00 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' 2024-03-28T05:14:00.609 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:00 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/178202819' entity='client.admin' 2024-03-28T05:14:01.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:01 smithi023 ceph-mon[31954]: from='mgr.14118 172.21.15.23:0/2063525849' entity='mgr.x' 2024-03-28T05:14:01.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:01 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3788960767' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-28T05:14:01.778 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:01 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: ignoring --setuser ceph since I am not root 2024-03-28T05:14:01.778 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:01 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: ignoring --setgroup ceph since I am not root 2024-03-28T05:14:02.420 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:02 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:02.165+0000 7ffba01b5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T05:14:02.420 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:02 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:02.260+0000 7ffba01b5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T05:14:02.627 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.908+0000 7fc27397f700 1 Processor -- start 2024-03-28T05:14:02.627 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.908+0000 7fc27397f700 1 -- start start 2024-03-28T05:14:02.627 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.909+0000 7fc27397f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc26c108480 0x7fc26c10a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:02.627 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.909+0000 7fc27397f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26c07ae70 con 0x7fc26c108480 2024-03-28T05:14:02.627 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.909+0000 7fc27171b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc26c108480 0x7fc26c10a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:02.627 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.909+0000 7fc27171b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc26c108480 0x7fc26c10a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54114/0 (socket says 172.21.15.23:54114) 2024-03-28T05:14:02.627 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.909+0000 7fc27171b700 1 -- 172.21.15.23:0/2089656614 learned_addr learned my addr 172.21.15.23:0/2089656614 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:02.627 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.909+0000 7fc27171b700 1 -- 172.21.15.23:0/2089656614 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26c10adb0 con 0x7fc26c108480 2024-03-28T05:14:02.627 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.909+0000 7fc27171b700 1 --2- 172.21.15.23:0/2089656614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc26c108480 0x7fc26c10a870 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc2680098a0 tx=0x7fc268009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba755dccf4efd0b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:02.627 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.910+0000 7fc263fff700 1 -- 172.21.15.23:0/2089656614 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc268004160 con 0x7fc26c108480 2024-03-28T05:14:02.627 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.910+0000 7fc263fff700 1 -- 172.21.15.23:0/2089656614 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc268004300 con 0x7fc26c108480 2024-03-28T05:14:02.627 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.910+0000 7fc263fff700 1 -- 172.21.15.23:0/2089656614 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc268011b20 con 0x7fc26c108480 2024-03-28T05:14:02.628 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.910+0000 7fc27397f700 1 -- 172.21.15.23:0/2089656614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc26c108480 msgr2=0x7fc26c10a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:02.628 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.910+0000 7fc27397f700 1 --2- 172.21.15.23:0/2089656614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc26c108480 0x7fc26c10a870 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc2680098a0 tx=0x7fc268009bb0 comp rx=0 tx=0).stop 2024-03-28T05:14:02.628 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.910+0000 7fc27397f700 1 -- 172.21.15.23:0/2089656614 shutdown_connections 2024-03-28T05:14:02.628 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.910+0000 7fc27397f700 1 --2- 172.21.15.23:0/2089656614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc26c108480 0x7fc26c10a870 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:02.628 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.910+0000 7fc27397f700 1 -- 172.21.15.23:0/2089656614 >> 172.21.15.23:0/2089656614 conn(0x7fc26c101f10 msgr2=0x7fc26c104330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:02.628 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.910+0000 7fc27397f700 1 -- 172.21.15.23:0/2089656614 shutdown_connections 2024-03-28T05:14:02.628 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.911+0000 7fc27397f700 1 -- 172.21.15.23:0/2089656614 wait complete. 2024-03-28T05:14:02.628 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.911+0000 7fc27397f700 1 Processor -- start 2024-03-28T05:14:02.628 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.911+0000 7fc27397f700 1 -- start start 2024-03-28T05:14:02.628 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.911+0000 7fc27397f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc26c108480 0x7fc26c107d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:02.628 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.911+0000 7fc27397f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26c10bdc0 con 0x7fc26c108480 2024-03-28T05:14:02.628 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.911+0000 7fc27171b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc26c108480 0x7fc26c107d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:02.628 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.911+0000 7fc27171b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc26c108480 0x7fc26c107d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54120/0 (socket says 172.21.15.23:54120) 2024-03-28T05:14:02.628 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.911+0000 7fc27171b700 1 -- 172.21.15.23:0/3788960767 learned_addr learned my addr 172.21.15.23:0/3788960767 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:02.628 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.912+0000 7fc27171b700 1 -- 172.21.15.23:0/3788960767 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26c108240 con 0x7fc26c108480 2024-03-28T05:14:02.628 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.912+0000 7fc27171b700 1 --2- 172.21.15.23:0/3788960767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc26c108480 0x7fc26c107d00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc2680117e0 tx=0x7fc268011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:02.629 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.912+0000 7fc2627fc700 1 -- 172.21.15.23:0/3788960767 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc268003ba0 con 0x7fc26c108480 2024-03-28T05:14:02.629 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.912+0000 7fc27397f700 1 -- 172.21.15.23:0/3788960767 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc26c106430 con 0x7fc26c108480 2024-03-28T05:14:02.629 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.912+0000 7fc2627fc700 1 -- 172.21.15.23:0/3788960767 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc26801a910 con 0x7fc26c108480 2024-03-28T05:14:02.629 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.912+0000 7fc27397f700 1 -- 172.21.15.23:0/3788960767 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc26c1068f0 con 0x7fc26c108480 2024-03-28T05:14:02.629 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.912+0000 7fc2627fc700 1 -- 172.21.15.23:0/3788960767 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc2680248f0 con 0x7fc26c108480 2024-03-28T05:14:02.629 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.913+0000 7fc2627fc700 1 -- 172.21.15.23:0/3788960767 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fc268022020 con 0x7fc26c108480 2024-03-28T05:14:02.629 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.913+0000 7fc27397f700 1 -- 172.21.15.23:0/3788960767 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc26c0637c0 con 0x7fc26c108480 2024-03-28T05:14:02.629 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.913+0000 7fc2627fc700 1 --2- 172.21.15.23:0/3788960767 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7fc258038f20 0x7fc25803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:02.629 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.913+0000 7fc2627fc700 1 -- 172.21.15.23:0/3788960767 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc26804d6a0 con 0x7fc26c108480 2024-03-28T05:14:02.629 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.913+0000 7fc270f1a700 1 --2- 172.21.15.23:0/3788960767 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7fc258038f20 0x7fc25803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:02.629 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.914+0000 7fc270f1a700 1 --2- 172.21.15.23:0/3788960767 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7fc258038f20 0x7fc25803b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc25c006fd0 tx=0x7fc25c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:02.629 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:00.918+0000 7fc2627fc700 1 -- 172.21.15.23:0/3788960767 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc2680172b0 con 0x7fc26c108480 2024-03-28T05:14:02.629 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:01.188+0000 7fc27397f700 1 -- 172.21.15.23:0/3788960767 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fc26c106b70 con 0x7fc26c108480 2024-03-28T05:14:02.629 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:01.581+0000 7fc2627fc700 1 -- 172.21.15.23:0/3788960767 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fc26800b5a0 con 0x7fc26c108480 2024-03-28T05:14:02.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:01.582+0000 7fc2627fc700 1 -- 172.21.15.23:0/3788960767 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fc26804fb80 con 0x7fc26c108480 2024-03-28T05:14:02.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:01.583+0000 7fc27397f700 1 -- 172.21.15.23:0/3788960767 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7fc258038f20 msgr2=0x7fc25803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:02.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:01.583+0000 7fc27397f700 1 --2- 172.21.15.23:0/3788960767 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7fc258038f20 0x7fc25803b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc25c006fd0 tx=0x7fc25c006e40 comp rx=0 tx=0).stop 2024-03-28T05:14:02.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:01.583+0000 7fc27397f700 1 -- 172.21.15.23:0/3788960767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc26c108480 msgr2=0x7fc26c107d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:02.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:01.583+0000 7fc27397f700 1 --2- 172.21.15.23:0/3788960767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc26c108480 0x7fc26c107d00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc2680117e0 tx=0x7fc268011ba0 comp rx=0 tx=0).stop 2024-03-28T05:14:02.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:01.583+0000 7fc27397f700 1 -- 172.21.15.23:0/3788960767 shutdown_connections 2024-03-28T05:14:02.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:01.583+0000 7fc27397f700 1 --2- 172.21.15.23:0/3788960767 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7fc258038f20 0x7fc25803b3e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:02.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:01.583+0000 7fc27397f700 1 --2- 172.21.15.23:0/3788960767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc26c108480 0x7fc26c107d00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:02.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:01.583+0000 7fc27397f700 1 -- 172.21.15.23:0/3788960767 >> 172.21.15.23:0/3788960767 conn(0x7fc26c101f10 msgr2=0x7fc26c1029a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:02.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:01.584+0000 7fc27397f700 1 -- 172.21.15.23:0/3788960767 shutdown_connections 2024-03-28T05:14:02.630 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:01.584+0000 7fc27397f700 1 -- 172.21.15.23:0/3788960767 wait complete. 2024-03-28T05:14:02.746 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:02 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3788960767' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-28T05:14:02.746 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:02 smithi023 ceph-mon[31954]: mgrmap e9: x(active, since 19s) 2024-03-28T05:14:03.024 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:02 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:02.930+0000 7ffba01b5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T05:14:03.024 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:03 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:03.027+0000 7ffba01b5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T05:14:03.278 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:03 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:03.175+0000 7ffba01b5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T05:14:03.278 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:03 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:03.273+0000 7ffba01b5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T05:14:03.586 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:03 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:03.363+0000 7ffba01b5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T05:14:03.848 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:03 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3659838235' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-28T05:14:03.848 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:03 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:03.641+0000 7ffba01b5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T05:14:03.848 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:03 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:03.759+0000 7ffba01b5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T05:14:03.848 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:03 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:03.850+0000 7ffba01b5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T05:14:03.929 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-28T05:14:03.938 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-28T05:14:03.938 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T05:14:03.938 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-28T05:14:03.938 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-28T05:14:03.938 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-28T05:14:03.938 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.059+0000 7f410fe5c700 1 Processor -- start 2024-03-28T05:14:03.938 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.060+0000 7f410fe5c700 1 -- start start 2024-03-28T05:14:03.938 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.060+0000 7f410fe5c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081084b0 0x7f410810a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:03.939 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.060+0000 7f410fe5c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410807ae70 con 0x7f41081084b0 2024-03-28T05:14:03.939 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.060+0000 7f410dbf8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081084b0 0x7f410810a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:03.939 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.060+0000 7f410dbf8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081084b0 0x7f410810a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45788/0 (socket says 172.21.15.23:45788) 2024-03-28T05:14:03.939 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.060+0000 7f410dbf8700 1 -- 172.21.15.23:0/2080676607 learned_addr learned my addr 172.21.15.23:0/2080676607 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:03.939 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.060+0000 7f410dbf8700 1 -- 172.21.15.23:0/2080676607 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410810ade0 con 0x7f41081084b0 2024-03-28T05:14:03.939 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.060+0000 7f410dbf8700 1 --2- 172.21.15.23:0/2080676607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081084b0 0x7f410810a8a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f41040098a0 tx=0x7f4104009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a20914937de26f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:03.939 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.061+0000 7f410cbf6700 1 -- 172.21.15.23:0/2080676607 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4104004160 con 0x7f41081084b0 2024-03-28T05:14:03.939 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.061+0000 7f410cbf6700 1 -- 172.21.15.23:0/2080676607 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4104004300 con 0x7f41081084b0 2024-03-28T05:14:03.939 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.061+0000 7f410cbf6700 1 -- 172.21.15.23:0/2080676607 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4104011b20 con 0x7f41081084b0 2024-03-28T05:14:03.939 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.061+0000 7f410fe5c700 1 -- 172.21.15.23:0/2080676607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081084b0 msgr2=0x7f410810a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:03.939 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.061+0000 7f410fe5c700 1 --2- 172.21.15.23:0/2080676607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081084b0 0x7f410810a8a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f41040098a0 tx=0x7f4104009bb0 comp rx=0 tx=0).stop 2024-03-28T05:14:03.939 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.061+0000 7f410fe5c700 1 -- 172.21.15.23:0/2080676607 shutdown_connections 2024-03-28T05:14:03.939 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.061+0000 7f410fe5c700 1 --2- 172.21.15.23:0/2080676607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081084b0 0x7f410810a8a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:03.939 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.061+0000 7f410fe5c700 1 -- 172.21.15.23:0/2080676607 >> 172.21.15.23:0/2080676607 conn(0x7f4108101ee0 msgr2=0x7f4108104300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:03.940 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.061+0000 7f410fe5c700 1 -- 172.21.15.23:0/2080676607 shutdown_connections 2024-03-28T05:14:03.940 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.061+0000 7f410fe5c700 1 -- 172.21.15.23:0/2080676607 wait complete. 2024-03-28T05:14:03.940 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.062+0000 7f410fe5c700 1 Processor -- start 2024-03-28T05:14:03.940 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.062+0000 7f410fe5c700 1 -- start start 2024-03-28T05:14:03.940 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.062+0000 7f410fe5c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081084b0 0x7f410819b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:03.940 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.062+0000 7f410fe5c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410810bdf0 con 0x7f41081084b0 2024-03-28T05:14:03.940 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.062+0000 7f410dbf8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081084b0 0x7f410819b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:03.940 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.062+0000 7f410dbf8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081084b0 0x7f410819b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45802/0 (socket says 172.21.15.23:45802) 2024-03-28T05:14:03.940 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.062+0000 7f410dbf8700 1 -- 172.21.15.23:0/3659838235 learned_addr learned my addr 172.21.15.23:0/3659838235 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:03.940 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.063+0000 7f410dbf8700 1 -- 172.21.15.23:0/3659838235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410819bed0 con 0x7f41081084b0 2024-03-28T05:14:03.940 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.063+0000 7f410dbf8700 1 --2- 172.21.15.23:0/3659838235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081084b0 0x7f410819b990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4104011b60 tx=0x7f4104011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:03.940 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.063+0000 7f40feffd700 1 -- 172.21.15.23:0/3659838235 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4104003b60 con 0x7f41081084b0 2024-03-28T05:14:03.940 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.063+0000 7f40feffd700 1 -- 172.21.15.23:0/3659838235 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f410401a910 con 0x7f41081084b0 2024-03-28T05:14:03.940 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.063+0000 7f410fe5c700 1 -- 172.21.15.23:0/3659838235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f410819c160 con 0x7f41081084b0 2024-03-28T05:14:03.940 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.063+0000 7f410fe5c700 1 -- 172.21.15.23:0/3659838235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f410819c5c0 con 0x7f41081084b0 2024-03-28T05:14:03.940 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.063+0000 7f40feffd700 1 -- 172.21.15.23:0/3659838235 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4104024930 con 0x7f41081084b0 2024-03-28T05:14:03.941 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.064+0000 7f40feffd700 1 -- 172.21.15.23:0/3659838235 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f4104022020 con 0x7f41081084b0 2024-03-28T05:14:03.941 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.064+0000 7f410fe5c700 1 -- 172.21.15.23:0/3659838235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4108069220 con 0x7f41081084b0 2024-03-28T05:14:03.941 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.064+0000 7f40feffd700 1 --2- 172.21.15.23:0/3659838235 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f40f4038fd0 0x7f40f403b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:03.941 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.064+0000 7f40feffd700 1 -- 172.21.15.23:0/3659838235 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f410404d160 con 0x7f41081084b0 2024-03-28T05:14:03.941 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.064+0000 7f410d3f7700 1 -- 172.21.15.23:0/3659838235 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f40f4038fd0 msgr2=0x7f40f403b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1530849123 2024-03-28T05:14:03.941 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.064+0000 7f410d3f7700 1 --2- 172.21.15.23:0/3659838235 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f40f4038fd0 0x7f40f403b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T05:14:03.941 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.069+0000 7f40feffd700 1 -- 172.21.15.23:0/3659838235 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f410404cab0 con 0x7f41081084b0 2024-03-28T05:14:03.941 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.265+0000 7f410d3f7700 1 -- 172.21.15.23:0/3659838235 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f40f4038fd0 msgr2=0x7f40f403b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1530849123 2024-03-28T05:14:03.941 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.265+0000 7f410d3f7700 1 --2- 172.21.15.23:0/3659838235 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f40f4038fd0 0x7f40f403b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T05:14:03.941 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.336+0000 7f410fe5c700 1 -- 172.21.15.23:0/3659838235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f410819ef70 con 0x7f41081084b0 2024-03-28T05:14:03.941 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.336+0000 7f40feffd700 1 -- 172.21.15.23:0/3659838235 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f410404f110 con 0x7f41081084b0 2024-03-28T05:14:03.941 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.338+0000 7f410fe5c700 1 -- 172.21.15.23:0/3659838235 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f40f4038fd0 msgr2=0x7f40f403b490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T05:14:03.941 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.338+0000 7f410fe5c700 1 --2- 172.21.15.23:0/3659838235 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f40f4038fd0 0x7f40f403b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:03.941 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.338+0000 7f410fe5c700 1 -- 172.21.15.23:0/3659838235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081084b0 msgr2=0x7f410819b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:03.941 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.338+0000 7f410fe5c700 1 --2- 172.21.15.23:0/3659838235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081084b0 0x7f410819b990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4104011b60 tx=0x7f4104011c40 comp rx=0 tx=0).stop 2024-03-28T05:14:03.941 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.338+0000 7f410fe5c700 1 -- 172.21.15.23:0/3659838235 shutdown_connections 2024-03-28T05:14:03.942 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.338+0000 7f410fe5c700 1 --2- 172.21.15.23:0/3659838235 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7f40f4038fd0 0x7f40f403b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:03.942 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.338+0000 7f410fe5c700 1 --2- 172.21.15.23:0/3659838235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081084b0 0x7f410819b990 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:03.942 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.338+0000 7f410fe5c700 1 -- 172.21.15.23:0/3659838235 >> 172.21.15.23:0/3659838235 conn(0x7f4108101ee0 msgr2=0x7f4108102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:03.942 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.338+0000 7f410fe5c700 1 -- 172.21.15.23:0/3659838235 shutdown_connections 2024-03-28T05:14:03.942 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:03.338+0000 7f410fe5c700 1 -- 172.21.15.23:0/3659838235 wait complete. 2024-03-28T05:14:03.942 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for the mgr to restart... 2024-03-28T05:14:03.942 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr epoch 9... 2024-03-28T05:14:04.136 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:03 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:03.957+0000 7ffba01b5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T05:14:04.528 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:04 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:04.148+0000 7ffba01b5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T05:14:05.276 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:05 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:05.146+0000 7ffba01b5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T05:14:05.276 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:05 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:05.279+0000 7ffba01b5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T05:14:05.905 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:05 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:05.617+0000 7ffba01b5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T05:14:05.905 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:05 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:05.708+0000 7ffba01b5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T05:14:05.905 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:05 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:05.908+0000 7ffba01b5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T05:14:06.278 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:06 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:06.097+0000 7ffba01b5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T05:14:06.644 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:06 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:06.387+0000 7ffba01b5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T05:14:06.644 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:06 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:06.647+0000 7ffba01b5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T05:14:07.028 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:06 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:06.889+0000 7ffba01b5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T05:14:08.028 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:07 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:07.717+0000 7ffba01b5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T05:14:08.623 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:08 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:08.301+0000 7ffba01b5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T05:14:08.623 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:08 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:08.403+0000 7ffba01b5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T05:14:08.623 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:08 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:08.626+0000 7ffba01b5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T05:14:09.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:08 smithi023 ceph-mon[31954]: Active manager daemon x restarted 2024-03-28T05:14:09.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:08 smithi023 ceph-mon[31954]: Activating manager daemon x 2024-03-28T05:14:09.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:08 smithi023 ceph-mon[31954]: osdmap e3: 0 total, 0 up, 0 in 2024-03-28T05:14:09.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:08 smithi023 ceph-mon[31954]: mgrmap e10: x(active, starting, since 0.00535195s) 2024-03-28T05:14:09.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:08 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T05:14:09.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:08 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T05:14:09.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:08 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T05:14:09.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:08 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T05:14:09.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:08 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T05:14:09.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:08 smithi023 ceph-mon[31954]: Manager daemon x is now available 2024-03-28T05:14:09.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:08 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:14:09.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:08 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:14:10.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T05:14:10.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T05:14:10.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:09 smithi023 ceph-mon[31954]: [28/Mar/2024:05:14:09] ENGINE Bus STARTING 2024-03-28T05:14:10.026 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:09 smithi023 ceph-mon[31954]: mgrmap e11: x(active, since 1.00988s) 2024-03-28T05:14:10.293 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-28T05:14:10.293 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-28T05:14:10.294 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-28T05:14:10.294 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-28T05:14:10.294 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.351+0000 7ff551e83700 1 Processor -- start 2024-03-28T05:14:10.294 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.351+0000 7ff551e83700 1 -- start start 2024-03-28T05:14:10.294 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.351+0000 7ff551e83700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff54c107a20 0x7ff54c107e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:10.294 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.351+0000 7ff551e83700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c108410 con 0x7ff54c107a20 2024-03-28T05:14:10.294 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.352+0000 7ff54b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff54c107a20 0x7ff54c107e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:10.294 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.352+0000 7ff54b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff54c107a20 0x7ff54c107e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45818/0 (socket says 172.21.15.23:45818) 2024-03-28T05:14:10.294 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.352+0000 7ff54b7fe700 1 -- 172.21.15.23:0/2648379124 learned_addr learned my addr 172.21.15.23:0/2648379124 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:10.294 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.352+0000 7ff54b7fe700 1 -- 172.21.15.23:0/2648379124 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54c108bd0 con 0x7ff54c107a20 2024-03-28T05:14:10.294 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.352+0000 7ff54b7fe700 1 --2- 172.21.15.23:0/2648379124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff54c107a20 0x7ff54c107e40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0098a0 tx=0x7ff53c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed67907c9efce40 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:10.294 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.352+0000 7ff54a7fc700 1 -- 172.21.15.23:0/2648379124 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff53c004160 con 0x7ff54c107a20 2024-03-28T05:14:10.294 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.352+0000 7ff54a7fc700 1 -- 172.21.15.23:0/2648379124 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff53c004300 con 0x7ff54c107a20 2024-03-28T05:14:10.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.352+0000 7ff54a7fc700 1 -- 172.21.15.23:0/2648379124 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff53c011b20 con 0x7ff54c107a20 2024-03-28T05:14:10.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.353+0000 7ff551e83700 1 -- 172.21.15.23:0/2648379124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff54c107a20 msgr2=0x7ff54c107e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:10.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.353+0000 7ff551e83700 1 --2- 172.21.15.23:0/2648379124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff54c107a20 0x7ff54c107e40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0098a0 tx=0x7ff53c009bb0 comp rx=0 tx=0).stop 2024-03-28T05:14:10.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.353+0000 7ff551e83700 1 -- 172.21.15.23:0/2648379124 shutdown_connections 2024-03-28T05:14:10.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.353+0000 7ff551e83700 1 --2- 172.21.15.23:0/2648379124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff54c107a20 0x7ff54c107e40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:10.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.353+0000 7ff551e83700 1 -- 172.21.15.23:0/2648379124 >> 172.21.15.23:0/2648379124 conn(0x7ff54c103020 msgr2=0x7ff54c105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:10.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.353+0000 7ff551e83700 1 -- 172.21.15.23:0/2648379124 shutdown_connections 2024-03-28T05:14:10.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.353+0000 7ff551e83700 1 -- 172.21.15.23:0/2648379124 wait complete. 2024-03-28T05:14:10.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.354+0000 7ff551e83700 1 Processor -- start 2024-03-28T05:14:10.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.354+0000 7ff551e83700 1 -- start start 2024-03-28T05:14:10.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.354+0000 7ff551e83700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff54c107a20 0x7ff54c19b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:10.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.354+0000 7ff551e83700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c10bf80 con 0x7ff54c107a20 2024-03-28T05:14:10.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.354+0000 7ff54b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff54c107a20 0x7ff54c19b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:10.295 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.354+0000 7ff54b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff54c107a20 0x7ff54c19b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45820/0 (socket says 172.21.15.23:45820) 2024-03-28T05:14:10.296 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.354+0000 7ff54b7fe700 1 -- 172.21.15.23:0/2137567858 learned_addr learned my addr 172.21.15.23:0/2137567858 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:10.296 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.355+0000 7ff54b7fe700 1 -- 172.21.15.23:0/2137567858 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54c19bea0 con 0x7ff54c107a20 2024-03-28T05:14:10.296 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.355+0000 7ff54b7fe700 1 --2- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff54c107a20 0x7ff54c19b960 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0117e0 tx=0x7ff53c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:10.296 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.355+0000 7ff548ff9700 1 -- 172.21.15.23:0/2137567858 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff53c003bf0 con 0x7ff54c107a20 2024-03-28T05:14:10.296 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.355+0000 7ff551e83700 1 -- 172.21.15.23:0/2137567858 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff54c19c130 con 0x7ff54c107a20 2024-03-28T05:14:10.296 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.355+0000 7ff548ff9700 1 -- 172.21.15.23:0/2137567858 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff53c01a910 con 0x7ff54c107a20 2024-03-28T05:14:10.296 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.355+0000 7ff548ff9700 1 -- 172.21.15.23:0/2137567858 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff53c0249a0 con 0x7ff54c107a20 2024-03-28T05:14:10.296 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.355+0000 7ff551e83700 1 -- 172.21.15.23:0/2137567858 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff54c19c590 con 0x7ff54c107a20 2024-03-28T05:14:10.296 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.356+0000 7ff548ff9700 1 -- 172.21.15.23:0/2137567858 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7ff53c022020 con 0x7ff54c107a20 2024-03-28T05:14:10.296 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.356+0000 7ff548ff9700 1 --2- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7ff534038f20 0x7ff53403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:10.296 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.356+0000 7ff548ff9700 1 -- 172.21.15.23:0/2137567858 --> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff53403baf0 con 0x7ff534038f20 2024-03-28T05:14:10.296 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.356+0000 7ff548ff9700 1 -- 172.21.15.23:0/2137567858 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff53c04d210 con 0x7ff54c107a20 2024-03-28T05:14:10.296 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.356+0000 7ff54affd700 1 -- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7ff534038f20 msgr2=0x7ff53403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1530849123 2024-03-28T05:14:10.296 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.356+0000 7ff54affd700 1 --2- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7ff534038f20 0x7ff53403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T05:14:10.296 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.557+0000 7ff54affd700 1 -- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7ff534038f20 msgr2=0x7ff53403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1530849123 2024-03-28T05:14:10.296 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.557+0000 7ff54affd700 1 --2- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7ff534038f20 0x7ff53403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T05:14:10.297 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.958+0000 7ff54affd700 1 -- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7ff534038f20 msgr2=0x7ff53403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1530849123 2024-03-28T05:14:10.297 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:04.958+0000 7ff54affd700 1 --2- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7ff534038f20 0x7ff53403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T05:14:10.297 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:05.759+0000 7ff54affd700 1 -- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7ff534038f20 msgr2=0x7ff53403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1530849123 2024-03-28T05:14:10.297 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:05.759+0000 7ff54affd700 1 --2- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7ff534038f20 0x7ff53403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T05:14:10.297 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:07.360+0000 7ff54affd700 1 -- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7ff534038f20 msgr2=0x7ff53403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1530849123 2024-03-28T05:14:10.297 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:07.360+0000 7ff54affd700 1 --2- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7ff534038f20 0x7ff53403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T05:14:10.297 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:08.643+0000 7ff548ff9700 1 -- 172.21.15.23:0/2137567858 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mgrmap(e 10) v1 ==== 45886+0+0 (secure 0 0 0) 0x7ff53c01a430 con 0x7ff54c107a20 2024-03-28T05:14:10.297 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:08.643+0000 7ff548ff9700 1 -- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7ff534038f20 msgr2=0x7ff53403b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T05:14:10.297 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:08.643+0000 7ff548ff9700 1 --2- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7ff534038f20 0x7ff53403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:10.297 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:09.646+0000 7ff548ff9700 1 -- 172.21.15.23:0/2137567858 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7ff53c04ee70 con 0x7ff54c107a20 2024-03-28T05:14:10.297 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:09.646+0000 7ff548ff9700 1 --2- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff53403c9a0 0x7ff53403ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:10.297 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:09.646+0000 7ff548ff9700 1 -- 172.21.15.23:0/2137567858 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff53c02be50 con 0x7ff53403c9a0 2024-03-28T05:14:10.297 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:09.647+0000 7ff54affd700 1 --2- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff53403c9a0 0x7ff53403ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:10.297 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:09.647+0000 7ff54affd700 1 --2- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff53403c9a0 0x7ff53403ed90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff540003a10 tx=0x7ff5400092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:10.297 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:09.648+0000 7ff548ff9700 1 -- 172.21.15.23:0/2137567858 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7ff53c02be50 con 0x7ff53403c9a0 2024-03-28T05:14:10.297 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:09.654+0000 7ff551e83700 1 -- 172.21.15.23:0/2137567858 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7ff54c069220 con 0x7ff53403c9a0 2024-03-28T05:14:10.298 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:09.655+0000 7ff548ff9700 1 -- 172.21.15.23:0/2137567858 <== mgr.14150 v2:172.21.15.23:6800/222178108 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7ff54c069220 con 0x7ff53403c9a0 2024-03-28T05:14:10.298 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:09.655+0000 7ff551e83700 1 -- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff53403c9a0 msgr2=0x7ff53403ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:10.298 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:09.655+0000 7ff551e83700 1 --2- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff53403c9a0 0x7ff53403ed90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff540003a10 tx=0x7ff5400092b0 comp rx=0 tx=0).stop 2024-03-28T05:14:10.298 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:09.655+0000 7ff551e83700 1 -- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff54c107a20 msgr2=0x7ff54c19b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:10.298 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:09.655+0000 7ff551e83700 1 --2- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff54c107a20 0x7ff54c19b960 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0117e0 tx=0x7ff53c011c10 comp rx=0 tx=0).stop 2024-03-28T05:14:10.298 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:09.655+0000 7ff551e83700 1 -- 172.21.15.23:0/2137567858 shutdown_connections 2024-03-28T05:14:10.298 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:09.655+0000 7ff551e83700 1 --2- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff53403c9a0 0x7ff53403ed90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:10.298 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:09.655+0000 7ff551e83700 1 --2- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff54c107a20 0x7ff54c19b960 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:10.298 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:09.655+0000 7ff551e83700 1 --2- 172.21.15.23:0/2137567858 >> [v2:172.21.15.23:6800/1530849123,v1:172.21.15.23:6801/1530849123] conn(0x7ff534038f20 0x7ff53403b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:10.298 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:09.655+0000 7ff551e83700 1 -- 172.21.15.23:0/2137567858 >> 172.21.15.23:0/2137567858 conn(0x7ff54c103020 msgr2=0x7ff54c103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:10.298 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:09.655+0000 7ff551e83700 1 -- 172.21.15.23:0/2137567858 shutdown_connections 2024-03-28T05:14:10.298 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:09.655+0000 7ff551e83700 1 -- 172.21.15.23:0/2137567858 wait complete. 2024-03-28T05:14:10.298 INFO:teuthology.orchestra.run.smithi023.stdout:mgr epoch 9 is available 2024-03-28T05:14:10.298 INFO:teuthology.orchestra.run.smithi023.stdout:Generating a dashboard self-signed certificate... 2024-03-28T05:14:10.882 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:10 smithi023 ceph-mon[31954]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-28T05:14:10.883 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:10 smithi023 ceph-mon[31954]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-28T05:14:10.883 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:10 smithi023 ceph-mon[31954]: [28/Mar/2024:05:14:09] ENGINE Serving on http://172.21.15.23:8765 2024-03-28T05:14:10.883 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:10 smithi023 ceph-mon[31954]: [28/Mar/2024:05:14:09] ENGINE Serving on https://172.21.15.23:7150 2024-03-28T05:14:10.883 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:10 smithi023 ceph-mon[31954]: [28/Mar/2024:05:14:09] ENGINE Bus STARTED 2024-03-28T05:14:11.507 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-28T05:14:11.507 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.666+0000 7f5ed1a90700 1 Processor -- start 2024-03-28T05:14:11.507 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.666+0000 7f5ed1a90700 1 -- start start 2024-03-28T05:14:11.507 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.667+0000 7f5ed1a90700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ecc107a20 0x7f5ecc107e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:11.507 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.667+0000 7f5ed1a90700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc108410 con 0x7f5ecc107a20 2024-03-28T05:14:11.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.667+0000 7f5ecb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ecc107a20 0x7f5ecc107e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:11.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.667+0000 7f5ecb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ecc107a20 0x7f5ecc107e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45894/0 (socket says 172.21.15.23:45894) 2024-03-28T05:14:11.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.667+0000 7f5ecb7fe700 1 -- 172.21.15.23:0/3457604280 learned_addr learned my addr 172.21.15.23:0/3457604280 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:11.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.667+0000 7f5ecb7fe700 1 -- 172.21.15.23:0/3457604280 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc108c20 con 0x7f5ecc107a20 2024-03-28T05:14:11.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.667+0000 7f5ecb7fe700 1 --2- 172.21.15.23:0/3457604280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ecc107a20 0x7f5ecc107e40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc0098a0 tx=0x7f5ebc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bff8265779758dd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:11.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.667+0000 7f5eca7fc700 1 -- 172.21.15.23:0/3457604280 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ebc004160 con 0x7f5ecc107a20 2024-03-28T05:14:11.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.667+0000 7f5eca7fc700 1 -- 172.21.15.23:0/3457604280 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ebc004300 con 0x7f5ecc107a20 2024-03-28T05:14:11.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.667+0000 7f5eca7fc700 1 -- 172.21.15.23:0/3457604280 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ebc011b20 con 0x7f5ecc107a20 2024-03-28T05:14:11.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.668+0000 7f5ed1a90700 1 -- 172.21.15.23:0/3457604280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ecc107a20 msgr2=0x7f5ecc107e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:11.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.668+0000 7f5ed1a90700 1 --2- 172.21.15.23:0/3457604280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ecc107a20 0x7f5ecc107e40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc0098a0 tx=0x7f5ebc009bb0 comp rx=0 tx=0).stop 2024-03-28T05:14:11.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.668+0000 7f5ed1a90700 1 -- 172.21.15.23:0/3457604280 shutdown_connections 2024-03-28T05:14:11.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.668+0000 7f5ed1a90700 1 --2- 172.21.15.23:0/3457604280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ecc107a20 0x7f5ecc107e40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:11.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.668+0000 7f5ed1a90700 1 -- 172.21.15.23:0/3457604280 >> 172.21.15.23:0/3457604280 conn(0x7f5ecc103040 msgr2=0x7f5ecc105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:11.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.668+0000 7f5ed1a90700 1 -- 172.21.15.23:0/3457604280 shutdown_connections 2024-03-28T05:14:11.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.668+0000 7f5ed1a90700 1 -- 172.21.15.23:0/3457604280 wait complete. 2024-03-28T05:14:11.508 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.669+0000 7f5ed1a90700 1 Processor -- start 2024-03-28T05:14:11.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.669+0000 7f5ed1a90700 1 -- start start 2024-03-28T05:14:11.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.669+0000 7f5ed1a90700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ecc107a20 0x7f5ecc19b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:11.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.669+0000 7f5ed1a90700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc10bfd0 con 0x7f5ecc107a20 2024-03-28T05:14:11.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.669+0000 7f5ecb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ecc107a20 0x7f5ecc19b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:11.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.669+0000 7f5ecb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ecc107a20 0x7f5ecc19b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45910/0 (socket says 172.21.15.23:45910) 2024-03-28T05:14:11.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.669+0000 7f5ecb7fe700 1 -- 172.21.15.23:0/4225138235 learned_addr learned my addr 172.21.15.23:0/4225138235 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:11.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.669+0000 7f5ecb7fe700 1 -- 172.21.15.23:0/4225138235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc19bed0 con 0x7f5ecc107a20 2024-03-28T05:14:11.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.670+0000 7f5ecb7fe700 1 --2- 172.21.15.23:0/4225138235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ecc107a20 0x7f5ecc19b990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc000c00 tx=0x7f5ebc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:11.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.670+0000 7f5ec8ff9700 1 -- 172.21.15.23:0/4225138235 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ebc003b60 con 0x7f5ecc107a20 2024-03-28T05:14:11.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.670+0000 7f5ed1a90700 1 -- 172.21.15.23:0/4225138235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ecc19c160 con 0x7f5ecc107a20 2024-03-28T05:14:11.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.670+0000 7f5ec8ff9700 1 -- 172.21.15.23:0/4225138235 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ebc01a910 con 0x7f5ecc107a20 2024-03-28T05:14:11.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.670+0000 7f5ec8ff9700 1 -- 172.21.15.23:0/4225138235 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ebc024930 con 0x7f5ecc107a20 2024-03-28T05:14:11.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.670+0000 7f5ed1a90700 1 -- 172.21.15.23:0/4225138235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ecc19c5c0 con 0x7f5ecc107a20 2024-03-28T05:14:11.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.671+0000 7f5ec8ff9700 1 -- 172.21.15.23:0/4225138235 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7f5ebc022020 con 0x7f5ecc107a20 2024-03-28T05:14:11.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.671+0000 7f5ed1a90700 1 -- 172.21.15.23:0/4225138235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ecc069220 con 0x7f5ecc107a20 2024-03-28T05:14:11.509 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.671+0000 7f5ec8ff9700 1 --2- 172.21.15.23:0/4225138235 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5eb8038e50 0x7f5eb803b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:11.510 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.671+0000 7f5ec8ff9700 1 -- 172.21.15.23:0/4225138235 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5ebc04cfd0 con 0x7f5ecc107a20 2024-03-28T05:14:11.510 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.671+0000 7f5ecaffd700 1 --2- 172.21.15.23:0/4225138235 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5eb8038e50 0x7f5eb803b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:11.510 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.671+0000 7f5ecaffd700 1 --2- 172.21.15.23:0/4225138235 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5eb8038e50 0x7f5eb803b310 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4006fd0 tx=0x7f5eb4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:11.510 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.676+0000 7f5ec8ff9700 1 -- 172.21.15.23:0/4225138235 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5ebc01ab70 con 0x7f5ecc107a20 2024-03-28T05:14:11.510 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.694+0000 7f5ec8ff9700 1 -- 172.21.15.23:0/4225138235 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f5ebc02d3f0 con 0x7f5ecc107a20 2024-03-28T05:14:11.510 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:10.885+0000 7f5ed1a90700 1 -- 172.21.15.23:0/4225138235 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ecc104280 con 0x7f5eb8038e50 2024-03-28T05:14:11.510 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.007+0000 7f5ec8ff9700 1 -- 172.21.15.23:0/4225138235 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f5ecc104280 con 0x7f5eb8038e50 2024-03-28T05:14:11.510 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.009+0000 7f5ed1a90700 1 -- 172.21.15.23:0/4225138235 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5eb8038e50 msgr2=0x7f5eb803b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:11.510 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.009+0000 7f5ed1a90700 1 --2- 172.21.15.23:0/4225138235 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5eb8038e50 0x7f5eb803b310 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4006fd0 tx=0x7f5eb4006e40 comp rx=0 tx=0).stop 2024-03-28T05:14:11.510 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.009+0000 7f5ed1a90700 1 -- 172.21.15.23:0/4225138235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ecc107a20 msgr2=0x7f5ecc19b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:11.510 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.009+0000 7f5ed1a90700 1 --2- 172.21.15.23:0/4225138235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ecc107a20 0x7f5ecc19b990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc000c00 tx=0x7f5ebc011c10 comp rx=0 tx=0).stop 2024-03-28T05:14:11.510 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.009+0000 7f5ed1a90700 1 -- 172.21.15.23:0/4225138235 shutdown_connections 2024-03-28T05:14:11.510 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.009+0000 7f5ed1a90700 1 --2- 172.21.15.23:0/4225138235 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5eb8038e50 0x7f5eb803b310 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:11.510 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.009+0000 7f5ed1a90700 1 --2- 172.21.15.23:0/4225138235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ecc107a20 0x7f5ecc19b990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:11.510 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.009+0000 7f5ed1a90700 1 -- 172.21.15.23:0/4225138235 >> 172.21.15.23:0/4225138235 conn(0x7f5ecc103040 msgr2=0x7f5ecc103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:11.510 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.009+0000 7f5ed1a90700 1 -- 172.21.15.23:0/4225138235 shutdown_connections 2024-03-28T05:14:11.511 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.009+0000 7f5ed1a90700 1 -- 172.21.15.23:0/4225138235 wait complete. 2024-03-28T05:14:11.511 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial admin user... 2024-03-28T05:14:11.894 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:11 smithi023 ceph-mon[31954]: mgrmap e12: x(active, since 2s) 2024-03-28T05:14:11.895 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:11 smithi023 ceph-mon[31954]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:14:11.895 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:11 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:11.895 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:11 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:11.895 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:11 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:12.964 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$QcGh7e2NdSmJFZhJiUrzVe9OXbRyfpRzhHwn27Fr0BKDICU16xngO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711602852, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-28T05:14:12.964 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.928+0000 7f919323c700 1 Processor -- start 2024-03-28T05:14:12.964 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.928+0000 7f919323c700 1 -- start start 2024-03-28T05:14:12.964 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.928+0000 7f919323c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f918c106900 0x7f918c106d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:12.964 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.928+0000 7f919323c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c1072f0 con 0x7f918c106900 2024-03-28T05:14:12.964 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.928+0000 7f9190fd8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f918c106900 0x7f918c106d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:12.964 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.928+0000 7f9190fd8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f918c106900 0x7f918c106d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45916/0 (socket says 172.21.15.23:45916) 2024-03-28T05:14:12.964 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.928+0000 7f9190fd8700 1 -- 172.21.15.23:0/2729810324 learned_addr learned my addr 172.21.15.23:0/2729810324 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:12.964 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.928+0000 7f9190fd8700 1 -- 172.21.15.23:0/2729810324 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918c107b00 con 0x7f918c106900 2024-03-28T05:14:12.964 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.928+0000 7f9190fd8700 1 --2- 172.21.15.23:0/2729810324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f918c106900 0x7f918c106d20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f91800098a0 tx=0x7f9180009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbea2b2cc5cba204 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:12.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.929+0000 7f918b7fe700 1 -- 172.21.15.23:0/2729810324 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9180004160 con 0x7f918c106900 2024-03-28T05:14:12.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.929+0000 7f918b7fe700 1 -- 172.21.15.23:0/2729810324 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9180004300 con 0x7f918c106900 2024-03-28T05:14:12.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.929+0000 7f918b7fe700 1 -- 172.21.15.23:0/2729810324 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9180011b20 con 0x7f918c106900 2024-03-28T05:14:12.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.929+0000 7f919323c700 1 -- 172.21.15.23:0/2729810324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f918c106900 msgr2=0x7f918c106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:12.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.929+0000 7f919323c700 1 --2- 172.21.15.23:0/2729810324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f918c106900 0x7f918c106d20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f91800098a0 tx=0x7f9180009bb0 comp rx=0 tx=0).stop 2024-03-28T05:14:12.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.929+0000 7f919323c700 1 -- 172.21.15.23:0/2729810324 shutdown_connections 2024-03-28T05:14:12.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.929+0000 7f919323c700 1 --2- 172.21.15.23:0/2729810324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f918c106900 0x7f918c106d20 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:12.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.929+0000 7f919323c700 1 -- 172.21.15.23:0/2729810324 >> 172.21.15.23:0/2729810324 conn(0x7f918c101fa0 msgr2=0x7f918c1043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:12.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.929+0000 7f919323c700 1 -- 172.21.15.23:0/2729810324 shutdown_connections 2024-03-28T05:14:12.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.929+0000 7f919323c700 1 -- 172.21.15.23:0/2729810324 wait complete. 2024-03-28T05:14:12.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.930+0000 7f919323c700 1 Processor -- start 2024-03-28T05:14:12.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.930+0000 7f919323c700 1 -- start start 2024-03-28T05:14:12.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.930+0000 7f919323c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f918c106900 0x7f918c191ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:12.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.930+0000 7f919323c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c10aeb0 con 0x7f918c106900 2024-03-28T05:14:12.965 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.930+0000 7f9190fd8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f918c106900 0x7f918c191ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:12.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.930+0000 7f9190fd8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f918c106900 0x7f918c191ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45922/0 (socket says 172.21.15.23:45922) 2024-03-28T05:14:12.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.930+0000 7f9190fd8700 1 -- 172.21.15.23:0/4104193674 learned_addr learned my addr 172.21.15.23:0/4104193674 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:12.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.930+0000 7f9190fd8700 1 -- 172.21.15.23:0/4104193674 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918c191ff0 con 0x7f918c106900 2024-03-28T05:14:12.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.931+0000 7f9190fd8700 1 --2- 172.21.15.23:0/4104193674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f918c106900 0x7f918c191ab0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f91800116a0 tx=0x7f9180011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:12.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.931+0000 7f9189ffb700 1 -- 172.21.15.23:0/4104193674 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9180003a40 con 0x7f918c106900 2024-03-28T05:14:12.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.931+0000 7f919323c700 1 -- 172.21.15.23:0/4104193674 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f918c1901e0 con 0x7f918c106900 2024-03-28T05:14:12.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.931+0000 7f9189ffb700 1 -- 172.21.15.23:0/4104193674 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9180003be0 con 0x7f918c106900 2024-03-28T05:14:12.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.931+0000 7f9189ffb700 1 -- 172.21.15.23:0/4104193674 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f918001a720 con 0x7f918c106900 2024-03-28T05:14:12.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.931+0000 7f919323c700 1 -- 172.21.15.23:0/4104193674 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f918c1906a0 con 0x7f918c106900 2024-03-28T05:14:12.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.932+0000 7f9189ffb700 1 -- 172.21.15.23:0/4104193674 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f9180024430 con 0x7f918c106900 2024-03-28T05:14:12.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.932+0000 7f919323c700 1 -- 172.21.15.23:0/4104193674 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f918c069220 con 0x7f918c106900 2024-03-28T05:14:12.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.932+0000 7f9189ffb700 1 --2- 172.21.15.23:0/4104193674 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f917c038f20 0x7f917c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:12.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.932+0000 7f9189ffb700 1 -- 172.21.15.23:0/4104193674 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f918004d5b0 con 0x7f918c106900 2024-03-28T05:14:12.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.932+0000 7f918bfff700 1 --2- 172.21.15.23:0/4104193674 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f917c038f20 0x7f917c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:12.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.933+0000 7f918bfff700 1 --2- 172.21.15.23:0/4104193674 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f917c038f20 0x7f917c03b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9178006fd0 tx=0x7f9178006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:12.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:11.938+0000 7f9189ffb700 1 -- 172.21.15.23:0/4104193674 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9180011e20 con 0x7f918c106900 2024-03-28T05:14:12.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:12.166+0000 7f919323c700 1 -- 172.21.15.23:0/4104193674 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f918c195800 con 0x7f917c038f20 2024-03-28T05:14:12.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:12.413+0000 7f9189ffb700 1 -- 172.21.15.23:0/4104193674 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f918c195800 con 0x7f917c038f20 2024-03-28T05:14:12.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:12.416+0000 7f919323c700 1 -- 172.21.15.23:0/4104193674 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f917c038f20 msgr2=0x7f917c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:12.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:12.416+0000 7f919323c700 1 --2- 172.21.15.23:0/4104193674 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f917c038f20 0x7f917c03b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9178006fd0 tx=0x7f9178006e40 comp rx=0 tx=0).stop 2024-03-28T05:14:12.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:12.416+0000 7f919323c700 1 -- 172.21.15.23:0/4104193674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f918c106900 msgr2=0x7f918c191ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:12.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:12.416+0000 7f919323c700 1 --2- 172.21.15.23:0/4104193674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f918c106900 0x7f918c191ab0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f91800116a0 tx=0x7f9180011ba0 comp rx=0 tx=0).stop 2024-03-28T05:14:12.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:12.416+0000 7f919323c700 1 -- 172.21.15.23:0/4104193674 shutdown_connections 2024-03-28T05:14:12.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:12.416+0000 7f919323c700 1 --2- 172.21.15.23:0/4104193674 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f917c038f20 0x7f917c03b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:12.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:12.416+0000 7f919323c700 1 --2- 172.21.15.23:0/4104193674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f918c106900 0x7f918c191ab0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:12.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:12.416+0000 7f919323c700 1 -- 172.21.15.23:0/4104193674 >> 172.21.15.23:0/4104193674 conn(0x7f918c101fa0 msgr2=0x7f918c102e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:12.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:12.416+0000 7f919323c700 1 -- 172.21.15.23:0/4104193674 shutdown_connections 2024-03-28T05:14:12.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:12.416+0000 7f919323c700 1 -- 172.21.15.23:0/4104193674 wait complete. 2024-03-28T05:14:12.968 INFO:teuthology.orchestra.run.smithi023.stdout:Fetching dashboard port number... 2024-03-28T05:14:13.586 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:13 smithi023 ceph-mon[31954]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:14:13.586 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:13 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:13.586 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:13 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:14.162 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 8443 2024-03-28T05:14:14.162 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.359+0000 7f1a7845b700 1 Processor -- start 2024-03-28T05:14:14.162 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.360+0000 7f1a7845b700 1 -- start start 2024-03-28T05:14:14.162 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.360+0000 7f1a7845b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a70108460 0x7f1a7010a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:14.162 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.360+0000 7f1a7845b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7007aeb0 con 0x7f1a70108460 2024-03-28T05:14:14.163 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.360+0000 7f1a761f7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a70108460 0x7f1a7010a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:14.163 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.360+0000 7f1a761f7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a70108460 0x7f1a7010a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45230/0 (socket says 172.21.15.23:45230) 2024-03-28T05:14:14.163 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.360+0000 7f1a761f7700 1 -- 172.21.15.23:0/3148331197 learned_addr learned my addr 172.21.15.23:0/3148331197 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:14.163 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.360+0000 7f1a761f7700 1 -- 172.21.15.23:0/3148331197 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7010ad90 con 0x7f1a70108460 2024-03-28T05:14:14.163 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.361+0000 7f1a761f7700 1 --2- 172.21.15.23:0/3148331197 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a70108460 0x7f1a7010a850 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c0098a0 tx=0x7f1a6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=149442db5111fcb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:14.163 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.361+0000 7f1a751f5700 1 -- 172.21.15.23:0/3148331197 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a6c004160 con 0x7f1a70108460 2024-03-28T05:14:14.163 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.361+0000 7f1a751f5700 1 -- 172.21.15.23:0/3148331197 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a6c004300 con 0x7f1a70108460 2024-03-28T05:14:14.163 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.361+0000 7f1a751f5700 1 -- 172.21.15.23:0/3148331197 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a6c011b20 con 0x7f1a70108460 2024-03-28T05:14:14.163 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.361+0000 7f1a7845b700 1 -- 172.21.15.23:0/3148331197 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a70108460 msgr2=0x7f1a7010a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:14.163 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.361+0000 7f1a7845b700 1 --2- 172.21.15.23:0/3148331197 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a70108460 0x7f1a7010a850 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c0098a0 tx=0x7f1a6c009bb0 comp rx=0 tx=0).stop 2024-03-28T05:14:14.163 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.362+0000 7f1a7845b700 1 -- 172.21.15.23:0/3148331197 shutdown_connections 2024-03-28T05:14:14.163 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.362+0000 7f1a7845b700 1 --2- 172.21.15.23:0/3148331197 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a70108460 0x7f1a7010a850 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:14.163 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.362+0000 7f1a7845b700 1 -- 172.21.15.23:0/3148331197 >> 172.21.15.23:0/3148331197 conn(0x7f1a70101f10 msgr2=0x7f1a70104330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:14.163 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.362+0000 7f1a7845b700 1 -- 172.21.15.23:0/3148331197 shutdown_connections 2024-03-28T05:14:14.163 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.362+0000 7f1a7845b700 1 -- 172.21.15.23:0/3148331197 wait complete. 2024-03-28T05:14:14.163 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.362+0000 7f1a7845b700 1 Processor -- start 2024-03-28T05:14:14.163 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.362+0000 7f1a7845b700 1 -- start start 2024-03-28T05:14:14.164 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.362+0000 7f1a7845b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a70108460 0x7f1a70107ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:14.164 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.362+0000 7f1a7845b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7010bd90 con 0x7f1a70108460 2024-03-28T05:14:14.164 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.363+0000 7f1a761f7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a70108460 0x7f1a70107ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:14.164 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.363+0000 7f1a761f7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a70108460 0x7f1a70107ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45246/0 (socket says 172.21.15.23:45246) 2024-03-28T05:14:14.164 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.363+0000 7f1a761f7700 1 -- 172.21.15.23:0/2998261859 learned_addr learned my addr 172.21.15.23:0/2998261859 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:14.164 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.363+0000 7f1a761f7700 1 -- 172.21.15.23:0/2998261859 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a70108220 con 0x7f1a70108460 2024-03-28T05:14:14.164 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.363+0000 7f1a761f7700 1 --2- 172.21.15.23:0/2998261859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a70108460 0x7f1a70107ce0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c0116a0 tx=0x7f1a6c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:14.164 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.363+0000 7f1a677fe700 1 -- 172.21.15.23:0/2998261859 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a6c003a40 con 0x7f1a70108460 2024-03-28T05:14:14.164 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.363+0000 7f1a677fe700 1 -- 172.21.15.23:0/2998261859 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a6c003be0 con 0x7f1a70108460 2024-03-28T05:14:14.164 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.363+0000 7f1a7845b700 1 -- 172.21.15.23:0/2998261859 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a70106410 con 0x7f1a70108460 2024-03-28T05:14:14.164 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.363+0000 7f1a677fe700 1 -- 172.21.15.23:0/2998261859 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a6c01a750 con 0x7f1a70108460 2024-03-28T05:14:14.164 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.363+0000 7f1a7845b700 1 -- 172.21.15.23:0/2998261859 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a701068d0 con 0x7f1a70108460 2024-03-28T05:14:14.164 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.364+0000 7f1a677fe700 1 -- 172.21.15.23:0/2998261859 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f1a6c01aa00 con 0x7f1a70108460 2024-03-28T05:14:14.164 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.364+0000 7f1a677fe700 1 --2- 172.21.15.23:0/2998261859 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1a5c038f80 0x7f1a5c03b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:14.164 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.364+0000 7f1a677fe700 1 -- 172.21.15.23:0/2998261859 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1a6c04d4d0 con 0x7f1a70108460 2024-03-28T05:14:14.164 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.364+0000 7f1a759f6700 1 --2- 172.21.15.23:0/2998261859 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1a5c038f80 0x7f1a5c03b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:14.165 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.364+0000 7f1a7845b700 1 -- 172.21.15.23:0/2998261859 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a700637c0 con 0x7f1a70108460 2024-03-28T05:14:14.165 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.365+0000 7f1a759f6700 1 --2- 172.21.15.23:0/2998261859 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1a5c038f80 0x7f1a5c03b440 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1a60006fd0 tx=0x7f1a60006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:14.165 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.370+0000 7f1a677fe700 1 -- 172.21.15.23:0/2998261859 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1a6c011e20 con 0x7f1a70108460 2024-03-28T05:14:14.165 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.575+0000 7f1a7845b700 1 -- 172.21.15.23:0/2998261859 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f1a70106f50 con 0x7f1a70108460 2024-03-28T05:14:14.165 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.576+0000 7f1a677fe700 1 -- 172.21.15.23:0/2998261859 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f1a6c01ad40 con 0x7f1a70108460 2024-03-28T05:14:14.165 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.578+0000 7f1a7845b700 1 -- 172.21.15.23:0/2998261859 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1a5c038f80 msgr2=0x7f1a5c03b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:14.165 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.578+0000 7f1a7845b700 1 --2- 172.21.15.23:0/2998261859 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1a5c038f80 0x7f1a5c03b440 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1a60006fd0 tx=0x7f1a60006e40 comp rx=0 tx=0).stop 2024-03-28T05:14:14.165 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.578+0000 7f1a7845b700 1 -- 172.21.15.23:0/2998261859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a70108460 msgr2=0x7f1a70107ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:14.165 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.578+0000 7f1a7845b700 1 --2- 172.21.15.23:0/2998261859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a70108460 0x7f1a70107ce0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c0116a0 tx=0x7f1a6c011ba0 comp rx=0 tx=0).stop 2024-03-28T05:14:14.165 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.578+0000 7f1a7845b700 1 -- 172.21.15.23:0/2998261859 shutdown_connections 2024-03-28T05:14:14.165 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.578+0000 7f1a7845b700 1 --2- 172.21.15.23:0/2998261859 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1a5c038f80 0x7f1a5c03b440 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:14.165 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.578+0000 7f1a7845b700 1 --2- 172.21.15.23:0/2998261859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a70108460 0x7f1a70107ce0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:14.165 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.578+0000 7f1a7845b700 1 -- 172.21.15.23:0/2998261859 >> 172.21.15.23:0/2998261859 conn(0x7f1a70101f10 msgr2=0x7f1a70102a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:14.165 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.578+0000 7f1a7845b700 1 -- 172.21.15.23:0/2998261859 shutdown_connections 2024-03-28T05:14:14.165 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:13.578+0000 7f1a7845b700 1 -- 172.21.15.23:0/2998261859 wait complete. 2024-03-28T05:14:14.168 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T05:14:14.168 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-28T05:14:14.175 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T05:14:14.175 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-28T05:14:14.175 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-28T05:14:14.175 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-28T05:14:14.177 INFO:teuthology.orchestra.run.smithi023.stdout:Ceph Dashboard is now available at: 2024-03-28T05:14:14.177 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:14:14.177 INFO:teuthology.orchestra.run.smithi023.stdout: URL: https://smithi023.front.sepia.ceph.com:8443/ 2024-03-28T05:14:14.177 INFO:teuthology.orchestra.run.smithi023.stdout: User: admin 2024-03-28T05:14:14.177 INFO:teuthology.orchestra.run.smithi023.stdout: Password: ytvejz2f48 2024-03-28T05:14:14.177 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:14:14.177 INFO:teuthology.orchestra.run.smithi023.stdout:Saving cluster configuration to /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config directory 2024-03-28T05:14:14.177 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling autotune for osd_memory_target 2024-03-28T05:14:14.726 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:14 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2998261859' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-28T05:14:15.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.604+0000 7f7c2f361700 1 Processor -- start 2024-03-28T05:14:15.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.605+0000 7f7c2f361700 1 -- start start 2024-03-28T05:14:15.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.605+0000 7f7c2f361700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c28108490 0x7f7c2810a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:15.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.605+0000 7f7c2f361700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2807ae70 con 0x7f7c28108490 2024-03-28T05:14:15.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.605+0000 7f7c2d0fd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c28108490 0x7f7c2810a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:15.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.605+0000 7f7c2d0fd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c28108490 0x7f7c2810a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45248/0 (socket says 172.21.15.23:45248) 2024-03-28T05:14:15.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.605+0000 7f7c2d0fd700 1 -- 172.21.15.23:0/3553939179 learned_addr learned my addr 172.21.15.23:0/3553939179 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:15.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.605+0000 7f7c2d0fd700 1 -- 172.21.15.23:0/3553939179 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2810adc0 con 0x7f7c28108490 2024-03-28T05:14:15.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.606+0000 7f7c2d0fd700 1 --2- 172.21.15.23:0/3553939179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c28108490 0x7f7c2810a880 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7c240098a0 tx=0x7f7c24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13a635513b5473de server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:15.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.606+0000 7f7c1ffff700 1 -- 172.21.15.23:0/3553939179 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c24004160 con 0x7f7c28108490 2024-03-28T05:14:15.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.606+0000 7f7c1ffff700 1 -- 172.21.15.23:0/3553939179 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c24004300 con 0x7f7c28108490 2024-03-28T05:14:15.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.606+0000 7f7c1ffff700 1 -- 172.21.15.23:0/3553939179 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c24011b20 con 0x7f7c28108490 2024-03-28T05:14:15.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.606+0000 7f7c2f361700 1 -- 172.21.15.23:0/3553939179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c28108490 msgr2=0x7f7c2810a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:15.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.606+0000 7f7c2f361700 1 --2- 172.21.15.23:0/3553939179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c28108490 0x7f7c2810a880 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7c240098a0 tx=0x7f7c24009bb0 comp rx=0 tx=0).stop 2024-03-28T05:14:15.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.606+0000 7f7c2f361700 1 -- 172.21.15.23:0/3553939179 shutdown_connections 2024-03-28T05:14:15.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.606+0000 7f7c2f361700 1 --2- 172.21.15.23:0/3553939179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c28108490 0x7f7c2810a880 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:15.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.606+0000 7f7c2f361700 1 -- 172.21.15.23:0/3553939179 >> 172.21.15.23:0/3553939179 conn(0x7f7c28101f20 msgr2=0x7f7c28104340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:15.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.606+0000 7f7c2f361700 1 -- 172.21.15.23:0/3553939179 shutdown_connections 2024-03-28T05:14:15.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.606+0000 7f7c2f361700 1 -- 172.21.15.23:0/3553939179 wait complete. 2024-03-28T05:14:15.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.607+0000 7f7c2f361700 1 Processor -- start 2024-03-28T05:14:15.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.607+0000 7f7c2f361700 1 -- start start 2024-03-28T05:14:15.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.607+0000 7f7c2f361700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c28108490 0x7f7c2819b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:15.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.607+0000 7f7c2f361700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2810bdd0 con 0x7f7c28108490 2024-03-28T05:14:15.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.607+0000 7f7c2d0fd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c28108490 0x7f7c2819b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:15.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.607+0000 7f7c2d0fd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c28108490 0x7f7c2819b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45254/0 (socket says 172.21.15.23:45254) 2024-03-28T05:14:15.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.607+0000 7f7c2d0fd700 1 -- 172.21.15.23:0/2931324393 learned_addr learned my addr 172.21.15.23:0/2931324393 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:15.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.608+0000 7f7c2d0fd700 1 -- 172.21.15.23:0/2931324393 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2819bea0 con 0x7f7c28108490 2024-03-28T05:14:15.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.608+0000 7f7c2d0fd700 1 --2- 172.21.15.23:0/2931324393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c28108490 0x7f7c2819b960 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7c240116a0 tx=0x7f7c24011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:15.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.608+0000 7f7c1e7fc700 1 -- 172.21.15.23:0/2931324393 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c24003a40 con 0x7f7c28108490 2024-03-28T05:14:15.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.608+0000 7f7c1e7fc700 1 -- 172.21.15.23:0/2931324393 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c24003be0 con 0x7f7c28108490 2024-03-28T05:14:15.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.608+0000 7f7c2f361700 1 -- 172.21.15.23:0/2931324393 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c2819c130 con 0x7f7c28108490 2024-03-28T05:14:15.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.608+0000 7f7c1e7fc700 1 -- 172.21.15.23:0/2931324393 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c2401a750 con 0x7f7c28108490 2024-03-28T05:14:15.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.608+0000 7f7c2f361700 1 -- 172.21.15.23:0/2931324393 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c2819c590 con 0x7f7c28108490 2024-03-28T05:14:15.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.609+0000 7f7c1e7fc700 1 -- 172.21.15.23:0/2931324393 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f7c24024aa0 con 0x7f7c28108490 2024-03-28T05:14:15.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.609+0000 7f7c1e7fc700 1 --2- 172.21.15.23:0/2931324393 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f7c14038f20 0x7f7c1403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:15.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.609+0000 7f7c2f361700 1 -- 172.21.15.23:0/2931324393 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c280637c0 con 0x7f7c28108490 2024-03-28T05:14:15.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.609+0000 7f7c1e7fc700 1 -- 172.21.15.23:0/2931324393 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7c2404d650 con 0x7f7c28108490 2024-03-28T05:14:15.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.609+0000 7f7c2c8fc700 1 --2- 172.21.15.23:0/2931324393 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f7c14038f20 0x7f7c1403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:15.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.610+0000 7f7c2c8fc700 1 --2- 172.21.15.23:0/2931324393 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f7c14038f20 0x7f7c1403b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7c18006fd0 tx=0x7f7c18006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:15.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.614+0000 7f7c1e7fc700 1 -- 172.21.15.23:0/2931324393 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7c24011e20 con 0x7f7c28108490 2024-03-28T05:14:15.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.826+0000 7f7c2f361700 1 -- 172.21.15.23:0/2931324393 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f7c2819f020 con 0x7f7c28108490 2024-03-28T05:14:15.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.826+0000 7f7c1e7fc700 1 -- 172.21.15.23:0/2931324393 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f7c2402d770 con 0x7f7c28108490 2024-03-28T05:14:15.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.829+0000 7f7c2f361700 1 -- 172.21.15.23:0/2931324393 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f7c14038f20 msgr2=0x7f7c1403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:15.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.829+0000 7f7c2f361700 1 --2- 172.21.15.23:0/2931324393 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f7c14038f20 0x7f7c1403b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7c18006fd0 tx=0x7f7c18006e40 comp rx=0 tx=0).stop 2024-03-28T05:14:15.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.829+0000 7f7c2f361700 1 -- 172.21.15.23:0/2931324393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c28108490 msgr2=0x7f7c2819b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:15.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.829+0000 7f7c2f361700 1 --2- 172.21.15.23:0/2931324393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c28108490 0x7f7c2819b960 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7c240116a0 tx=0x7f7c24011ba0 comp rx=0 tx=0).stop 2024-03-28T05:14:15.449 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.829+0000 7f7c2f361700 1 -- 172.21.15.23:0/2931324393 shutdown_connections 2024-03-28T05:14:15.449 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.829+0000 7f7c2f361700 1 --2- 172.21.15.23:0/2931324393 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f7c14038f20 0x7f7c1403b3e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:15.449 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.829+0000 7f7c2f361700 1 --2- 172.21.15.23:0/2931324393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c28108490 0x7f7c2819b960 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:15.449 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.829+0000 7f7c2f361700 1 -- 172.21.15.23:0/2931324393 >> 172.21.15.23:0/2931324393 conn(0x7f7c28101f20 msgr2=0x7f7c28102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:15.449 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.829+0000 7f7c2f361700 1 -- 172.21.15.23:0/2931324393 shutdown_connections 2024-03-28T05:14:15.449 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:14.829+0000 7f7c2f361700 1 -- 172.21.15.23:0/2931324393 wait complete. 2024-03-28T05:14:16.496 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:16 smithi023 ceph-mon[31954]: mgrmap e13: x(active, since 6s) 2024-03-28T05:14:16.496 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:16 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3803534983' entity='client.admin' 2024-03-28T05:14:16.773 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.854+0000 7ffbb4dce700 1 Processor -- start 2024-03-28T05:14:16.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.854+0000 7ffbb4dce700 1 -- start start 2024-03-28T05:14:16.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.854+0000 7ffbb4dce700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbb0106910 0x7ffbb0106d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:16.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.854+0000 7ffbb4dce700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb0107300 con 0x7ffbb0106910 2024-03-28T05:14:16.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.854+0000 7ffbae59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbb0106910 0x7ffbb0106d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:16.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.855+0000 7ffbae59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbb0106910 0x7ffbb0106d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45258/0 (socket says 172.21.15.23:45258) 2024-03-28T05:14:16.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.855+0000 7ffbae59c700 1 -- 172.21.15.23:0/6732179 learned_addr learned my addr 172.21.15.23:0/6732179 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:16.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.855+0000 7ffbae59c700 1 -- 172.21.15.23:0/6732179 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb0107b10 con 0x7ffbb0106910 2024-03-28T05:14:16.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.855+0000 7ffbae59c700 1 --2- 172.21.15.23:0/6732179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbb0106910 0x7ffbb0106d30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ffba00098a0 tx=0x7ffba0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0e4d2f99a1d5ad9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:16.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.855+0000 7ffbadd9b700 1 -- 172.21.15.23:0/6732179 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffba0004160 con 0x7ffbb0106910 2024-03-28T05:14:16.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.855+0000 7ffbadd9b700 1 -- 172.21.15.23:0/6732179 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffba0004300 con 0x7ffbb0106910 2024-03-28T05:14:16.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.855+0000 7ffbadd9b700 1 -- 172.21.15.23:0/6732179 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffba0011b20 con 0x7ffbb0106910 2024-03-28T05:14:16.774 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.856+0000 7ffbb4dce700 1 -- 172.21.15.23:0/6732179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbb0106910 msgr2=0x7ffbb0106d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:16.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.856+0000 7ffbb4dce700 1 --2- 172.21.15.23:0/6732179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbb0106910 0x7ffbb0106d30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ffba00098a0 tx=0x7ffba0009bb0 comp rx=0 tx=0).stop 2024-03-28T05:14:16.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.856+0000 7ffbb4dce700 1 -- 172.21.15.23:0/6732179 shutdown_connections 2024-03-28T05:14:16.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.856+0000 7ffbb4dce700 1 --2- 172.21.15.23:0/6732179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbb0106910 0x7ffbb0106d30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:16.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.856+0000 7ffbb4dce700 1 -- 172.21.15.23:0/6732179 >> 172.21.15.23:0/6732179 conn(0x7ffbb0101f70 msgr2=0x7ffbb0104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:16.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.856+0000 7ffbb4dce700 1 -- 172.21.15.23:0/6732179 shutdown_connections 2024-03-28T05:14:16.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.856+0000 7ffbb4dce700 1 -- 172.21.15.23:0/6732179 wait complete. 2024-03-28T05:14:16.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.857+0000 7ffbb4dce700 1 Processor -- start 2024-03-28T05:14:16.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.857+0000 7ffbb4dce700 1 -- start start 2024-03-28T05:14:16.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.857+0000 7ffbb4dce700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbb0106910 0x7ffbb019b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:16.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.857+0000 7ffbb4dce700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb010aec0 con 0x7ffbb0106910 2024-03-28T05:14:16.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.857+0000 7ffbae59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbb0106910 0x7ffbb019b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:16.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.857+0000 7ffbae59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbb0106910 0x7ffbb019b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45270/0 (socket says 172.21.15.23:45270) 2024-03-28T05:14:16.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.857+0000 7ffbae59c700 1 -- 172.21.15.23:0/3803534983 learned_addr learned my addr 172.21.15.23:0/3803534983 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:16.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.857+0000 7ffbae59c700 1 -- 172.21.15.23:0/3803534983 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb019bf10 con 0x7ffbb0106910 2024-03-28T05:14:16.775 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.857+0000 7ffbae59c700 1 --2- 172.21.15.23:0/3803534983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbb0106910 0x7ffbb019b9d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ffba0000c00 tx=0x7ffba0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:16.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.857+0000 7ffba77fe700 1 -- 172.21.15.23:0/3803534983 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffba0003b60 con 0x7ffbb0106910 2024-03-28T05:14:16.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.857+0000 7ffba77fe700 1 -- 172.21.15.23:0/3803534983 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffba001a910 con 0x7ffbb0106910 2024-03-28T05:14:16.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.857+0000 7ffbb4dce700 1 -- 172.21.15.23:0/3803534983 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb019c1a0 con 0x7ffbb0106910 2024-03-28T05:14:16.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.857+0000 7ffba77fe700 1 -- 172.21.15.23:0/3803534983 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffba0024930 con 0x7ffbb0106910 2024-03-28T05:14:16.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.857+0000 7ffbb4dce700 1 -- 172.21.15.23:0/3803534983 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb019c600 con 0x7ffbb0106910 2024-03-28T05:14:16.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.858+0000 7ffba77fe700 1 -- 172.21.15.23:0/3803534983 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7ffba0022020 con 0x7ffbb0106910 2024-03-28T05:14:16.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.859+0000 7ffbb4dce700 1 -- 172.21.15.23:0/3803534983 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb00637c0 con 0x7ffbb0106910 2024-03-28T05:14:16.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.859+0000 7ffba77fe700 1 --2- 172.21.15.23:0/3803534983 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ffb9c039010 0x7ffb9c03b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:16.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.859+0000 7ffba77fe700 1 -- 172.21.15.23:0/3803534983 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ffba004d440 con 0x7ffbb0106910 2024-03-28T05:14:16.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.859+0000 7ffba7fff700 1 --2- 172.21.15.23:0/3803534983 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ffb9c039010 0x7ffb9c03b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:16.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.859+0000 7ffba7fff700 1 --2- 172.21.15.23:0/3803534983 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ffb9c039010 0x7ffb9c03b4d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ffb98006fd0 tx=0x7ffb98006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:16.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:15.864+0000 7ffba77fe700 1 -- 172.21.15.23:0/3803534983 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffba000b3e0 con 0x7ffbb0106910 2024-03-28T05:14:16.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:16.185+0000 7ffbb4dce700 1 -- 172.21.15.23:0/3803534983 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7ffbb0062da0 con 0x7ffbb0106910 2024-03-28T05:14:16.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:16.190+0000 7ffba77fe700 1 -- 172.21.15.23:0/3803534983 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7ffba001ab30 con 0x7ffbb0106910 2024-03-28T05:14:16.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:16.192+0000 7ffbb4dce700 1 -- 172.21.15.23:0/3803534983 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ffb9c039010 msgr2=0x7ffb9c03b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:16.776 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:16.192+0000 7ffbb4dce700 1 --2- 172.21.15.23:0/3803534983 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ffb9c039010 0x7ffb9c03b4d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ffb98006fd0 tx=0x7ffb98006e40 comp rx=0 tx=0).stop 2024-03-28T05:14:16.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:16.192+0000 7ffbb4dce700 1 -- 172.21.15.23:0/3803534983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbb0106910 msgr2=0x7ffbb019b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:16.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:16.192+0000 7ffbb4dce700 1 --2- 172.21.15.23:0/3803534983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbb0106910 0x7ffbb019b9d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ffba0000c00 tx=0x7ffba0011c10 comp rx=0 tx=0).stop 2024-03-28T05:14:16.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:16.192+0000 7ffbb4dce700 1 -- 172.21.15.23:0/3803534983 shutdown_connections 2024-03-28T05:14:16.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:16.192+0000 7ffbb4dce700 1 --2- 172.21.15.23:0/3803534983 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ffb9c039010 0x7ffb9c03b4d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:16.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:16.192+0000 7ffbb4dce700 1 --2- 172.21.15.23:0/3803534983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbb0106910 0x7ffbb019b9d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:16.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:16.192+0000 7ffbb4dce700 1 -- 172.21.15.23:0/3803534983 >> 172.21.15.23:0/3803534983 conn(0x7ffbb0101f70 msgr2=0x7ffbb0102e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:16.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:16.192+0000 7ffbb4dce700 1 -- 172.21.15.23:0/3803534983 shutdown_connections 2024-03-28T05:14:16.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T05:14:16.192+0000 7ffbb4dce700 1 -- 172.21.15.23:0/3803534983 wait complete. 2024-03-28T05:14:16.777 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-28T05:14:16.777 INFO:teuthology.orchestra.run.smithi023.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-28T05:14:16.777 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:14:16.777 INFO:teuthology.orchestra.run.smithi023.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-28T05:14:16.777 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:14:16.777 INFO:teuthology.orchestra.run.smithi023.stdout:Or, if you are only running a single cluster on this host: 2024-03-28T05:14:16.777 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:14:16.777 INFO:teuthology.orchestra.run.smithi023.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-28T05:14:16.777 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:14:16.778 INFO:teuthology.orchestra.run.smithi023.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-28T05:14:16.778 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:14:16.778 INFO:teuthology.orchestra.run.smithi023.stdout: ceph telemetry on 2024-03-28T05:14:16.778 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:14:16.778 INFO:teuthology.orchestra.run.smithi023.stdout:For more information see: 2024-03-28T05:14:16.778 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:14:16.778 INFO:teuthology.orchestra.run.smithi023.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-28T05:14:16.778 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:14:16.778 INFO:teuthology.orchestra.run.smithi023.stdout:Bootstrap complete. 2024-03-28T05:14:16.812 INFO:tasks.cephadm:Fetching config... 2024-03-28T05:14:16.813 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T05:14:16.813 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-28T05:14:16.828 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-28T05:14:16.828 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T05:14:16.828 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-28T05:14:16.883 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-28T05:14:16.883 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T05:14:16.883 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-28T05:14:16.950 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-28T05:14:16.950 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T05:14:16.951 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-28T05:14:17.005 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-28T05:14:17.005 DEBUG:teuthology.orchestra.run.smithi023:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJm9JJyDDWwQrMhwNXtEGvp00PVOSTVjxw5bl5xgpTZFd2JUIegKutwN1l5Zozcw2gNVppK/hukUK49IYN1qfpC464mZUje438hcotyzRJvKDov2FU/7AAtVA7VcIWzjicD04dDIFk7I637dusTKotawi1inz2RalnaQBlsThkvqCkujx3iZ0zWqyVDvXpOdnWe7PlmdqTKdvZWe+oqLmU8tqaheIkFoEHUg/KkbLV9jJnR62lC9JSGhX3Tyz+tYSti7i5N8iNpcywRF3NSrnEXNV9xuULcLCwtbfn6vYxsMdblZggu+es14YB8MidwDqhvgQPQ4t5rM4s8QGsBGtJV6mp3EJtLymsvPCjX2w7iNYSRBJLhRdz9uFJIsUVUxPlP9COvTsEp3fIfVGLzoM7wZi6MbfOzUaDBqcdhqG2gJd+fNMOcuueILfkpD7uw6Q8KuRvnJcM1+5QSpT+C78wu+/OqNxEDcX8ZNSKODPXpxPcqne/mkJS8zLatNQv4dM= ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T05:14:17.083 INFO:teuthology.orchestra.run.smithi023.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJm9JJyDDWwQrMhwNXtEGvp00PVOSTVjxw5bl5xgpTZFd2JUIegKutwN1l5Zozcw2gNVppK/hukUK49IYN1qfpC464mZUje438hcotyzRJvKDov2FU/7AAtVA7VcIWzjicD04dDIFk7I637dusTKotawi1inz2RalnaQBlsThkvqCkujx3iZ0zWqyVDvXpOdnWe7PlmdqTKdvZWe+oqLmU8tqaheIkFoEHUg/KkbLV9jJnR62lC9JSGhX3Tyz+tYSti7i5N8iNpcywRF3NSrnEXNV9xuULcLCwtbfn6vYxsMdblZggu+es14YB8MidwDqhvgQPQ4t5rM4s8QGsBGtJV6mp3EJtLymsvPCjX2w7iNYSRBJLhRdz9uFJIsUVUxPlP9COvTsEp3fIfVGLzoM7wZi6MbfOzUaDBqcdhqG2gJd+fNMOcuueILfkpD7uw6Q8KuRvnJcM1+5QSpT+C78wu+/OqNxEDcX8ZNSKODPXpxPcqne/mkJS8zLatNQv4dM= ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:14:17.098 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJm9JJyDDWwQrMhwNXtEGvp00PVOSTVjxw5bl5xgpTZFd2JUIegKutwN1l5Zozcw2gNVppK/hukUK49IYN1qfpC464mZUje438hcotyzRJvKDov2FU/7AAtVA7VcIWzjicD04dDIFk7I637dusTKotawi1inz2RalnaQBlsThkvqCkujx3iZ0zWqyVDvXpOdnWe7PlmdqTKdvZWe+oqLmU8tqaheIkFoEHUg/KkbLV9jJnR62lC9JSGhX3Tyz+tYSti7i5N8iNpcywRF3NSrnEXNV9xuULcLCwtbfn6vYxsMdblZggu+es14YB8MidwDqhvgQPQ4t5rM4s8QGsBGtJV6mp3EJtLymsvPCjX2w7iNYSRBJLhRdz9uFJIsUVUxPlP9COvTsEp3fIfVGLzoM7wZi6MbfOzUaDBqcdhqG2gJd+fNMOcuueILfkpD7uw6Q8KuRvnJcM1+5QSpT+C78wu+/OqNxEDcX8ZNSKODPXpxPcqne/mkJS8zLatNQv4dM= ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T05:14:17.143 INFO:teuthology.orchestra.run.smithi098.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJm9JJyDDWwQrMhwNXtEGvp00PVOSTVjxw5bl5xgpTZFd2JUIegKutwN1l5Zozcw2gNVppK/hukUK49IYN1qfpC464mZUje438hcotyzRJvKDov2FU/7AAtVA7VcIWzjicD04dDIFk7I637dusTKotawi1inz2RalnaQBlsThkvqCkujx3iZ0zWqyVDvXpOdnWe7PlmdqTKdvZWe+oqLmU8tqaheIkFoEHUg/KkbLV9jJnR62lC9JSGhX3Tyz+tYSti7i5N8iNpcywRF3NSrnEXNV9xuULcLCwtbfn6vYxsMdblZggu+es14YB8MidwDqhvgQPQ4t5rM4s8QGsBGtJV6mp3EJtLymsvPCjX2w7iNYSRBJLhRdz9uFJIsUVUxPlP9COvTsEp3fIfVGLzoM7wZi6MbfOzUaDBqcdhqG2gJd+fNMOcuueILfkpD7uw6Q8KuRvnJcM1+5QSpT+C78wu+/OqNxEDcX8ZNSKODPXpxPcqne/mkJS8zLatNQv4dM= ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:14:17.158 DEBUG:teuthology.orchestra.run.smithi113:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJm9JJyDDWwQrMhwNXtEGvp00PVOSTVjxw5bl5xgpTZFd2JUIegKutwN1l5Zozcw2gNVppK/hukUK49IYN1qfpC464mZUje438hcotyzRJvKDov2FU/7AAtVA7VcIWzjicD04dDIFk7I637dusTKotawi1inz2RalnaQBlsThkvqCkujx3iZ0zWqyVDvXpOdnWe7PlmdqTKdvZWe+oqLmU8tqaheIkFoEHUg/KkbLV9jJnR62lC9JSGhX3Tyz+tYSti7i5N8iNpcywRF3NSrnEXNV9xuULcLCwtbfn6vYxsMdblZggu+es14YB8MidwDqhvgQPQ4t5rM4s8QGsBGtJV6mp3EJtLymsvPCjX2w7iNYSRBJLhRdz9uFJIsUVUxPlP9COvTsEp3fIfVGLzoM7wZi6MbfOzUaDBqcdhqG2gJd+fNMOcuueILfkpD7uw6Q8KuRvnJcM1+5QSpT+C78wu+/OqNxEDcX8ZNSKODPXpxPcqne/mkJS8zLatNQv4dM= ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T05:14:17.203 INFO:teuthology.orchestra.run.smithi113.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJm9JJyDDWwQrMhwNXtEGvp00PVOSTVjxw5bl5xgpTZFd2JUIegKutwN1l5Zozcw2gNVppK/hukUK49IYN1qfpC464mZUje438hcotyzRJvKDov2FU/7AAtVA7VcIWzjicD04dDIFk7I637dusTKotawi1inz2RalnaQBlsThkvqCkujx3iZ0zWqyVDvXpOdnWe7PlmdqTKdvZWe+oqLmU8tqaheIkFoEHUg/KkbLV9jJnR62lC9JSGhX3Tyz+tYSti7i5N8iNpcywRF3NSrnEXNV9xuULcLCwtbfn6vYxsMdblZggu+es14YB8MidwDqhvgQPQ4t5rM4s8QGsBGtJV6mp3EJtLymsvPCjX2w7iNYSRBJLhRdz9uFJIsUVUxPlP9COvTsEp3fIfVGLzoM7wZi6MbfOzUaDBqcdhqG2gJd+fNMOcuueILfkpD7uw6Q8KuRvnJcM1+5QSpT+C78wu+/OqNxEDcX8ZNSKODPXpxPcqne/mkJS8zLatNQv4dM= ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:14:17.218 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-28T05:14:17.592 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:14:19.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.938+0000 7fd9611fb700 1 -- 172.21.15.23:0/1697095104 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd95c101be0 msgr2=0x7fd95c102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:19.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.938+0000 7fd9611fb700 1 --2- 172.21.15.23:0/1697095104 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd95c101be0 0x7fd95c102000 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd94c009910 tx=0x7fd94c009c20 comp rx=0 tx=0).stop 2024-03-28T05:14:19.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.938+0000 7fd9611fb700 1 -- 172.21.15.23:0/1697095104 shutdown_connections 2024-03-28T05:14:19.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.938+0000 7fd9611fb700 1 --2- 172.21.15.23:0/1697095104 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd95c101be0 0x7fd95c102000 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:19.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.938+0000 7fd9611fb700 1 -- 172.21.15.23:0/1697095104 >> 172.21.15.23:0/1697095104 conn(0x7fd95c0fd180 msgr2=0x7fd95c0ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:19.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.938+0000 7fd9611fb700 1 -- 172.21.15.23:0/1697095104 shutdown_connections 2024-03-28T05:14:19.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.938+0000 7fd9611fb700 1 -- 172.21.15.23:0/1697095104 wait complete. 2024-03-28T05:14:19.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.939+0000 7fd9611fb700 1 Processor -- start 2024-03-28T05:14:19.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.939+0000 7fd9611fb700 1 -- start start 2024-03-28T05:14:19.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.940+0000 7fd9611fb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd95c101be0 0x7fd95c1972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:19.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.940+0000 7fd9611fb700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c107c70 con 0x7fd95c101be0 2024-03-28T05:14:19.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.940+0000 7fd95ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd95c101be0 0x7fd95c1972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:19.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.940+0000 7fd95ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd95c101be0 0x7fd95c1972c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45282/0 (socket says 172.21.15.23:45282) 2024-03-28T05:14:19.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.940+0000 7fd95ad9d700 1 -- 172.21.15.23:0/3359300450 learned_addr learned my addr 172.21.15.23:0/3359300450 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:19.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.940+0000 7fd95ad9d700 1 -- 172.21.15.23:0/3359300450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95c197800 con 0x7fd95c101be0 2024-03-28T05:14:19.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.940+0000 7fd95ad9d700 1 --2- 172.21.15.23:0/3359300450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd95c101be0 0x7fd95c1972c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd94c006010 tx=0x7fd94c005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:19.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.940+0000 7fd943fff700 1 -- 172.21.15.23:0/3359300450 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd94c027070 con 0x7fd95c101be0 2024-03-28T05:14:19.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.941+0000 7fd943fff700 1 -- 172.21.15.23:0/3359300450 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd94c00be40 con 0x7fd95c101be0 2024-03-28T05:14:19.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.941+0000 7fd943fff700 1 -- 172.21.15.23:0/3359300450 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd94c00f460 con 0x7fd95c101be0 2024-03-28T05:14:19.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.941+0000 7fd9611fb700 1 -- 172.21.15.23:0/3359300450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95c197a90 con 0x7fd95c101be0 2024-03-28T05:14:19.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.941+0000 7fd9611fb700 1 -- 172.21.15.23:0/3359300450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd95c19a780 con 0x7fd95c101be0 2024-03-28T05:14:19.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.941+0000 7fd943fff700 1 -- 172.21.15.23:0/3359300450 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fd94c02a070 con 0x7fd95c101be0 2024-03-28T05:14:19.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.942+0000 7fd943fff700 1 --2- 172.21.15.23:0/3359300450 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd944039020 0x7fd94403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:19.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.942+0000 7fd9611fb700 1 -- 172.21.15.23:0/3359300450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd95c1083f0 con 0x7fd95c101be0 2024-03-28T05:14:19.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.942+0000 7fd943fff700 1 -- 172.21.15.23:0/3359300450 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd94c01e030 con 0x7fd95c101be0 2024-03-28T05:14:19.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.942+0000 7fd95a59c700 1 --2- 172.21.15.23:0/3359300450 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd944039020 0x7fd94403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:19.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.942+0000 7fd95a59c700 1 --2- 172.21.15.23:0/3359300450 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd944039020 0x7fd94403b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd950006fd0 tx=0x7fd950006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:19.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:19.947+0000 7fd943fff700 1 -- 172.21.15.23:0/3359300450 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd94c013070 con 0x7fd95c101be0 2024-03-28T05:14:20.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:20.153+0000 7fd9611fb700 1 -- 172.21.15.23:0/3359300450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fd95c02db40 con 0x7fd95c101be0 2024-03-28T05:14:20.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:20.156+0000 7fd943fff700 1 -- 172.21.15.23:0/3359300450 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fd94c02a360 con 0x7fd95c101be0 2024-03-28T05:14:20.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:20.159+0000 7fd9611fb700 1 -- 172.21.15.23:0/3359300450 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd944039020 msgr2=0x7fd94403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:20.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:20.159+0000 7fd9611fb700 1 --2- 172.21.15.23:0/3359300450 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd944039020 0x7fd94403b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd950006fd0 tx=0x7fd950006e40 comp rx=0 tx=0).stop 2024-03-28T05:14:20.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:20.159+0000 7fd9611fb700 1 -- 172.21.15.23:0/3359300450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd95c101be0 msgr2=0x7fd95c1972c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:20.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:20.159+0000 7fd9611fb700 1 --2- 172.21.15.23:0/3359300450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd95c101be0 0x7fd95c1972c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd94c006010 tx=0x7fd94c005050 comp rx=0 tx=0).stop 2024-03-28T05:14:20.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:20.159+0000 7fd9611fb700 1 -- 172.21.15.23:0/3359300450 shutdown_connections 2024-03-28T05:14:20.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:20.159+0000 7fd9611fb700 1 --2- 172.21.15.23:0/3359300450 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd944039020 0x7fd94403b4e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:20.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:20.159+0000 7fd9611fb700 1 --2- 172.21.15.23:0/3359300450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd95c101be0 0x7fd95c1972c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:20.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:20.159+0000 7fd9611fb700 1 -- 172.21.15.23:0/3359300450 >> 172.21.15.23:0/3359300450 conn(0x7fd95c0fd180 msgr2=0x7fd95c0fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:20.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:20.159+0000 7fd9611fb700 1 -- 172.21.15.23:0/3359300450 shutdown_connections 2024-03-28T05:14:20.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:20.159+0000 7fd9611fb700 1 -- 172.21.15.23:0/3359300450 wait complete. 2024-03-28T05:14:21.036 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-28T05:14:21.036 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-28T05:14:21.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:21 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3359300450' entity='client.admin' 2024-03-28T05:14:21.608 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:14:22.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:22.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:22.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi023", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:14:22.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:22.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:14:22.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:23.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.060+0000 7f5b0018c700 1 -- 172.21.15.23:0/711974018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5af8100b00 msgr2=0x7f5af8100f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:23.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.060+0000 7f5b0018c700 1 --2- 172.21.15.23:0/711974018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5af8100b00 0x7f5af8100f20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5af4009910 tx=0x7f5af4009c20 comp rx=0 tx=0).stop 2024-03-28T05:14:23.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.060+0000 7f5b0018c700 1 -- 172.21.15.23:0/711974018 shutdown_connections 2024-03-28T05:14:23.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.060+0000 7f5b0018c700 1 --2- 172.21.15.23:0/711974018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5af8100b00 0x7f5af8100f20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:23.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.060+0000 7f5b0018c700 1 -- 172.21.15.23:0/711974018 >> 172.21.15.23:0/711974018 conn(0x7f5af80fc060 msgr2=0x7f5af80fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:23.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.061+0000 7f5b0018c700 1 -- 172.21.15.23:0/711974018 shutdown_connections 2024-03-28T05:14:23.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.061+0000 7f5b0018c700 1 -- 172.21.15.23:0/711974018 wait complete. 2024-03-28T05:14:23.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.061+0000 7f5b0018c700 1 Processor -- start 2024-03-28T05:14:23.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.061+0000 7f5b0018c700 1 -- start start 2024-03-28T05:14:23.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.061+0000 7f5b0018c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5af8100b00 0x7f5af818dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:23.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.061+0000 7f5b0018c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af8106b90 con 0x7f5af8100b00 2024-03-28T05:14:23.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.061+0000 7f5afdf28700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5af8100b00 0x7f5af818dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:23.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.061+0000 7f5afdf28700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5af8100b00 0x7f5af818dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54484/0 (socket says 172.21.15.23:54484) 2024-03-28T05:14:23.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.061+0000 7f5afdf28700 1 -- 172.21.15.23:0/1523272783 learned_addr learned my addr 172.21.15.23:0/1523272783 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:23.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.062+0000 7f5afdf28700 1 -- 172.21.15.23:0/1523272783 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af818e200 con 0x7f5af8100b00 2024-03-28T05:14:23.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.062+0000 7f5afdf28700 1 --2- 172.21.15.23:0/1523272783 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5af8100b00 0x7f5af818dcc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5af4006010 tx=0x7f5af4005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:23.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.062+0000 7f5aeaffd700 1 -- 172.21.15.23:0/1523272783 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5af4027070 con 0x7f5af8100b00 2024-03-28T05:14:23.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.062+0000 7f5aeaffd700 1 -- 172.21.15.23:0/1523272783 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5af400be40 con 0x7f5af8100b00 2024-03-28T05:14:23.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.062+0000 7f5b0018c700 1 -- 172.21.15.23:0/1523272783 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af818c3f0 con 0x7f5af8100b00 2024-03-28T05:14:23.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.062+0000 7f5aeaffd700 1 -- 172.21.15.23:0/1523272783 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5af400f460 con 0x7f5af8100b00 2024-03-28T05:14:23.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.062+0000 7f5b0018c700 1 -- 172.21.15.23:0/1523272783 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af818c930 con 0x7f5af8100b00 2024-03-28T05:14:23.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.063+0000 7f5aeaffd700 1 -- 172.21.15.23:0/1523272783 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f5af402a070 con 0x7f5af8100b00 2024-03-28T05:14:23.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.063+0000 7f5b0018c700 1 -- 172.21.15.23:0/1523272783 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af8107310 con 0x7f5af8100b00 2024-03-28T05:14:23.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.063+0000 7f5aeaffd700 1 --2- 172.21.15.23:0/1523272783 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5ae4038fc0 0x7f5ae403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:23.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.063+0000 7f5aeaffd700 1 -- 172.21.15.23:0/1523272783 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5af401e030 con 0x7f5af8100b00 2024-03-28T05:14:23.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.063+0000 7f5afd727700 1 --2- 172.21.15.23:0/1523272783 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5ae4038fc0 0x7f5ae403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:23.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.063+0000 7f5afd727700 1 --2- 172.21.15.23:0/1523272783 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5ae4038fc0 0x7f5ae403b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5aec006fd0 tx=0x7f5aec006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:23.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.068+0000 7f5aeaffd700 1 -- 172.21.15.23:0/1523272783 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5af4013070 con 0x7f5af8100b00 2024-03-28T05:14:23.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.275+0000 7f5b0018c700 1 -- 172.21.15.23:0/1523272783 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f5af818d040 con 0x7f5ae4038fc0 2024-03-28T05:14:23.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.281+0000 7f5aeaffd700 1 -- 172.21.15.23:0/1523272783 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5af818d040 con 0x7f5ae4038fc0 2024-03-28T05:14:23.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.283+0000 7f5b0018c700 1 -- 172.21.15.23:0/1523272783 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5ae4038fc0 msgr2=0x7f5ae403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:23.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.283+0000 7f5b0018c700 1 --2- 172.21.15.23:0/1523272783 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5ae4038fc0 0x7f5ae403b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5aec006fd0 tx=0x7f5aec006e40 comp rx=0 tx=0).stop 2024-03-28T05:14:23.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.283+0000 7f5b0018c700 1 -- 172.21.15.23:0/1523272783 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5af8100b00 msgr2=0x7f5af818dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:23.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.283+0000 7f5b0018c700 1 --2- 172.21.15.23:0/1523272783 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5af8100b00 0x7f5af818dcc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5af4006010 tx=0x7f5af4005050 comp rx=0 tx=0).stop 2024-03-28T05:14:23.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.283+0000 7f5b0018c700 1 -- 172.21.15.23:0/1523272783 shutdown_connections 2024-03-28T05:14:23.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.283+0000 7f5b0018c700 1 --2- 172.21.15.23:0/1523272783 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5ae4038fc0 0x7f5ae403b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:23.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.283+0000 7f5b0018c700 1 --2- 172.21.15.23:0/1523272783 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5af8100b00 0x7f5af818dcc0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:23.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.283+0000 7f5b0018c700 1 -- 172.21.15.23:0/1523272783 >> 172.21.15.23:0/1523272783 conn(0x7f5af80fc060 msgr2=0x7f5af80fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:23.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.283+0000 7f5b0018c700 1 -- 172.21.15.23:0/1523272783 shutdown_connections 2024-03-28T05:14:23.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:23.283+0000 7f5b0018c700 1 -- 172.21.15.23:0/1523272783 wait complete. 2024-03-28T05:14:24.021 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi098 2024-03-28T05:14:24.021 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-28T05:14:24.022 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/etc/ceph/ceph.conf 2024-03-28T05:14:24.040 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-28T05:14:24.040 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T05:14:24.097 INFO:tasks.cephadm:Adding host smithi098 to orchestrator... 2024-03-28T05:14:24.097 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph orch host add smithi098 2024-03-28T05:14:24.334 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:14:24.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:24 smithi023 ceph-mon[31954]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:14:24.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:24 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:24.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:24 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:14:24.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:24 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:14:24.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:24 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:14:24.535 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:24 smithi023 ceph-mon[31954]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-28T05:14:24.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:24 smithi023 ceph-mon[31954]: Updating smithi023:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:14:24.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:24 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:24.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:24 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:24.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:24 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:25.491 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:25 smithi023 ceph-mon[31954]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-28T05:14:25.491 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:25 smithi023 ceph-mon[31954]: Updating smithi023:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T05:14:25.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.710+0000 7f54fea9c700 1 -- 172.21.15.23:0/409604735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54f8102df0 msgr2=0x7f54f81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.710+0000 7f54fea9c700 1 --2- 172.21.15.23:0/409604735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54f8102df0 0x7f54f81031d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f54ec009910 tx=0x7f54ec009c20 comp rx=0 tx=0).stop 2024-03-28T05:14:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.711+0000 7f54fea9c700 1 -- 172.21.15.23:0/409604735 shutdown_connections 2024-03-28T05:14:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.711+0000 7f54fea9c700 1 --2- 172.21.15.23:0/409604735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54f8102df0 0x7f54f81031d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.711+0000 7f54fea9c700 1 -- 172.21.15.23:0/409604735 >> 172.21.15.23:0/409604735 conn(0x7f54f80fc580 msgr2=0x7f54f80fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.711+0000 7f54fea9c700 1 -- 172.21.15.23:0/409604735 shutdown_connections 2024-03-28T05:14:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.711+0000 7f54fea9c700 1 -- 172.21.15.23:0/409604735 wait complete. 2024-03-28T05:14:25.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.711+0000 7f54fea9c700 1 Processor -- start 2024-03-28T05:14:25.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.711+0000 7f54fea9c700 1 -- start start 2024-03-28T05:14:25.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.711+0000 7f54fea9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54f8102df0 0x7f54f81976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:25.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.711+0000 7f54fea9c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f810c210 con 0x7f54f8102df0 2024-03-28T05:14:25.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.711+0000 7f54fc838700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54f8102df0 0x7f54f81976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:25.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.711+0000 7f54fc838700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54f8102df0 0x7f54f81976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54502/0 (socket says 172.21.15.23:54502) 2024-03-28T05:14:25.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.711+0000 7f54fc838700 1 -- 172.21.15.23:0/2065532034 learned_addr learned my addr 172.21.15.23:0/2065532034 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:25.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.712+0000 7f54fc838700 1 -- 172.21.15.23:0/2065532034 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f8197c10 con 0x7f54f8102df0 2024-03-28T05:14:25.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.712+0000 7f54fc838700 1 --2- 172.21.15.23:0/2065532034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54f8102df0 0x7f54f81976d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f54ec006010 tx=0x7f54ec005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:25.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.712+0000 7f54f5ffb700 1 -- 172.21.15.23:0/2065532034 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54ec026070 con 0x7f54f8102df0 2024-03-28T05:14:25.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.712+0000 7f54f5ffb700 1 -- 172.21.15.23:0/2065532034 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f54ec00bc30 con 0x7f54f8102df0 2024-03-28T05:14:25.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.712+0000 7f54fea9c700 1 -- 172.21.15.23:0/2065532034 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f8197ea0 con 0x7f54f8102df0 2024-03-28T05:14:25.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.712+0000 7f54f5ffb700 1 -- 172.21.15.23:0/2065532034 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54ec003ab0 con 0x7f54f8102df0 2024-03-28T05:14:25.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.712+0000 7f54fea9c700 1 -- 172.21.15.23:0/2065532034 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f8198300 con 0x7f54f8102df0 2024-03-28T05:14:25.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.713+0000 7f54f5ffb700 1 -- 172.21.15.23:0/2065532034 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f54ec029070 con 0x7f54f8102df0 2024-03-28T05:14:25.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.713+0000 7f54fea9c700 1 -- 172.21.15.23:0/2065532034 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f806a810 con 0x7f54f8102df0 2024-03-28T05:14:25.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.713+0000 7f54f5ffb700 1 --2- 172.21.15.23:0/2065532034 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f54e8039020 0x7f54e803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:25.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.713+0000 7f54f5ffb700 1 -- 172.21.15.23:0/2065532034 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f54ec01e070 con 0x7f54f8102df0 2024-03-28T05:14:25.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.714+0000 7f54f7fff700 1 --2- 172.21.15.23:0/2065532034 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f54e8039020 0x7f54e803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:25.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.714+0000 7f54f7fff700 1 --2- 172.21.15.23:0/2065532034 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f54e8039020 0x7f54e803b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f54e4006fd0 tx=0x7f54e4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:25.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.719+0000 7f54f5ffb700 1 -- 172.21.15.23:0/2065532034 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f54ec013070 con 0x7f54f8102df0 2024-03-28T05:14:25.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:25.928+0000 7f54fea9c700 1 -- 172.21.15.23:0/2065532034 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi098", "target": ["mon-mgr", ""]}) v1 -- 0x7f54f8198a30 con 0x7f54e8039020 2024-03-28T05:14:27.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:26 smithi023 ceph-mon[31954]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi098", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:14:27.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:26 smithi023 ceph-mon[31954]: Deploying cephadm binary to smithi098 2024-03-28T05:14:30.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:29 smithi023 ceph-mon[31954]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:14:30.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:30.209+0000 7f54f5ffb700 1 -- 172.21.15.23:0/2065532034 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f54f8198a30 con 0x7f54e8039020 2024-03-28T05:14:30.207 INFO:teuthology.orchestra.run.smithi023.stdout:Added host 'smithi098' with addr '172.21.15.98' 2024-03-28T05:14:30.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:30.212+0000 7f54fea9c700 1 -- 172.21.15.23:0/2065532034 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f54e8039020 msgr2=0x7f54e803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:30.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:30.212+0000 7f54fea9c700 1 --2- 172.21.15.23:0/2065532034 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f54e8039020 0x7f54e803b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f54e4006fd0 tx=0x7f54e4006e40 comp rx=0 tx=0).stop 2024-03-28T05:14:30.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:30.212+0000 7f54fea9c700 1 -- 172.21.15.23:0/2065532034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54f8102df0 msgr2=0x7f54f81976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:30.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:30.212+0000 7f54fea9c700 1 --2- 172.21.15.23:0/2065532034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54f8102df0 0x7f54f81976d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f54ec006010 tx=0x7f54ec005050 comp rx=0 tx=0).stop 2024-03-28T05:14:30.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:30.212+0000 7f54fea9c700 1 -- 172.21.15.23:0/2065532034 shutdown_connections 2024-03-28T05:14:30.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:30.212+0000 7f54fea9c700 1 --2- 172.21.15.23:0/2065532034 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f54e8039020 0x7f54e803b4e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:30.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:30.212+0000 7f54fea9c700 1 --2- 172.21.15.23:0/2065532034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54f8102df0 0x7f54f81976d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:30.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:30.212+0000 7f54fea9c700 1 -- 172.21.15.23:0/2065532034 >> 172.21.15.23:0/2065532034 conn(0x7f54f80fc580 msgr2=0x7f54f8103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:30.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:30.212+0000 7f54fea9c700 1 -- 172.21.15.23:0/2065532034 shutdown_connections 2024-03-28T05:14:30.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:30.212+0000 7f54fea9c700 1 -- 172.21.15.23:0/2065532034 wait complete. 2024-03-28T05:14:30.841 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-28T05:14:31.085 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:14:31.468 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:31 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:31.468 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:31 smithi023 ceph-mon[31954]: Added host smithi098 2024-03-28T05:14:31.468 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:31 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:14:31.468 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:31 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:32.515 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:32 smithi023 ceph-mon[31954]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:14:32.516 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:32 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:32.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.540+0000 7f3acbe07700 1 -- 172.21.15.23:0/2811607756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ac4106a60 msgr2=0x7f3ac4106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:32.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.540+0000 7f3acbe07700 1 --2- 172.21.15.23:0/2811607756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ac4106a60 0x7f3ac4106e40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4009910 tx=0x7f3ab4009c20 comp rx=0 tx=0).stop 2024-03-28T05:14:32.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.540+0000 7f3acbe07700 1 -- 172.21.15.23:0/2811607756 shutdown_connections 2024-03-28T05:14:32.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.540+0000 7f3acbe07700 1 --2- 172.21.15.23:0/2811607756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ac4106a60 0x7f3ac4106e40 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:32.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.540+0000 7f3acbe07700 1 -- 172.21.15.23:0/2811607756 >> 172.21.15.23:0/2811607756 conn(0x7f3ac40776d0 msgr2=0x7f3ac40fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:32.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.540+0000 7f3acbe07700 1 -- 172.21.15.23:0/2811607756 shutdown_connections 2024-03-28T05:14:32.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.540+0000 7f3acbe07700 1 -- 172.21.15.23:0/2811607756 wait complete. 2024-03-28T05:14:32.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.541+0000 7f3acbe07700 1 Processor -- start 2024-03-28T05:14:32.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.541+0000 7f3acbe07700 1 -- start start 2024-03-28T05:14:32.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.541+0000 7f3acbe07700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ac4106a60 0x7f3ac41984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:32.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.541+0000 7f3acbe07700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac410b0a0 con 0x7f3ac4106a60 2024-03-28T05:14:32.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.541+0000 7f3ac9ba3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ac4106a60 0x7f3ac41984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:32.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.541+0000 7f3ac9ba3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ac4106a60 0x7f3ac41984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54514/0 (socket says 172.21.15.23:54514) 2024-03-28T05:14:32.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.541+0000 7f3ac9ba3700 1 -- 172.21.15.23:0/1674951045 learned_addr learned my addr 172.21.15.23:0/1674951045 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:32.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.541+0000 7f3ac9ba3700 1 -- 172.21.15.23:0/1674951045 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac4198a30 con 0x7f3ac4106a60 2024-03-28T05:14:32.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.542+0000 7f3ac9ba3700 1 --2- 172.21.15.23:0/1674951045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ac4106a60 0x7f3ac41984f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4005850 tx=0x7f3ab4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:32.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.542+0000 7f3abaffd700 1 -- 172.21.15.23:0/1674951045 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3ab4026070 con 0x7f3ac4106a60 2024-03-28T05:14:32.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.542+0000 7f3abaffd700 1 -- 172.21.15.23:0/1674951045 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ab400bbd0 con 0x7f3ac4106a60 2024-03-28T05:14:32.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.542+0000 7f3abaffd700 1 -- 172.21.15.23:0/1674951045 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3ab4003970 con 0x7f3ac4106a60 2024-03-28T05:14:32.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.542+0000 7f3acbe07700 1 -- 172.21.15.23:0/1674951045 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac4198d20 con 0x7f3ac4106a60 2024-03-28T05:14:32.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.542+0000 7f3acbe07700 1 -- 172.21.15.23:0/1674951045 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac41991e0 con 0x7f3ac4106a60 2024-03-28T05:14:32.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.543+0000 7f3abaffd700 1 -- 172.21.15.23:0/1674951045 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f3ab4029070 con 0x7f3ac4106a60 2024-03-28T05:14:32.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.543+0000 7f3acbe07700 1 -- 172.21.15.23:0/1674951045 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac41021a0 con 0x7f3ac4106a60 2024-03-28T05:14:32.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.543+0000 7f3abaffd700 1 --2- 172.21.15.23:0/1674951045 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3ab0039020 0x7f3ab003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:32.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.543+0000 7f3abaffd700 1 -- 172.21.15.23:0/1674951045 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3ab401e070 con 0x7f3ac4106a60 2024-03-28T05:14:32.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.543+0000 7f3ac93a2700 1 --2- 172.21.15.23:0/1674951045 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3ab0039020 0x7f3ab003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:32.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.543+0000 7f3ac93a2700 1 --2- 172.21.15.23:0/1674951045 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3ab0039020 0x7f3ab003b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0006fd0 tx=0x7f3ac0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:32.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.548+0000 7f3abaffd700 1 -- 172.21.15.23:0/1674951045 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3ab4013070 con 0x7f3ac4106a60 2024-03-28T05:14:32.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.759+0000 7f3acbe07700 1 -- 172.21.15.23:0/1674951045 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3ac4069b20 con 0x7f3ab0039020 2024-03-28T05:14:32.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.760+0000 7f3abaffd700 1 -- 172.21.15.23:0/1674951045 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f3ac4069b20 con 0x7f3ab0039020 2024-03-28T05:14:32.757 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:14:32.757 INFO:teuthology.orchestra.run.smithi023.stdout:[{"addr": "172.21.15.23", "hostname": "smithi023", "labels": [], "status": ""}, {"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}] 2024-03-28T05:14:32.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.762+0000 7f3acbe07700 1 -- 172.21.15.23:0/1674951045 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3ab0039020 msgr2=0x7f3ab003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:32.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.762+0000 7f3acbe07700 1 --2- 172.21.15.23:0/1674951045 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3ab0039020 0x7f3ab003b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0006fd0 tx=0x7f3ac0006e40 comp rx=0 tx=0).stop 2024-03-28T05:14:32.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.762+0000 7f3acbe07700 1 -- 172.21.15.23:0/1674951045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ac4106a60 msgr2=0x7f3ac41984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:32.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.762+0000 7f3acbe07700 1 --2- 172.21.15.23:0/1674951045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ac4106a60 0x7f3ac41984f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4005850 tx=0x7f3ab4005030 comp rx=0 tx=0).stop 2024-03-28T05:14:32.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.762+0000 7f3acbe07700 1 -- 172.21.15.23:0/1674951045 shutdown_connections 2024-03-28T05:14:32.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.762+0000 7f3acbe07700 1 --2- 172.21.15.23:0/1674951045 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3ab0039020 0x7f3ab003b4e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:32.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.762+0000 7f3acbe07700 1 --2- 172.21.15.23:0/1674951045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ac4106a60 0x7f3ac41984f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:32.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.762+0000 7f3acbe07700 1 -- 172.21.15.23:0/1674951045 >> 172.21.15.23:0/1674951045 conn(0x7f3ac40776d0 msgr2=0x7f3ac4108580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:32.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.762+0000 7f3acbe07700 1 -- 172.21.15.23:0/1674951045 shutdown_connections 2024-03-28T05:14:32.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:32.762+0000 7f3acbe07700 1 -- 172.21.15.23:0/1674951045 wait complete. 2024-03-28T05:14:33.337 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi113 2024-03-28T05:14:33.337 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-28T05:14:33.337 DEBUG:teuthology.orchestra.run.smithi113:> dd of=/etc/ceph/ceph.conf 2024-03-28T05:14:33.356 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-28T05:14:33.356 DEBUG:teuthology.orchestra.run.smithi113:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T05:14:33.413 INFO:tasks.cephadm:Adding host smithi113 to orchestrator... 2024-03-28T05:14:33.413 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph orch host add smithi113 2024-03-28T05:14:33.650 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:14:34.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:33 smithi023 ceph-mon[31954]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:14:34.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:33 smithi023 ceph-mon[31954]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T05:14:34.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:33 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:35.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.079+0000 7f3bfca21700 1 -- 172.21.15.23:0/4112146837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bf8106a50 msgr2=0x7f3bf8106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:35.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.079+0000 7f3bfca21700 1 --2- 172.21.15.23:0/4112146837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bf8106a50 0x7f3bf8106e30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3be8009910 tx=0x7f3be8009c20 comp rx=0 tx=0).stop 2024-03-28T05:14:35.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.079+0000 7f3bfca21700 1 -- 172.21.15.23:0/4112146837 shutdown_connections 2024-03-28T05:14:35.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.079+0000 7f3bfca21700 1 --2- 172.21.15.23:0/4112146837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bf8106a50 0x7f3bf8106e30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:35.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.079+0000 7f3bfca21700 1 -- 172.21.15.23:0/4112146837 >> 172.21.15.23:0/4112146837 conn(0x7f3bf80fc540 msgr2=0x7f3bf80fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:35.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.079+0000 7f3bfca21700 1 -- 172.21.15.23:0/4112146837 shutdown_connections 2024-03-28T05:14:35.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.079+0000 7f3bfca21700 1 -- 172.21.15.23:0/4112146837 wait complete. 2024-03-28T05:14:35.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.080+0000 7f3bfca21700 1 Processor -- start 2024-03-28T05:14:35.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.080+0000 7f3bfca21700 1 -- start start 2024-03-28T05:14:35.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.080+0000 7f3bfca21700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bf8106a50 0x7f3bf81965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:35.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.080+0000 7f3bfca21700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf810b090 con 0x7f3bf8106a50 2024-03-28T05:14:35.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.080+0000 7f3bf659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bf8106a50 0x7f3bf81965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:35.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.080+0000 7f3bf659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bf8106a50 0x7f3bf81965a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43042/0 (socket says 172.21.15.23:43042) 2024-03-28T05:14:35.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.080+0000 7f3bf659c700 1 -- 172.21.15.23:0/2837341554 learned_addr learned my addr 172.21.15.23:0/2837341554 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:35.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.080+0000 7f3bf659c700 1 -- 172.21.15.23:0/2837341554 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf8196ae0 con 0x7f3bf8106a50 2024-03-28T05:14:35.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.081+0000 7f3bf659c700 1 --2- 172.21.15.23:0/2837341554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bf8106a50 0x7f3bf81965a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3be8005850 tx=0x7f3be8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:35.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.081+0000 7f3be77fe700 1 -- 172.21.15.23:0/2837341554 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3be8026070 con 0x7f3bf8106a50 2024-03-28T05:14:35.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.081+0000 7f3bfca21700 1 -- 172.21.15.23:0/2837341554 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf8196dd0 con 0x7f3bf8106a50 2024-03-28T05:14:35.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.081+0000 7f3be77fe700 1 -- 172.21.15.23:0/2837341554 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3be800bbd0 con 0x7f3bf8106a50 2024-03-28T05:14:35.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.081+0000 7f3be77fe700 1 -- 172.21.15.23:0/2837341554 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3be8003970 con 0x7f3bf8106a50 2024-03-28T05:14:35.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.081+0000 7f3bfca21700 1 -- 172.21.15.23:0/2837341554 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf819ae70 con 0x7f3bf8106a50 2024-03-28T05:14:35.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.082+0000 7f3be77fe700 1 -- 172.21.15.23:0/2837341554 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f3be8029070 con 0x7f3bf8106a50 2024-03-28T05:14:35.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.082+0000 7f3bfca21700 1 -- 172.21.15.23:0/2837341554 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf8102190 con 0x7f3bf8106a50 2024-03-28T05:14:35.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.082+0000 7f3be77fe700 1 --2- 172.21.15.23:0/2837341554 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3be0038fc0 0x7f3be003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:35.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.082+0000 7f3be77fe700 1 -- 172.21.15.23:0/2837341554 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3be801e070 con 0x7f3bf8106a50 2024-03-28T05:14:35.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.082+0000 7f3bf5d9b700 1 --2- 172.21.15.23:0/2837341554 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3be0038fc0 0x7f3be003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:35.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.082+0000 7f3bf5d9b700 1 --2- 172.21.15.23:0/2837341554 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3be0038fc0 0x7f3be003b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3bec006fd0 tx=0x7f3bec006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:35.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.087+0000 7f3be77fe700 1 -- 172.21.15.23:0/2837341554 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3be8013070 con 0x7f3bf8106a50 2024-03-28T05:14:35.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:35.311+0000 7f3bfca21700 1 -- 172.21.15.23:0/2837341554 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi113", "target": ["mon-mgr", ""]}) v1 -- 0x7f3bf8197900 con 0x7f3be0038fc0 2024-03-28T05:14:36.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:35 smithi023 ceph-mon[31954]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:14:36.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:35 smithi023 ceph-mon[31954]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi113", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:14:36.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:35 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:36.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:35 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:36.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:35 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:36.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:35 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:36.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:35 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:14:36.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:35 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:14:36.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:35 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:14:37.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:36 smithi023 ceph-mon[31954]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-28T05:14:37.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:36 smithi023 ceph-mon[31954]: Deploying cephadm binary to smithi113 2024-03-28T05:14:37.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:36 smithi023 ceph-mon[31954]: Updating smithi098:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:14:37.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:36 smithi023 ceph-mon[31954]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-28T05:14:37.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:36 smithi023 ceph-mon[31954]: Updating smithi098:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T05:14:38.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:37 smithi023 ceph-mon[31954]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:14:39.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:39.695+0000 7f3be77fe700 1 -- 172.21.15.23:0/2837341554 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f3bf8197900 con 0x7f3be0038fc0 2024-03-28T05:14:39.692 INFO:teuthology.orchestra.run.smithi023.stdout:Added host 'smithi113' with addr '172.21.15.113' 2024-03-28T05:14:39.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:39.697+0000 7f3bfca21700 1 -- 172.21.15.23:0/2837341554 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3be0038fc0 msgr2=0x7f3be003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:39.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:39.697+0000 7f3bfca21700 1 --2- 172.21.15.23:0/2837341554 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3be0038fc0 0x7f3be003b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3bec006fd0 tx=0x7f3bec006e40 comp rx=0 tx=0).stop 2024-03-28T05:14:39.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:39.697+0000 7f3bfca21700 1 -- 172.21.15.23:0/2837341554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bf8106a50 msgr2=0x7f3bf81965a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:39.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:39.697+0000 7f3bfca21700 1 --2- 172.21.15.23:0/2837341554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bf8106a50 0x7f3bf81965a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3be8005850 tx=0x7f3be8005030 comp rx=0 tx=0).stop 2024-03-28T05:14:39.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:39.697+0000 7f3bfca21700 1 -- 172.21.15.23:0/2837341554 shutdown_connections 2024-03-28T05:14:39.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:39.697+0000 7f3bfca21700 1 --2- 172.21.15.23:0/2837341554 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3be0038fc0 0x7f3be003b480 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:39.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:39.697+0000 7f3bfca21700 1 --2- 172.21.15.23:0/2837341554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bf8106a50 0x7f3bf81965a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:39.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:39.697+0000 7f3bfca21700 1 -- 172.21.15.23:0/2837341554 >> 172.21.15.23:0/2837341554 conn(0x7f3bf80fc540 msgr2=0x7f3bf81085d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:39.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:39.697+0000 7f3bfca21700 1 -- 172.21.15.23:0/2837341554 shutdown_connections 2024-03-28T05:14:39.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:39.697+0000 7f3bfca21700 1 -- 172.21.15.23:0/2837341554 wait complete. 2024-03-28T05:14:40.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:40 smithi023 ceph-mon[31954]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:14:40.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:40 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:40.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:40 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:40.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:40 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:40.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:40 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:40.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:40 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:14:40.250 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-28T05:14:40.488 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:14:41.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:41 smithi023 ceph-mon[31954]: Added host smithi113 2024-03-28T05:14:41.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:41 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:41.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:41 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:41.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.840+0000 7f1974e91700 1 -- 172.21.15.23:0/970538661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1970101dd0 msgr2=0x7f19701021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:41.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.840+0000 7f1974e91700 1 --2- 172.21.15.23:0/970538661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1970101dd0 0x7f19701021b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1964009910 tx=0x7f1964009c20 comp rx=0 tx=0).stop 2024-03-28T05:14:41.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.840+0000 7f1974e91700 1 -- 172.21.15.23:0/970538661 shutdown_connections 2024-03-28T05:14:41.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.840+0000 7f1974e91700 1 --2- 172.21.15.23:0/970538661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1970101dd0 0x7f19701021b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:41.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.840+0000 7f1974e91700 1 -- 172.21.15.23:0/970538661 >> 172.21.15.23:0/970538661 conn(0x7f19700fd6a0 msgr2=0x7f19700ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:41.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.840+0000 7f1974e91700 1 -- 172.21.15.23:0/970538661 shutdown_connections 2024-03-28T05:14:41.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.840+0000 7f1974e91700 1 -- 172.21.15.23:0/970538661 wait complete. 2024-03-28T05:14:41.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.840+0000 7f1974e91700 1 Processor -- start 2024-03-28T05:14:41.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.840+0000 7f1974e91700 1 -- start start 2024-03-28T05:14:41.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.841+0000 7f1974e91700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1970101dd0 0x7f1970074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:41.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.841+0000 7f1974e91700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197010c1a0 con 0x7f1970101dd0 2024-03-28T05:14:41.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.841+0000 7f196e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1970101dd0 0x7f1970074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:41.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.841+0000 7f196e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1970101dd0 0x7f1970074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43060/0 (socket says 172.21.15.23:43060) 2024-03-28T05:14:41.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.841+0000 7f196e59c700 1 -- 172.21.15.23:0/673367757 learned_addr learned my addr 172.21.15.23:0/673367757 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:41.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.841+0000 7f196e59c700 1 -- 172.21.15.23:0/673367757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1970076380 con 0x7f1970101dd0 2024-03-28T05:14:41.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.841+0000 7f196e59c700 1 --2- 172.21.15.23:0/673367757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1970101dd0 0x7f1970074490 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1964006010 tx=0x7f1964005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:41.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.841+0000 7f195f7fe700 1 -- 172.21.15.23:0/673367757 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1964026070 con 0x7f1970101dd0 2024-03-28T05:14:41.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.841+0000 7f1974e91700 1 -- 172.21.15.23:0/673367757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1970074a30 con 0x7f1970101dd0 2024-03-28T05:14:41.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.841+0000 7f195f7fe700 1 -- 172.21.15.23:0/673367757 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f196400bc30 con 0x7f1970101dd0 2024-03-28T05:14:41.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.841+0000 7f195f7fe700 1 -- 172.21.15.23:0/673367757 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19640039d0 con 0x7f1970101dd0 2024-03-28T05:14:41.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.841+0000 7f1974e91700 1 -- 172.21.15.23:0/673367757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1970074ef0 con 0x7f1970101dd0 2024-03-28T05:14:41.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.842+0000 7f195f7fe700 1 -- 172.21.15.23:0/673367757 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f1964029070 con 0x7f1970101dd0 2024-03-28T05:14:41.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.842+0000 7f1974e91700 1 -- 172.21.15.23:0/673367757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19701032b0 con 0x7f1970101dd0 2024-03-28T05:14:41.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.842+0000 7f195f7fe700 1 --2- 172.21.15.23:0/673367757 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1958038fc0 0x7f195803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:41.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.842+0000 7f195f7fe700 1 -- 172.21.15.23:0/673367757 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f196401e070 con 0x7f1970101dd0 2024-03-28T05:14:41.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.843+0000 7f196dd9b700 1 --2- 172.21.15.23:0/673367757 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1958038fc0 0x7f195803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:41.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.843+0000 7f196dd9b700 1 --2- 172.21.15.23:0/673367757 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1958038fc0 0x7f195803b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1960006fd0 tx=0x7f1960006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:41.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:41.849+0000 7f195f7fe700 1 -- 172.21.15.23:0/673367757 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1964013070 con 0x7f1970101dd0 2024-03-28T05:14:42.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:42.056+0000 7f1974e91700 1 -- 172.21.15.23:0/673367757 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f19700759c0 con 0x7f1958038fc0 2024-03-28T05:14:42.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:42.057+0000 7f195f7fe700 1 -- 172.21.15.23:0/673367757 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+239 (secure 0 0 0) 0x7f19700759c0 con 0x7f1958038fc0 2024-03-28T05:14:42.054 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:14:42.054 INFO:teuthology.orchestra.run.smithi023.stdout:[{"addr": "172.21.15.23", "hostname": "smithi023", "labels": [], "status": ""}, {"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}, {"addr": "172.21.15.113", "hostname": "smithi113", "labels": [], "status": ""}] 2024-03-28T05:14:42.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:42.059+0000 7f1974e91700 1 -- 172.21.15.23:0/673367757 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1958038fc0 msgr2=0x7f195803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:42.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:42.059+0000 7f1974e91700 1 --2- 172.21.15.23:0/673367757 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1958038fc0 0x7f195803b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1960006fd0 tx=0x7f1960006e40 comp rx=0 tx=0).stop 2024-03-28T05:14:42.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:42.059+0000 7f1974e91700 1 -- 172.21.15.23:0/673367757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1970101dd0 msgr2=0x7f1970074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:42.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:42.059+0000 7f1974e91700 1 --2- 172.21.15.23:0/673367757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1970101dd0 0x7f1970074490 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1964006010 tx=0x7f1964005030 comp rx=0 tx=0).stop 2024-03-28T05:14:42.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:42.059+0000 7f1974e91700 1 -- 172.21.15.23:0/673367757 shutdown_connections 2024-03-28T05:14:42.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:42.060+0000 7f1974e91700 1 --2- 172.21.15.23:0/673367757 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1958038fc0 0x7f195803b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:42.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:42.060+0000 7f1974e91700 1 --2- 172.21.15.23:0/673367757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1970101dd0 0x7f1970074490 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:42.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:42.060+0000 7f1974e91700 1 -- 172.21.15.23:0/673367757 >> 172.21.15.23:0/673367757 conn(0x7f19700fd6a0 msgr2=0x7f1970104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:42.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:42.060+0000 7f1974e91700 1 -- 172.21.15.23:0/673367757 shutdown_connections 2024-03-28T05:14:42.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:42.060+0000 7f1974e91700 1 -- 172.21.15.23:0/673367757 wait complete. 2024-03-28T05:14:42.411 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:42 smithi023 ceph-mon[31954]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:14:42.721 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-28T05:14:42.721 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-28T05:14:42.958 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:14:43.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:43 smithi023 ceph-mon[31954]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T05:14:43.339 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:43 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:44.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.338+0000 7f1c2b40a700 1 -- 172.21.15.23:0/1498764879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c24106ba0 msgr2=0x7f1c24106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:44.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.338+0000 7f1c2b40a700 1 --2- 172.21.15.23:0/1498764879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c24106ba0 0x7f1c24106f80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1c20009910 tx=0x7f1c20009c20 comp rx=0 tx=0).stop 2024-03-28T05:14:44.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.339+0000 7f1c2b40a700 1 -- 172.21.15.23:0/1498764879 shutdown_connections 2024-03-28T05:14:44.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.339+0000 7f1c2b40a700 1 --2- 172.21.15.23:0/1498764879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c24106ba0 0x7f1c24106f80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:44.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.339+0000 7f1c2b40a700 1 -- 172.21.15.23:0/1498764879 >> 172.21.15.23:0/1498764879 conn(0x7f1c240fbce0 msgr2=0x7f1c240fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:44.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.339+0000 7f1c2b40a700 1 -- 172.21.15.23:0/1498764879 shutdown_connections 2024-03-28T05:14:44.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.339+0000 7f1c2b40a700 1 -- 172.21.15.23:0/1498764879 wait complete. 2024-03-28T05:14:44.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.339+0000 7f1c2b40a700 1 Processor -- start 2024-03-28T05:14:44.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.339+0000 7f1c2b40a700 1 -- start start 2024-03-28T05:14:44.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.339+0000 7f1c2b40a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c24106ba0 0x7f1c2419d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:44.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.339+0000 7f1c2b40a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2410c300 con 0x7f1c24106ba0 2024-03-28T05:14:44.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.339+0000 7f1c291a6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c24106ba0 0x7f1c2419d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:44.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.339+0000 7f1c291a6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c24106ba0 0x7f1c2419d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53388/0 (socket says 172.21.15.23:53388) 2024-03-28T05:14:44.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.340+0000 7f1c291a6700 1 -- 172.21.15.23:0/3726618071 learned_addr learned my addr 172.21.15.23:0/3726618071 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:14:44.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.340+0000 7f1c291a6700 1 -- 172.21.15.23:0/3726618071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2419d910 con 0x7f1c24106ba0 2024-03-28T05:14:44.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.340+0000 7f1c291a6700 1 --2- 172.21.15.23:0/3726618071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c24106ba0 0x7f1c2419d3d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1c20005850 tx=0x7f1c20005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:44.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.340+0000 7f1c1a7fc700 1 -- 172.21.15.23:0/3726618071 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c20026070 con 0x7f1c24106ba0 2024-03-28T05:14:44.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.340+0000 7f1c2b40a700 1 -- 172.21.15.23:0/3726618071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2419dba0 con 0x7f1c24106ba0 2024-03-28T05:14:44.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.340+0000 7f1c1a7fc700 1 -- 172.21.15.23:0/3726618071 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c2000bc30 con 0x7f1c24106ba0 2024-03-28T05:14:44.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.340+0000 7f1c1a7fc700 1 -- 172.21.15.23:0/3726618071 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c20003a20 con 0x7f1c24106ba0 2024-03-28T05:14:44.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.340+0000 7f1c2b40a700 1 -- 172.21.15.23:0/3726618071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c24197590 con 0x7f1c24106ba0 2024-03-28T05:14:44.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.341+0000 7f1c1a7fc700 1 -- 172.21.15.23:0/3726618071 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f1c20029070 con 0x7f1c24106ba0 2024-03-28T05:14:44.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.341+0000 7f1c2b40a700 1 -- 172.21.15.23:0/3726618071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c241032e0 con 0x7f1c24106ba0 2024-03-28T05:14:44.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.341+0000 7f1c1a7fc700 1 --2- 172.21.15.23:0/3726618071 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1c10038fc0 0x7f1c1003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:44.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.341+0000 7f1c1a7fc700 1 -- 172.21.15.23:0/3726618071 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1c2001e070 con 0x7f1c24106ba0 2024-03-28T05:14:44.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.342+0000 7f1c289a5700 1 --2- 172.21.15.23:0/3726618071 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1c10038fc0 0x7f1c1003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:44.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.342+0000 7f1c289a5700 1 --2- 172.21.15.23:0/3726618071 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1c10038fc0 0x7f1c1003b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1c14006fd0 tx=0x7f1c14006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:44.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.346+0000 7f1c1a7fc700 1 -- 172.21.15.23:0/3726618071 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1c20052020 con 0x7f1c24106ba0 2024-03-28T05:14:44.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:44 smithi023 ceph-mon[31954]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:14:44.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:44.557+0000 7f1c2b40a700 1 -- 172.21.15.23:0/3726618071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f1c240637c0 con 0x7f1c24106ba0 2024-03-28T05:14:45.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:45.267+0000 7f1c1a7fc700 1 -- 172.21.15.23:0/3726618071 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1c20030090 con 0x7f1c24106ba0 2024-03-28T05:14:45.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:45.270+0000 7f1c2b40a700 1 -- 172.21.15.23:0/3726618071 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1c10038fc0 msgr2=0x7f1c1003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:45.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:45.270+0000 7f1c2b40a700 1 --2- 172.21.15.23:0/3726618071 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1c10038fc0 0x7f1c1003b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1c14006fd0 tx=0x7f1c14006e40 comp rx=0 tx=0).stop 2024-03-28T05:14:45.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:45.270+0000 7f1c2b40a700 1 -- 172.21.15.23:0/3726618071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c24106ba0 msgr2=0x7f1c2419d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:45.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:45.270+0000 7f1c2b40a700 1 --2- 172.21.15.23:0/3726618071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c24106ba0 0x7f1c2419d3d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1c20005850 tx=0x7f1c20005030 comp rx=0 tx=0).stop 2024-03-28T05:14:45.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:45.270+0000 7f1c2b40a700 1 -- 172.21.15.23:0/3726618071 shutdown_connections 2024-03-28T05:14:45.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:45.270+0000 7f1c2b40a700 1 --2- 172.21.15.23:0/3726618071 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1c10038fc0 0x7f1c1003b480 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:45.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:45.270+0000 7f1c2b40a700 1 --2- 172.21.15.23:0/3726618071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c24106ba0 0x7f1c2419d3d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:45.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:45.270+0000 7f1c2b40a700 1 -- 172.21.15.23:0/3726618071 >> 172.21.15.23:0/3726618071 conn(0x7f1c240fbce0 msgr2=0x7f1c24109760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:45.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:45.270+0000 7f1c2b40a700 1 -- 172.21.15.23:0/3726618071 shutdown_connections 2024-03-28T05:14:45.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:14:45.271+0000 7f1c2b40a700 1 -- 172.21.15.23:0/3726618071 wait complete. 2024-03-28T05:14:45.268 INFO:teuthology.orchestra.run.smithi023.stderr:adjusted tunables profile to default 2024-03-28T05:14:45.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:45 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3726618071' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-28T05:14:45.868 INFO:tasks.cephadm:Adding mon.a on smithi023 2024-03-28T05:14:45.868 INFO:tasks.cephadm:Adding mon.b on smithi098 2024-03-28T05:14:45.868 INFO:tasks.cephadm:Adding mon.c on smithi113 2024-03-28T05:14:45.869 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi023:172.21.15.23=a;smithi098:172.21.15.98=b;smithi113:172.21.15.113=c' 2024-03-28T05:14:46.101 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:14:46.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:46 smithi023 ceph-mon[31954]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:14:46.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:46 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3726618071' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-28T05:14:46.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:46 smithi023 ceph-mon[31954]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T05:14:46.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:46 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:46.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:46 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:46.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:46 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:46.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:46 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:46.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:46 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi113", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:14:46.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:46 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:14:46.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:46 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:14:46.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:46 smithi023 ceph-mon[31954]: Updating smithi113:/etc/ceph/ceph.conf 2024-03-28T05:14:46.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:46 smithi023 ceph-mon[31954]: Updating smithi113:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:14:47.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:47 smithi023 ceph-mon[31954]: Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-03-28T05:14:47.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:47 smithi023 ceph-mon[31954]: Updating smithi113:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T05:14:47.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:47 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:47.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:47 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:47.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:47 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:47.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.572+0000 7fa73f837700 1 -- 172.21.15.113:0/1480019776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa738100b20 msgr2=0x7fa738100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:47.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.572+0000 7fa73f837700 1 --2- 172.21.15.113:0/1480019776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa738100b20 0x7fa738100f00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa734009910 tx=0x7fa734009c20 comp rx=0 tx=0).stop 2024-03-28T05:14:47.570 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.572+0000 7fa73f837700 1 -- 172.21.15.113:0/1480019776 shutdown_connections 2024-03-28T05:14:47.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.572+0000 7fa73f837700 1 --2- 172.21.15.113:0/1480019776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa738100b20 0x7fa738100f00 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:47.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.572+0000 7fa73f837700 1 -- 172.21.15.113:0/1480019776 >> 172.21.15.113:0/1480019776 conn(0x7fa7380fc580 msgr2=0x7fa7380fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:47.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.572+0000 7fa73f837700 1 -- 172.21.15.113:0/1480019776 shutdown_connections 2024-03-28T05:14:47.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.572+0000 7fa73f837700 1 -- 172.21.15.113:0/1480019776 wait complete. 2024-03-28T05:14:47.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.573+0000 7fa73f837700 1 Processor -- start 2024-03-28T05:14:47.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.573+0000 7fa73f837700 1 -- start start 2024-03-28T05:14:47.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.573+0000 7fa73f837700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa738100b20 0x7fa7381a3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:47.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.573+0000 7fa73f837700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa738112820 con 0x7fa738100b20 2024-03-28T05:14:47.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.573+0000 7fa73d5d3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa738100b20 0x7fa7381a3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:47.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.573+0000 7fa73d5d3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa738100b20 0x7fa7381a3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.113:37326/0 (socket says 172.21.15.113:37326) 2024-03-28T05:14:47.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.573+0000 7fa73d5d3700 1 -- 172.21.15.113:0/2455535212 learned_addr learned my addr 172.21.15.113:0/2455535212 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-03-28T05:14:47.571 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.573+0000 7fa73d5d3700 1 -- 172.21.15.113:0/2455535212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7381a3f80 con 0x7fa738100b20 2024-03-28T05:14:47.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.574+0000 7fa73d5d3700 1 --2- 172.21.15.113:0/2455535212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa738100b20 0x7fa7381a3a40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa734006010 tx=0x7fa734005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:47.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.574+0000 7fa72e7fc700 1 -- 172.21.15.113:0/2455535212 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa734026070 con 0x7fa738100b20 2024-03-28T05:14:47.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.574+0000 7fa73f837700 1 -- 172.21.15.113:0/2455535212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7381a4210 con 0x7fa738100b20 2024-03-28T05:14:47.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.574+0000 7fa72e7fc700 1 -- 172.21.15.113:0/2455535212 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa73400bc30 con 0x7fa738100b20 2024-03-28T05:14:47.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.574+0000 7fa72e7fc700 1 -- 172.21.15.113:0/2455535212 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa734003ab0 con 0x7fa738100b20 2024-03-28T05:14:47.572 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.574+0000 7fa73f837700 1 -- 172.21.15.113:0/2455535212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa73819db10 con 0x7fa738100b20 2024-03-28T05:14:47.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.575+0000 7fa73f837700 1 -- 172.21.15.113:0/2455535212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa73806a8d0 con 0x7fa738100b20 2024-03-28T05:14:47.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.575+0000 7fa72e7fc700 1 -- 172.21.15.113:0/2455535212 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fa734029070 con 0x7fa738100b20 2024-03-28T05:14:47.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.575+0000 7fa72e7fc700 1 --2- 172.21.15.113:0/2455535212 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa724038fc0 0x7fa72403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:47.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.575+0000 7fa72e7fc700 1 -- 172.21.15.113:0/2455535212 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa73401e070 con 0x7fa738100b20 2024-03-28T05:14:47.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.575+0000 7fa73cdd2700 1 --2- 172.21.15.113:0/2455535212 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa724038fc0 0x7fa72403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:47.573 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.576+0000 7fa73cdd2700 1 --2- 172.21.15.113:0/2455535212 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa724038fc0 0x7fa72403b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa728006fd0 tx=0x7fa728006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:47.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.580+0000 7fa72e7fc700 1 -- 172.21.15.113:0/2455535212 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa734013070 con 0x7fa738100b20 2024-03-28T05:14:47.786 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.789+0000 7fa73f837700 1 -- 172.21.15.113:0/2455535212 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi023:172.21.15.23=a;smithi098:172.21.15.98=b;smithi113:172.21.15.113=c", "target": ["mon-mgr", ""]}) v1 -- 0x7fa73819efd0 con 0x7fa724038fc0 2024-03-28T05:14:47.794 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.798+0000 7fa72e7fc700 1 -- 172.21.15.113:0/2455535212 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa73819efd0 con 0x7fa724038fc0 2024-03-28T05:14:47.795 INFO:teuthology.orchestra.run.smithi113.stdout:Scheduled mon update... 2024-03-28T05:14:47.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.800+0000 7fa73f837700 1 -- 172.21.15.113:0/2455535212 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa724038fc0 msgr2=0x7fa72403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:47.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.800+0000 7fa73f837700 1 --2- 172.21.15.113:0/2455535212 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa724038fc0 0x7fa72403b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa728006fd0 tx=0x7fa728006e40 comp rx=0 tx=0).stop 2024-03-28T05:14:47.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.800+0000 7fa73f837700 1 -- 172.21.15.113:0/2455535212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa738100b20 msgr2=0x7fa7381a3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:47.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.800+0000 7fa73f837700 1 --2- 172.21.15.113:0/2455535212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa738100b20 0x7fa7381a3a40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa734006010 tx=0x7fa734005050 comp rx=0 tx=0).stop 2024-03-28T05:14:47.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.800+0000 7fa73f837700 1 -- 172.21.15.113:0/2455535212 shutdown_connections 2024-03-28T05:14:47.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.800+0000 7fa73f837700 1 --2- 172.21.15.113:0/2455535212 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa724038fc0 0x7fa72403b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:47.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.800+0000 7fa73f837700 1 --2- 172.21.15.113:0/2455535212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa738100b20 0x7fa7381a3a40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:47.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.800+0000 7fa73f837700 1 -- 172.21.15.113:0/2455535212 >> 172.21.15.113:0/2455535212 conn(0x7fa7380fc580 msgr2=0x7fa7380fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:47.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.800+0000 7fa73f837700 1 -- 172.21.15.113:0/2455535212 shutdown_connections 2024-03-28T05:14:47.797 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:47.800+0000 7fa73f837700 1 -- 172.21.15.113:0/2455535212 wait complete. 2024-03-28T05:14:48.366 DEBUG:teuthology.orchestra.run.smithi098:mon.b> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mon.b.service 2024-03-28T05:14:48.369 DEBUG:teuthology.orchestra.run.smithi113:mon.c> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mon.c.service 2024-03-28T05:14:48.371 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T05:14:48.371 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T05:14:48.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:48 smithi023 ceph-mon[31954]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:14:48.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:48.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:14:48.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:14:48.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:14:48.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:48.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T05:14:48.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:14:49.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:49 smithi023 ceph-mon[31954]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi023:172.21.15.23=a;smithi098:172.21.15.98=b;smithi113:172.21.15.113=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:14:49.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:49 smithi023 ceph-mon[31954]: Saving service mon spec with placement smithi023:172.21.15.23=a;smithi098:172.21.15.98=b;smithi113:172.21.15.113=c;count:3 2024-03-28T05:14:49.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:49 smithi023 ceph-mon[31954]: Deploying daemon mon.c on smithi113 2024-03-28T05:14:50.027 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:14:50.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:50 smithi023 ceph-mon[31954]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:14:52.469 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.469+0000 7f86921b3700 1 -- 172.21.15.113:0/3630187567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f868c105a60 msgr2=0x7f868c105e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:52.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.469+0000 7f86921b3700 1 --2- 172.21.15.113:0/3630187567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f868c105a60 0x7f868c105e40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f867c009910 tx=0x7f867c009c20 comp rx=0 tx=0).stop 2024-03-28T05:14:52.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.469+0000 7f86921b3700 1 -- 172.21.15.113:0/3630187567 shutdown_connections 2024-03-28T05:14:52.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.469+0000 7f86921b3700 1 --2- 172.21.15.113:0/3630187567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f868c105a60 0x7f868c105e40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:52.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.469+0000 7f86921b3700 1 -- 172.21.15.113:0/3630187567 >> 172.21.15.113:0/3630187567 conn(0x7f868c0fbce0 msgr2=0x7f868c0fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:52.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.469+0000 7f86921b3700 1 -- 172.21.15.113:0/3630187567 shutdown_connections 2024-03-28T05:14:52.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.469+0000 7f86921b3700 1 -- 172.21.15.113:0/3630187567 wait complete. 2024-03-28T05:14:52.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.470+0000 7f86921b3700 1 Processor -- start 2024-03-28T05:14:52.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.470+0000 7f86921b3700 1 -- start start 2024-03-28T05:14:52.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.470+0000 7f86921b3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f868c105a60 0x7f868c198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:52.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.470+0000 7f86921b3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868c10b1e0 con 0x7f868c105a60 2024-03-28T05:14:52.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.470+0000 7f868b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f868c105a60 0x7f868c198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:52.470 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.470+0000 7f868b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f868c105a60 0x7f868c198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.113:58756/0 (socket says 172.21.15.113:58756) 2024-03-28T05:14:52.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.470+0000 7f868b7fe700 1 -- 172.21.15.113:0/2935793539 learned_addr learned my addr 172.21.15.113:0/2935793539 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-03-28T05:14:52.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.470+0000 7f868b7fe700 1 -- 172.21.15.113:0/2935793539 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868c198ad0 con 0x7f868c105a60 2024-03-28T05:14:52.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.471+0000 7f868b7fe700 1 --2- 172.21.15.113:0/2935793539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f868c105a60 0x7f868c198590 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f867c005850 tx=0x7f867c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:52.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.471+0000 7f86897fa700 1 -- 172.21.15.113:0/2935793539 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f867c026070 con 0x7f868c105a60 2024-03-28T05:14:52.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.471+0000 7f86921b3700 1 -- 172.21.15.113:0/2935793539 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f868c198d60 con 0x7f868c105a60 2024-03-28T05:14:52.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.471+0000 7f86897fa700 1 -- 172.21.15.113:0/2935793539 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f867c00bc30 con 0x7f868c105a60 2024-03-28T05:14:52.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.471+0000 7f86897fa700 1 -- 172.21.15.113:0/2935793539 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f867c003a20 con 0x7f868c105a60 2024-03-28T05:14:52.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.471+0000 7f86921b3700 1 -- 172.21.15.113:0/2935793539 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f868c1991c0 con 0x7f868c105a60 2024-03-28T05:14:52.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.472+0000 7f86921b3700 1 -- 172.21.15.113:0/2935793539 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f868c102220 con 0x7f868c105a60 2024-03-28T05:14:52.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.472+0000 7f86897fa700 1 -- 172.21.15.113:0/2935793539 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f867c029070 con 0x7f868c105a60 2024-03-28T05:14:52.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.472+0000 7f86897fa700 1 --2- 172.21.15.113:0/2935793539 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8678038fc0 0x7f867803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:14:52.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.472+0000 7f86897fa700 1 -- 172.21.15.113:0/2935793539 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f867c01e070 con 0x7f868c105a60 2024-03-28T05:14:52.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.472+0000 7f8683fff700 1 --2- 172.21.15.113:0/2935793539 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8678038fc0 0x7f867803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:14:52.471 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.473+0000 7f8683fff700 1 --2- 172.21.15.113:0/2935793539 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8678038fc0 0x7f867803b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8674006fd0 tx=0x7f8674006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:14:52.474 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.477+0000 7f86897fa700 1 -- 172.21.15.113:0/2935793539 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f867c013070 con 0x7f868c105a60 2024-03-28T05:14:52.611 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.614+0000 7f86897fa700 1 -- 172.21.15.113:0/2935793539 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f867c00b6f0 con 0x7f868c105a60 2024-03-28T05:14:52.764 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:52.767+0000 7f86921b3700 1 -- 172.21.15.113:0/2935793539 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f868c0637c0 con 0x7f868c105a60 2024-03-28T05:14:52.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:52 smithi023 ceph-mon[31954]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:14:57.662 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:57.665+0000 7f86897fa700 1 -- 172.21.15.113:0/2935793539 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1015 (secure 0 0 0) 0x7f867c030090 con 0x7f868c105a60 2024-03-28T05:14:57.662 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-28T05:14:57.662 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":2,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","modified":"2024-03-28T05:14:52.612579Z","created":"2024-03-28T05:13:10.204871Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-28T05:14:57.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:57.668+0000 7f86921b3700 1 -- 172.21.15.113:0/2935793539 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8678038fc0 msgr2=0x7f867803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:57.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:57.668+0000 7f86921b3700 1 --2- 172.21.15.113:0/2935793539 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8678038fc0 0x7f867803b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8674006fd0 tx=0x7f8674006e40 comp rx=0 tx=0).stop 2024-03-28T05:14:57.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:57.668+0000 7f86921b3700 1 -- 172.21.15.113:0/2935793539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f868c105a60 msgr2=0x7f868c198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:14:57.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:57.668+0000 7f86921b3700 1 --2- 172.21.15.113:0/2935793539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f868c105a60 0x7f868c198590 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f867c005850 tx=0x7f867c005030 comp rx=0 tx=0).stop 2024-03-28T05:14:57.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:57.668+0000 7f86921b3700 1 -- 172.21.15.113:0/2935793539 shutdown_connections 2024-03-28T05:14:57.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:57.668+0000 7f86921b3700 1 --2- 172.21.15.113:0/2935793539 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8678038fc0 0x7f867803b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:57.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:57.668+0000 7f86921b3700 1 --2- 172.21.15.113:0/2935793539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f868c105a60 0x7f868c198590 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:14:57.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:57.668+0000 7f86921b3700 1 -- 172.21.15.113:0/2935793539 >> 172.21.15.113:0/2935793539 conn(0x7f868c0fbce0 msgr2=0x7f868c1067d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:14:57.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:57.668+0000 7f86921b3700 1 -- 172.21.15.113:0/2935793539 shutdown_connections 2024-03-28T05:14:57.665 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:14:57.668+0000 7f86921b3700 1 -- 172.21.15.113:0/2935793539 wait complete. 2024-03-28T05:14:57.666 INFO:teuthology.orchestra.run.smithi113.stderr:dumped monmap epoch 2 2024-03-28T05:14:57.969 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:14:57 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T05:14:58.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T05:14:58.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: mon.a calling monitor election 2024-03-28T05:14:58.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:14:58.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:14:58.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: from='client.? 172.21.15.113:0/2935793539' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T05:14:58.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:14:58.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:14:58.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: mon.c calling monitor election 2024-03-28T05:14:58.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:14:58.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:14:58.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:14:58.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:14:58.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:14:58.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T05:14:58.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: monmap e2: 2 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],c=[v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T05:14:58.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: fsmap 2024-03-28T05:14:58.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T05:14:58.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: mgrmap e13: x(active, since 49s) 2024-03-28T05:14:58.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: overall HEALTH_OK 2024-03-28T05:14:58.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:58.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:58.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:14:58.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:57 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T05:14:58.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:14:58 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:14:58.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:14:58 smithi113 ceph-mon[37164]: Deploying daemon mon.b on smithi098 2024-03-28T05:14:58.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:14:58 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:14:59.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:58 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:14:59.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:58 smithi023 ceph-mon[31954]: Deploying daemon mon.b on smithi098 2024-03-28T05:14:59.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:58 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:14:59.230 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T05:14:59.230 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T05:14:59.468 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T05:14:59.839 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:14:59 smithi113 ceph-mon[37164]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:00.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:14:59 smithi023 ceph-mon[31954]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:00.028 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:14:59 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:14:59.613+0000 7ffb56357700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-28T05:15:01.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.040+0000 7fafc5c32700 1 -- 172.21.15.113:0/2745430314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fafc0100df0 msgr2=0x7fafc01011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:01.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.040+0000 7fafc5c32700 1 --2- 172.21.15.113:0/2745430314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fafc0100df0 0x7fafc01011d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fafb800b0a0 tx=0x7fafb800b3b0 comp rx=0 tx=0).stop 2024-03-28T05:15:01.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.040+0000 7fafc5c32700 1 -- 172.21.15.113:0/2745430314 shutdown_connections 2024-03-28T05:15:01.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.040+0000 7fafc5c32700 1 --2- 172.21.15.113:0/2745430314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fafc0100df0 0x7fafc01011d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:01.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.040+0000 7fafc5c32700 1 -- 172.21.15.113:0/2745430314 >> 172.21.15.113:0/2745430314 conn(0x7fafc00fc6a0 msgr2=0x7fafc00feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:15:01.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.040+0000 7fafc5c32700 1 -- 172.21.15.113:0/2745430314 shutdown_connections 2024-03-28T05:15:01.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.040+0000 7fafc5c32700 1 -- 172.21.15.113:0/2745430314 wait complete. 2024-03-28T05:15:01.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.041+0000 7fafc5c32700 1 Processor -- start 2024-03-28T05:15:01.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.041+0000 7fafc5c32700 1 -- start start 2024-03-28T05:15:01.040 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.041+0000 7fafc5c32700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fafc0100df0 0x7fafc019c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:01.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.041+0000 7fafc5c32700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafc019c990 0x7fafc0196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:01.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.041+0000 7fafc5c32700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc010b1c0 con 0x7fafc0100df0 2024-03-28T05:15:01.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.041+0000 7fafc5c32700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc010b040 con 0x7fafc019c990 2024-03-28T05:15:01.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.042+0000 7fafbeffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafc019c990 0x7fafc0196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:01.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.042+0000 7fafbeffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafc019c990 0x7fafc0196610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:51538/0 (socket says 172.21.15.113:51538) 2024-03-28T05:15:01.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.042+0000 7fafbeffd700 1 -- 172.21.15.113:0/3617960164 learned_addr learned my addr 172.21.15.113:0/3617960164 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-03-28T05:15:01.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.042+0000 7fafbeffd700 1 -- 172.21.15.113:0/3617960164 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafc019c990 msgr2=0x7fafc0196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-28T05:15:01.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.042+0000 7fafbeffd700 1 -- 172.21.15.113:0/3617960164 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafc019c990 msgr2=0x7fafc0196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T05:15:01.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.042+0000 7fafbeffd700 1 --2- 172.21.15.113:0/3617960164 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafc019c990 0x7fafc0196610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T05:15:01.045 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.042+0000 7fafbeffd700 1 --2- 172.21.15.113:0/3617960164 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafc019c990 0x7fafc0196610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T05:15:01.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.042+0000 7fafbf7fe700 1 --2- 172.21.15.113:0/3617960164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fafc0100df0 0x7fafc019c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:01.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.042+0000 7fafbf7fe700 1 -- 172.21.15.113:0/3617960164 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafc019c990 msgr2=0x7fafc0196610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T05:15:01.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.042+0000 7fafbf7fe700 1 --2- 172.21.15.113:0/3617960164 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafc019c990 0x7fafc0196610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:01.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.042+0000 7fafbf7fe700 1 -- 172.21.15.113:0/3617960164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc019cfd0 con 0x7fafc0100df0 2024-03-28T05:15:01.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.042+0000 7fafbf7fe700 1 --2- 172.21.15.113:0/3617960164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fafc0100df0 0x7fafc019c450 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fafb80062a0 tx=0x7fafb8007930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:15:01.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.043+0000 7fafbcff9700 1 -- 172.21.15.113:0/3617960164 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fafb8007bd0 con 0x7fafc0100df0 2024-03-28T05:15:01.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.043+0000 7fafbcff9700 1 -- 172.21.15.113:0/3617960164 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafb801d920 con 0x7fafc0100df0 2024-03-28T05:15:01.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.043+0000 7fafc5c32700 1 -- 172.21.15.113:0/3617960164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc0196b50 con 0x7fafc0100df0 2024-03-28T05:15:01.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.043+0000 7fafbcff9700 1 -- 172.21.15.113:0/3617960164 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fafb8012a70 con 0x7fafc0100df0 2024-03-28T05:15:01.046 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.043+0000 7fafc5c32700 1 -- 172.21.15.113:0/3617960164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc01970e0 con 0x7fafc0100df0 2024-03-28T05:15:01.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.044+0000 7fafc5c32700 1 -- 172.21.15.113:0/3617960164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc01022d0 con 0x7fafc0100df0 2024-03-28T05:15:01.047 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.044+0000 7fafbcff9700 1 -- 172.21.15.113:0/3617960164 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fafb8024020 con 0x7fafc0100df0 2024-03-28T05:15:01.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.044+0000 7fafbcff9700 1 --2- 172.21.15.113:0/3617960164 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fafa8038d60 0x7fafa803b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:01.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.044+0000 7fafbcff9700 1 -- 172.21.15.113:0/3617960164 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fafb8022020 con 0x7fafc0100df0 2024-03-28T05:15:01.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.044+0000 7fafbeffd700 1 --2- 172.21.15.113:0/3617960164 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fafa8038d60 0x7fafa803b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:01.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.044+0000 7fafbeffd700 1 --2- 172.21.15.113:0/3617960164 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fafa8038d60 0x7fafa803b220 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fafb4009180 tx=0x7fafb4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:15:01.048 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.049+0000 7fafbcff9700 1 -- 172.21.15.113:0/3617960164 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fafb8011bd0 con 0x7fafc0100df0 2024-03-28T05:15:01.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.322+0000 7fafc5c32700 1 -- 172.21.15.113:0/3617960164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fafc0000900 con 0x7fafc0100df0 2024-03-28T05:15:01.319 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.323+0000 7fafbcff9700 1 -- 172.21.15.113:0/3617960164 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1015 (secure 0 0 0) 0x7fafb8025370 con 0x7fafc0100df0 2024-03-28T05:15:01.320 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-28T05:15:01.320 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":2,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","modified":"2024-03-28T05:14:52.612579Z","created":"2024-03-28T05:13:10.204871Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-28T05:15:01.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.325+0000 7fafc5c32700 1 -- 172.21.15.113:0/3617960164 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fafa8038d60 msgr2=0x7fafa803b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:01.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.325+0000 7fafc5c32700 1 --2- 172.21.15.113:0/3617960164 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fafa8038d60 0x7fafa803b220 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fafb4009180 tx=0x7fafb4008040 comp rx=0 tx=0).stop 2024-03-28T05:15:01.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.325+0000 7fafc5c32700 1 -- 172.21.15.113:0/3617960164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fafc0100df0 msgr2=0x7fafc019c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:01.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.325+0000 7fafc5c32700 1 --2- 172.21.15.113:0/3617960164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fafc0100df0 0x7fafc019c450 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fafb80062a0 tx=0x7fafb8007930 comp rx=0 tx=0).stop 2024-03-28T05:15:01.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.325+0000 7fafc5c32700 1 -- 172.21.15.113:0/3617960164 shutdown_connections 2024-03-28T05:15:01.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.325+0000 7fafc5c32700 1 --2- 172.21.15.113:0/3617960164 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fafa8038d60 0x7fafa803b220 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:01.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.325+0000 7fafc5c32700 1 --2- 172.21.15.113:0/3617960164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fafc0100df0 0x7fafc019c450 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:01.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.325+0000 7fafc5c32700 1 --2- 172.21.15.113:0/3617960164 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fafc019c990 0x7fafc0196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:01.322 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.325+0000 7fafc5c32700 1 -- 172.21.15.113:0/3617960164 >> 172.21.15.113:0/3617960164 conn(0x7fafc00fc6a0 msgr2=0x7fafc01027c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:15:01.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.325+0000 7fafc5c32700 1 -- 172.21.15.113:0/3617960164 shutdown_connections 2024-03-28T05:15:01.323 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:01.325+0000 7fafc5c32700 1 -- 172.21.15.113:0/3617960164 wait complete. 2024-03-28T05:15:01.323 INFO:teuthology.orchestra.run.smithi113.stderr:dumped monmap epoch 2 2024-03-28T05:15:01.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: mon.b@-1(synchronizing).mds e1 new map 2024-03-28T05:15:01.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-28T05:15:01.500 INFO:journalctl@ceph.mon.b.smithi098.stdout: e1 2024-03-28T05:15:01.501 INFO:journalctl@ceph.mon.b.smithi098.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T05:15:01.501 INFO:journalctl@ceph.mon.b.smithi098.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T05:15:01.501 INFO:journalctl@ceph.mon.b.smithi098.stdout: legacy client fscid: -1 2024-03-28T05:15:01.501 INFO:journalctl@ceph.mon.b.smithi098.stdout: 2024-03-28T05:15:01.501 INFO:journalctl@ceph.mon.b.smithi098.stdout: No filesystems configured 2024-03-28T05:15:01.501 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-28T05:15:01.501 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-28T05:15:01.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-28T05:15:01.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-28T05:15:01.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-28T05:15:01.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-28T05:15:01.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-28T05:15:01.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T05:15:01.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T05:15:01.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T05:15:01.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-28T05:15:01.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: Deploying cephadm binary to smithi113 2024-03-28T05:15:01.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: Updating smithi098:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:15:01.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-28T05:15:01.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: Updating smithi098:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T05:15:01.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:01.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:01.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:01.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:01.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:01.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:01.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:15:01.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: Added host smithi113 2024-03-28T05:15:01.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:01.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:01.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:01.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T05:15:01.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:01.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:01.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3726618071' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-28T05:15:01.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:01.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3726618071' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-28T05:15:01.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T05:15:01.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:01.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:01.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:01.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:01.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi113", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:15:01.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:01.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:15:01.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: Updating smithi113:/etc/ceph/ceph.conf 2024-03-28T05:15:01.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: Updating smithi113:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:15:01.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-03-28T05:15:01.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: Updating smithi113:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T05:15:01.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:01.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:01.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:01.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:01.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:01.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:15:01.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:01.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:15:01.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:01.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T05:15:01.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:01.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi023:172.21.15.23=a;smithi098:172.21.15.98=b;smithi113:172.21.15.113=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:15:01.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: Saving service mon spec with placement smithi023:172.21.15.23=a;smithi098:172.21.15.98=b;smithi113:172.21.15.113=c;count:3 2024-03-28T05:15:01.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: Deploying daemon mon.c on smithi113 2024-03-28T05:15:01.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:01.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:01.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T05:15:01.868 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: mon.a calling monitor election 2024-03-28T05:15:01.868 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:15:01.868 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:01.868 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='client.? 172.21.15.113:0/2935793539' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T05:15:01.868 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:15:01.868 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:15:01.868 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: mon.c calling monitor election 2024-03-28T05:15:01.868 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:01.868 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:15:01.868 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:15:01.868 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:01.868 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:15:01.869 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T05:15:01.869 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: monmap e2: 2 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],c=[v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T05:15:01.869 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: fsmap 2024-03-28T05:15:01.869 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T05:15:01.869 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: mgrmap e13: x(active, since 49s) 2024-03-28T05:15:01.869 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: overall HEALTH_OK 2024-03-28T05:15:01.869 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:01.869 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:01.869 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:01.869 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T05:15:01.869 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:01.869 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: Deploying daemon mon.b on smithi098 2024-03-28T05:15:01.869 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:15:01.869 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:01.869 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 ceph-mon[36087]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-28T05:15:01.869 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 bash[36069]: 0b0ed0d41fb2ad098a5567d430dce3fa8aa76917852a763888579c5bf36e7f41 2024-03-28T05:15:01.869 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:01 smithi098 systemd[1]: Started Ceph mon.b for bf37ebc4-ecc1-11ee-95d0-87774f69a715. 2024-03-28T05:15:02.918 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T05:15:02.919 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T05:15:03.161 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T05:15:06.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:06.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T05:15:06.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:06.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: mon.a calling monitor election 2024-03-28T05:15:06.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: mon.c calling monitor election 2024-03-28T05:15:06.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:15:06.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:06.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:06.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:06.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:06.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:06.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:06.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:06.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T05:15:06.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T05:15:06.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: fsmap 2024-03-28T05:15:06.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T05:15:06.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: mgrmap e13: x(active, since 57s) 2024-03-28T05:15:06.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-28T05:15:06.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-28T05:15:06.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-28T05:15:06.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: mon.b (rank 2) addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] is down (out of quorum) 2024-03-28T05:15:06.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:06.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:07.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:07.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T05:15:07.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:07.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: mon.a calling monitor election 2024-03-28T05:15:07.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: mon.c calling monitor election 2024-03-28T05:15:07.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:15:07.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:07.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:07.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:07.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:07.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:07.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:07.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:07.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T05:15:07.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T05:15:07.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: fsmap 2024-03-28T05:15:07.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T05:15:07.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: mgrmap e13: x(active, since 57s) 2024-03-28T05:15:07.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-28T05:15:07.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-28T05:15:07.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-28T05:15:07.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: mon.b (rank 2) addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] is down (out of quorum) 2024-03-28T05:15:07.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:07.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:07.690 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.693+0000 7fd0fb2ee700 1 -- 172.21.15.113:0/392931800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0e00035c0 msgr2=0x7fd0e0005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:07.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.693+0000 7fd0fb2ee700 1 --2- 172.21.15.113:0/392931800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0e00035c0 0x7fd0e0005a50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4101b90 tx=0x7fd0f000feb0 comp rx=0 tx=0).stop 2024-03-28T05:15:07.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.693+0000 7fd0fb2ee700 1 -- 172.21.15.113:0/392931800 shutdown_connections 2024-03-28T05:15:07.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.693+0000 7fd0fb2ee700 1 --2- 172.21.15.113:0/392931800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0e00035c0 0x7fd0e0005a50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:07.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.693+0000 7fd0fb2ee700 1 -- 172.21.15.113:0/392931800 >> 172.21.15.113:0/392931800 conn(0x7fd0f40fd730 msgr2=0x7fd0f40ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:15:07.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.693+0000 7fd0fb2ee700 1 -- 172.21.15.113:0/392931800 shutdown_connections 2024-03-28T05:15:07.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.694+0000 7fd0fb2ee700 1 -- 172.21.15.113:0/392931800 wait complete. 2024-03-28T05:15:07.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.694+0000 7fd0fb2ee700 1 Processor -- start 2024-03-28T05:15:07.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.694+0000 7fd0fb2ee700 1 -- start start 2024-03-28T05:15:07.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.695+0000 7fd0fb2ee700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0e00035c0 0x7fd0f419d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:07.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.695+0000 7fd0fb2ee700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0f419db90 0x7fd0f4197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:07.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.695+0000 7fd0fb2ee700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0f4197c60 0x7fd0f4198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:07.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.695+0000 7fd0fb2ee700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f4109ca0 con 0x7fd0e00035c0 2024-03-28T05:15:07.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.695+0000 7fd0fb2ee700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f4109b20 con 0x7fd0f4197c60 2024-03-28T05:15:07.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.695+0000 7fd0fb2ee700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f4109e20 con 0x7fd0f419db90 2024-03-28T05:15:07.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.695+0000 7fd0f8889700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0f419db90 0x7fd0f4197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:07.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.695+0000 7fd0f8889700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0f419db90 0x7fd0f4197720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:51544/0 (socket says 172.21.15.113:51544) 2024-03-28T05:15:07.694 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.695+0000 7fd0f8889700 1 -- 172.21.15.113:0/1254278712 learned_addr learned my addr 172.21.15.113:0/1254278712 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-03-28T05:15:07.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.695+0000 7fd0f908a700 1 --2- 172.21.15.113:0/1254278712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0e00035c0 0x7fd0f419d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:07.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.695+0000 7fd0f988b700 1 --2- 172.21.15.113:0/1254278712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0f4197c60 0x7fd0f4198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:07.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.695+0000 7fd0f8889700 1 -- 172.21.15.113:0/1254278712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0f4197c60 msgr2=0x7fd0f4198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:07.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.695+0000 7fd0f8889700 1 --2- 172.21.15.113:0/1254278712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0f4197c60 0x7fd0f4198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:07.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.695+0000 7fd0f8889700 1 -- 172.21.15.113:0/1254278712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0e00035c0 msgr2=0x7fd0f419d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:07.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.695+0000 7fd0f8889700 1 --2- 172.21.15.113:0/1254278712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0e00035c0 0x7fd0f419d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:07.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.695+0000 7fd0f8889700 1 -- 172.21.15.113:0/1254278712 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f41a4400 con 0x7fd0f419db90 2024-03-28T05:15:07.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.695+0000 7fd0f8889700 1 --2- 172.21.15.113:0/1254278712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0f419db90 0x7fd0f4197720 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400b780 tx=0x7fd0e400ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:15:07.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.695+0000 7fd0ea7fc700 1 -- 172.21.15.113:0/1254278712 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4004230 con 0x7fd0f419db90 2024-03-28T05:15:07.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.696+0000 7fd0ea7fc700 1 -- 172.21.15.113:0/1254278712 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd0e40043d0 con 0x7fd0f419db90 2024-03-28T05:15:07.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.696+0000 7fd0ea7fc700 1 -- 172.21.15.113:0/1254278712 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4011c90 con 0x7fd0f419db90 2024-03-28T05:15:07.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.696+0000 7fd0fb2ee700 1 -- 172.21.15.113:0/1254278712 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0f41a4690 con 0x7fd0f419db90 2024-03-28T05:15:07.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.696+0000 7fd0fb2ee700 1 -- 172.21.15.113:0/1254278712 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f41a4c20 con 0x7fd0f419db90 2024-03-28T05:15:07.695 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.696+0000 7fd0ea7fc700 1 -- 172.21.15.113:0/1254278712 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fd0e401a400 con 0x7fd0f419db90 2024-03-28T05:15:07.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.697+0000 7fd0fb2ee700 1 -- 172.21.15.113:0/1254278712 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0f41033a0 con 0x7fd0f419db90 2024-03-28T05:15:07.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.697+0000 7fd0ea7fc700 1 --2- 172.21.15.113:0/1254278712 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd0d8039450 0x7fd0d803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:07.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.697+0000 7fd0ea7fc700 1 -- 172.21.15.113:0/1254278712 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd0e404bcf0 con 0x7fd0f419db90 2024-03-28T05:15:07.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.697+0000 7fd0f908a700 1 --2- 172.21.15.113:0/1254278712 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd0d8039450 0x7fd0d803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:07.696 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.697+0000 7fd0f908a700 1 --2- 172.21.15.113:0/1254278712 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd0d8039450 0x7fd0d803b910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd0f000f8d0 tx=0x7fd0f0005e60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:15:07.698 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.702+0000 7fd0ea7fc700 1 -- 172.21.15.113:0/1254278712 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd0e4018350 con 0x7fd0f419db90 2024-03-28T05:15:07.958 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:07 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:07.959 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:07 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:07.959 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:07 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:15:07.959 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:07 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:07.981 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.984+0000 7fd0fb2ee700 1 -- 172.21.15.113:0/1254278712 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0f41a4f10 con 0x7fd0f419db90 2024-03-28T05:15:07.981 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:07 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:07.981 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:07 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:07.981 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:07 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:15:07.981 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:07 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:07.982 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.984+0000 7fd0ea7fc700 1 -- 172.21.15.113:0/1254278712 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1279 (secure 0 0 0) 0x7fd0e400fd40 con 0x7fd0f419db90 2024-03-28T05:15:07.982 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-28T05:15:07.982 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":3,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","modified":"2024-03-28T05:15:01.509967Z","created":"2024-03-28T05:13:10.204871Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-28T05:15:07.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.987+0000 7fd0fb2ee700 1 -- 172.21.15.113:0/1254278712 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd0d8039450 msgr2=0x7fd0d803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:07.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.987+0000 7fd0fb2ee700 1 --2- 172.21.15.113:0/1254278712 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd0d8039450 0x7fd0d803b910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd0f000f8d0 tx=0x7fd0f0005e60 comp rx=0 tx=0).stop 2024-03-28T05:15:07.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.987+0000 7fd0fb2ee700 1 -- 172.21.15.113:0/1254278712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0f419db90 msgr2=0x7fd0f4197720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:07.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.987+0000 7fd0fb2ee700 1 --2- 172.21.15.113:0/1254278712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0f419db90 0x7fd0f4197720 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400b780 tx=0x7fd0e400ba90 comp rx=0 tx=0).stop 2024-03-28T05:15:07.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.987+0000 7fd0fb2ee700 1 -- 172.21.15.113:0/1254278712 shutdown_connections 2024-03-28T05:15:07.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.987+0000 7fd0fb2ee700 1 --2- 172.21.15.113:0/1254278712 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd0d8039450 0x7fd0d803b910 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:07.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.987+0000 7fd0fb2ee700 1 --2- 172.21.15.113:0/1254278712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0f4197c60 0x7fd0f4198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:07.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.987+0000 7fd0fb2ee700 1 --2- 172.21.15.113:0/1254278712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0e00035c0 0x7fd0f419d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:07.984 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.987+0000 7fd0fb2ee700 1 --2- 172.21.15.113:0/1254278712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0f419db90 0x7fd0f4197720 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:07.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.987+0000 7fd0fb2ee700 1 -- 172.21.15.113:0/1254278712 >> 172.21.15.113:0/1254278712 conn(0x7fd0f40fd730 msgr2=0x7fd0f41058a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:15:07.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.987+0000 7fd0fb2ee700 1 -- 172.21.15.113:0/1254278712 shutdown_connections 2024-03-28T05:15:07.985 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:07.987+0000 7fd0fb2ee700 1 -- 172.21.15.113:0/1254278712 wait complete. 2024-03-28T05:15:07.985 INFO:teuthology.orchestra.run.smithi113.stderr:dumped monmap epoch 3 2024-03-28T05:15:08.670 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-28T05:15:08.670 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-28T05:15:08.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:08.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T05:15:08.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:08.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: mon.a calling monitor election 2024-03-28T05:15:08.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: mon.c calling monitor election 2024-03-28T05:15:08.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:15:08.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:08.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:08.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:08.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:08.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:08.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:08.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:08.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T05:15:08.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T05:15:08.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: fsmap 2024-03-28T05:15:08.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T05:15:08.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: mgrmap e13: x(active, since 57s) 2024-03-28T05:15:08.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-28T05:15:08.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-28T05:15:08.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-28T05:15:08.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: mon.b (rank 2) addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] is down (out of quorum) 2024-03-28T05:15:08.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:08.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:08.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:08.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:08.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:15:08.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:08.925 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:15:09.822 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: mon.b calling monitor election 2024-03-28T05:15:09.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-28T05:15:09.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-28T05:15:09.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: Updating smithi113:/etc/ceph/ceph.conf 2024-03-28T05:15:09.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: Updating smithi023:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:15:09.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: Updating smithi098:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:15:09.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: Updating smithi113:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:15:09.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: mon.b calling monitor election 2024-03-28T05:15:09.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: mon.c calling monitor election 2024-03-28T05:15:09.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: mon.a calling monitor election 2024-03-28T05:15:09.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T05:15:09.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T05:15:09.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: fsmap 2024-03-28T05:15:09.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T05:15:09.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: mgrmap e13: x(active, since 59s) 2024-03-28T05:15:09.824 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-28T05:15:09.824 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: Cluster is now healthy 2024-03-28T05:15:09.824 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: overall HEALTH_OK 2024-03-28T05:15:09.824 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.824 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.824 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.824 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.824 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.824 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.824 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.824 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.824 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.824 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.824 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.824 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T05:15:09.824 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T05:15:09.825 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:09.825 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:09.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: mon.b calling monitor election 2024-03-28T05:15:09.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-28T05:15:09.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-28T05:15:09.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: Updating smithi113:/etc/ceph/ceph.conf 2024-03-28T05:15:09.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: Updating smithi023:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:15:09.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: Updating smithi098:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:15:09.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: Updating smithi113:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:15:09.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: mon.b calling monitor election 2024-03-28T05:15:09.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: mon.c calling monitor election 2024-03-28T05:15:09.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: mon.a calling monitor election 2024-03-28T05:15:09.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T05:15:09.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T05:15:09.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: fsmap 2024-03-28T05:15:09.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T05:15:09.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: mgrmap e13: x(active, since 59s) 2024-03-28T05:15:09.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-28T05:15:09.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: Cluster is now healthy 2024-03-28T05:15:09.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: overall HEALTH_OK 2024-03-28T05:15:09.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T05:15:09.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T05:15:09.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:09.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:09.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: mon.b calling monitor election 2024-03-28T05:15:09.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-28T05:15:09.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-28T05:15:09.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: Updating smithi113:/etc/ceph/ceph.conf 2024-03-28T05:15:09.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: Updating smithi023:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:15:09.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: Updating smithi098:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:15:09.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: Updating smithi113:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:15:09.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: mon.b calling monitor election 2024-03-28T05:15:09.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: mon.c calling monitor election 2024-03-28T05:15:09.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: mon.a calling monitor election 2024-03-28T05:15:09.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T05:15:09.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T05:15:09.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: fsmap 2024-03-28T05:15:09.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T05:15:09.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: mgrmap e13: x(active, since 59s) 2024-03-28T05:15:09.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-28T05:15:09.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: Cluster is now healthy 2024-03-28T05:15:09.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: overall HEALTH_OK 2024-03-28T05:15:09.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:09.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T05:15:09.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T05:15:09.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:09.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:15:10.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.458+0000 7fa5c558d700 1 -- 172.21.15.23:0/4118862919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5c0106a60 msgr2=0x7fa5c0106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:10.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.458+0000 7fa5c558d700 1 --2- 172.21.15.23:0/4118862919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5c0106a60 0x7fa5c0106e40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000b0a0 tx=0x7fa5b000b3b0 comp rx=0 tx=0).stop 2024-03-28T05:15:10.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.458+0000 7fa5c558d700 1 -- 172.21.15.23:0/4118862919 shutdown_connections 2024-03-28T05:15:10.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.458+0000 7fa5c558d700 1 --2- 172.21.15.23:0/4118862919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5c0106a60 0x7fa5c0106e40 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:10.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.458+0000 7fa5c558d700 1 -- 172.21.15.23:0/4118862919 >> 172.21.15.23:0/4118862919 conn(0x7fa5c00fc530 msgr2=0x7fa5c00fe950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:15:10.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.458+0000 7fa5c558d700 1 -- 172.21.15.23:0/4118862919 shutdown_connections 2024-03-28T05:15:10.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.458+0000 7fa5c558d700 1 -- 172.21.15.23:0/4118862919 wait complete. 2024-03-28T05:15:10.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.459+0000 7fa5c558d700 1 Processor -- start 2024-03-28T05:15:10.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.459+0000 7fa5c558d700 1 -- start start 2024-03-28T05:15:10.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.459+0000 7fa5c558d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c0106a60 0x7fa5c0198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:10.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.459+0000 7fa5c558d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c0198c70 0x7fa5c01990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:10.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.459+0000 7fa5c558d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5c01a17a0 0x7fa5c01a3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:10.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.459+0000 7fa5c558d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c010b050 con 0x7fa5c01a17a0 2024-03-28T05:15:10.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.459+0000 7fa5c558d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c010aed0 con 0x7fa5c0106a60 2024-03-28T05:15:10.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.459+0000 7fa5c558d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c010b1d0 con 0x7fa5c0198c70 2024-03-28T05:15:10.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.459+0000 7fa5bf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5c01a17a0 0x7fa5c01a3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:10.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.459+0000 7fa5bf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5c01a17a0 0x7fa5c01a3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51110/0 (socket says 172.21.15.23:51110) 2024-03-28T05:15:10.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.459+0000 7fa5bf7fe700 1 -- 172.21.15.23:0/1929481610 learned_addr learned my addr 172.21.15.23:0/1929481610 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:15:10.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.459+0000 7fa5beffd700 1 --2- 172.21.15.23:0/1929481610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c0106a60 0x7fa5c0198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:10.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.460+0000 7fa5be7fc700 1 --2- 172.21.15.23:0/1929481610 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c0198c70 0x7fa5c01990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:10.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.460+0000 7fa5bf7fe700 1 -- 172.21.15.23:0/1929481610 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c0198c70 msgr2=0x7fa5c01990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:10.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.460+0000 7fa5bf7fe700 1 --2- 172.21.15.23:0/1929481610 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c0198c70 0x7fa5c01990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:10.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.460+0000 7fa5bf7fe700 1 -- 172.21.15.23:0/1929481610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c0106a60 msgr2=0x7fa5c0198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:10.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.460+0000 7fa5bf7fe700 1 --2- 172.21.15.23:0/1929481610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c0106a60 0x7fa5c0198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:10.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.460+0000 7fa5bf7fe700 1 -- 172.21.15.23:0/1929481610 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c01a4400 con 0x7fa5c01a17a0 2024-03-28T05:15:10.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.460+0000 7fa5bf7fe700 1 --2- 172.21.15.23:0/1929481610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5c01a17a0 0x7fa5c01a3b90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800b550 tx=0x7fa5b800b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:15:10.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.460+0000 7fa5b7fff700 1 -- 172.21.15.23:0/1929481610 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8015020 con 0x7fa5c01a17a0 2024-03-28T05:15:10.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.460+0000 7fa5b7fff700 1 -- 172.21.15.23:0/1929481610 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5b8011d10 con 0x7fa5c01a17a0 2024-03-28T05:15:10.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.460+0000 7fa5b7fff700 1 -- 172.21.15.23:0/1929481610 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b800f2e0 con 0x7fa5c01a17a0 2024-03-28T05:15:10.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.460+0000 7fa5c558d700 1 -- 172.21.15.23:0/1929481610 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c01a46f0 con 0x7fa5c01a17a0 2024-03-28T05:15:10.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.460+0000 7fa5c558d700 1 -- 172.21.15.23:0/1929481610 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c01a4be0 con 0x7fa5c01a17a0 2024-03-28T05:15:10.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.461+0000 7fa5b7fff700 1 -- 172.21.15.23:0/1929481610 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fa5b8018020 con 0x7fa5c01a17a0 2024-03-28T05:15:10.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.461+0000 7fa5c558d700 1 -- 172.21.15.23:0/1929481610 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c00637c0 con 0x7fa5c01a17a0 2024-03-28T05:15:10.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.461+0000 7fa5b7fff700 1 --2- 172.21.15.23:0/1929481610 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa5ac039450 0x7fa5ac03b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:10.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.461+0000 7fa5b7fff700 1 -- 172.21.15.23:0/1929481610 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa5b804b8d0 con 0x7fa5c01a17a0 2024-03-28T05:15:10.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.461+0000 7fa5beffd700 1 --2- 172.21.15.23:0/1929481610 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa5ac039450 0x7fa5ac03b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:10.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.462+0000 7fa5beffd700 1 --2- 172.21.15.23:0/1929481610 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa5ac039450 0x7fa5ac03b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa5b00062a0 tx=0x7fa5b0005040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:15:10.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.467+0000 7fa5b7fff700 1 -- 172.21.15.23:0/1929481610 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa5b801f020 con 0x7fa5c01a17a0 2024-03-28T05:15:10.613 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:10 smithi098 ceph-mon[36087]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:10.613 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:10 smithi098 ceph-mon[36087]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T05:15:10.613 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:10 smithi098 ceph-mon[36087]: Reconfiguring daemon mon.a on smithi023 2024-03-28T05:15:10.644 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:10 smithi023 ceph-mon[31954]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:10.644 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:10 smithi023 ceph-mon[31954]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T05:15:10.644 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:10 smithi023 ceph-mon[31954]: Reconfiguring daemon mon.a on smithi023 2024-03-28T05:15:10.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.677+0000 7fa5c558d700 1 -- 172.21.15.23:0/1929481610 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fa5c01a4f40 con 0x7fa5c01a17a0 2024-03-28T05:15:10.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.677+0000 7fa5b7fff700 1 -- 172.21.15.23:0/1929481610 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+273 (secure 0 0 0) 0x7fa5b8021de0 con 0x7fa5c01a17a0 2024-03-28T05:15:10.673 INFO:teuthology.orchestra.run.smithi023.stdout:# minimal ceph.conf for bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:15:10.674 INFO:teuthology.orchestra.run.smithi023.stdout:[global] 2024-03-28T05:15:10.674 INFO:teuthology.orchestra.run.smithi023.stdout: fsid = bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:15:10.674 INFO:teuthology.orchestra.run.smithi023.stdout: mon_host = [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] 2024-03-28T05:15:10.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.679+0000 7fa5c558d700 1 -- 172.21.15.23:0/1929481610 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa5ac039450 msgr2=0x7fa5ac03b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:10.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.679+0000 7fa5c558d700 1 --2- 172.21.15.23:0/1929481610 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa5ac039450 0x7fa5ac03b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa5b00062a0 tx=0x7fa5b0005040 comp rx=0 tx=0).stop 2024-03-28T05:15:10.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.679+0000 7fa5c558d700 1 -- 172.21.15.23:0/1929481610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5c01a17a0 msgr2=0x7fa5c01a3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:10.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.679+0000 7fa5c558d700 1 --2- 172.21.15.23:0/1929481610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5c01a17a0 0x7fa5c01a3b90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800b550 tx=0x7fa5b800b860 comp rx=0 tx=0).stop 2024-03-28T05:15:10.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.679+0000 7fa5c558d700 1 -- 172.21.15.23:0/1929481610 shutdown_connections 2024-03-28T05:15:10.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.679+0000 7fa5c558d700 1 --2- 172.21.15.23:0/1929481610 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa5ac039450 0x7fa5ac03b910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:10.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.679+0000 7fa5c558d700 1 --2- 172.21.15.23:0/1929481610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5c01a17a0 0x7fa5c01a3b90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:10.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.679+0000 7fa5c558d700 1 --2- 172.21.15.23:0/1929481610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa5c0106a60 0x7fa5c0198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:10.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.679+0000 7fa5c558d700 1 --2- 172.21.15.23:0/1929481610 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa5c0198c70 0x7fa5c01990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:10.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.679+0000 7fa5c558d700 1 -- 172.21.15.23:0/1929481610 >> 172.21.15.23:0/1929481610 conn(0x7fa5c00fc530 msgr2=0x7fa5c0102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:15:10.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.679+0000 7fa5c558d700 1 -- 172.21.15.23:0/1929481610 shutdown_connections 2024-03-28T05:15:10.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:10.679+0000 7fa5c558d700 1 -- 172.21.15.23:0/1929481610 wait complete. 2024-03-28T05:15:10.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:10 smithi113 ceph-mon[37164]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:10.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:10 smithi113 ceph-mon[37164]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T05:15:10.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:10 smithi113 ceph-mon[37164]: Reconfiguring daemon mon.a on smithi023 2024-03-28T05:15:11.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:11 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1929481610' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:11.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:11 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:11.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:11 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:11.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:11 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T05:15:11.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:11 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T05:15:11.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:11 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:11.828 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-28T05:15:11.828 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T05:15:11.829 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T05:15:11.858 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T05:15:11.858 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T05:15:11.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:11 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1929481610' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:11.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:11 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:11.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:11 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:11.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:11 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T05:15:11.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:11 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T05:15:11.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:11 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:11.927 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-28T05:15:11.927 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T05:15:11.957 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-28T05:15:11.957 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T05:15:11.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:11 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1929481610' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:11.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:11 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:11.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:11 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:11.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:11 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T05:15:11.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:11 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T05:15:11.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:11 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:12.024 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-28T05:15:12.025 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T05:15:12.059 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-28T05:15:12.059 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T05:15:12.128 INFO:tasks.cephadm:Adding mgr.x on smithi023 2024-03-28T05:15:12.128 INFO:tasks.cephadm:Adding mgr.y on smithi098 2024-03-28T05:15:12.128 INFO:tasks.cephadm:Adding mgr.z on smithi113 2024-03-28T05:15:12.129 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph orch apply mgr '3;smithi023=x;smithi098=y;smithi113=z' 2024-03-28T05:15:12.404 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T05:15:12.828 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:12 smithi113 ceph-mon[37164]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:12.828 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:12 smithi113 ceph-mon[37164]: Reconfiguring mon.b (monmap changed)... 2024-03-28T05:15:12.828 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:12 smithi113 ceph-mon[37164]: Reconfiguring daemon mon.b on smithi098 2024-03-28T05:15:12.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:12 smithi098 ceph-mon[36087]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:12.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:12 smithi098 ceph-mon[36087]: Reconfiguring mon.b (monmap changed)... 2024-03-28T05:15:12.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:12 smithi098 ceph-mon[36087]: Reconfiguring daemon mon.b on smithi098 2024-03-28T05:15:13.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:12 smithi023 ceph-mon[31954]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:13.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:12 smithi023 ceph-mon[31954]: Reconfiguring mon.b (monmap changed)... 2024-03-28T05:15:13.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:12 smithi023 ceph-mon[31954]: Reconfiguring daemon mon.b on smithi098 2024-03-28T05:15:14.472 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:14 smithi113 ceph-mon[37164]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:14.472 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:14 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:14.472 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:14 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:14.472 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:14 smithi113 ceph-mon[37164]: Reconfiguring mon.c (monmap changed)... 2024-03-28T05:15:14.473 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:14 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T05:15:14.473 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:14 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T05:15:14.473 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:14 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:14.473 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:14 smithi113 ceph-mon[37164]: Reconfiguring daemon mon.c on smithi113 2024-03-28T05:15:14.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:14 smithi023 ceph-mon[31954]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:14.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:14 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:14.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:14 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:14.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:14 smithi023 ceph-mon[31954]: Reconfiguring mon.c (monmap changed)... 2024-03-28T05:15:14.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:14 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T05:15:14.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:14 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T05:15:14.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:14 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:14.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:14 smithi023 ceph-mon[31954]: Reconfiguring daemon mon.c on smithi113 2024-03-28T05:15:14.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.577+0000 7fa8d41a3700 1 -- 172.21.15.113:0/758158672 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8cc1033e0 msgr2=0x7fa8cc103800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:14.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.577+0000 7fa8d41a3700 1 --2- 172.21.15.113:0/758158672 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8cc1033e0 0x7fa8cc103800 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8009910 tx=0x7fa8c8009c20 comp rx=0 tx=0).stop 2024-03-28T05:15:14.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.577+0000 7fa8d41a3700 1 -- 172.21.15.113:0/758158672 shutdown_connections 2024-03-28T05:15:14.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.577+0000 7fa8d41a3700 1 --2- 172.21.15.113:0/758158672 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8cc1033e0 0x7fa8cc103800 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:14.576 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.577+0000 7fa8d41a3700 1 -- 172.21.15.113:0/758158672 >> 172.21.15.113:0/758158672 conn(0x7fa8cc0fd280 msgr2=0x7fa8cc0ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:15:14.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.577+0000 7fa8d41a3700 1 -- 172.21.15.113:0/758158672 shutdown_connections 2024-03-28T05:15:14.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.577+0000 7fa8d41a3700 1 -- 172.21.15.113:0/758158672 wait complete. 2024-03-28T05:15:14.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.578+0000 7fa8d41a3700 1 Processor -- start 2024-03-28T05:15:14.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.578+0000 7fa8d41a3700 1 -- start start 2024-03-28T05:15:14.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.578+0000 7fa8d41a3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8cc1033e0 0x7fa8cc1975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:14.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.578+0000 7fa8d41a3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8cc197b30 0x7fa8cc19cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:14.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.578+0000 7fa8d41a3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8cc197fb0 0x7fa8cc19f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:14.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.578+0000 7fa8d41a3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc107e50 con 0x7fa8cc1033e0 2024-03-28T05:15:14.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.578+0000 7fa8d41a3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc107cd0 con 0x7fa8cc197fb0 2024-03-28T05:15:14.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.578+0000 7fa8d41a3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc107fd0 con 0x7fa8cc197b30 2024-03-28T05:15:14.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.578+0000 7fa8d173e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8cc197b30 0x7fa8cc19cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:14.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.578+0000 7fa8d173e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8cc197b30 0x7fa8cc19cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:53824/0 (socket says 172.21.15.113:53824) 2024-03-28T05:15:14.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.578+0000 7fa8d173e700 1 -- 172.21.15.113:0/3884044870 learned_addr learned my addr 172.21.15.113:0/3884044870 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-03-28T05:15:14.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.578+0000 7fa8d1f3f700 1 --2- 172.21.15.113:0/3884044870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8cc1033e0 0x7fa8cc1975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:14.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.578+0000 7fa8d2740700 1 --2- 172.21.15.113:0/3884044870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8cc197fb0 0x7fa8cc19f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:14.577 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.579+0000 7fa8d173e700 1 -- 172.21.15.113:0/3884044870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8cc197fb0 msgr2=0x7fa8cc19f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.579+0000 7fa8d173e700 1 --2- 172.21.15.113:0/3884044870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8cc197fb0 0x7fa8cc19f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.579+0000 7fa8d173e700 1 -- 172.21.15.113:0/3884044870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8cc1033e0 msgr2=0x7fa8cc1975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.579+0000 7fa8d173e700 1 --2- 172.21.15.113:0/3884044870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8cc1033e0 0x7fa8cc1975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.579+0000 7fa8d173e700 1 -- 172.21.15.113:0/3884044870 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8cc19fa00 con 0x7fa8cc197b30 2024-03-28T05:15:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.579+0000 7fa8d2740700 1 --2- 172.21.15.113:0/3884044870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8cc197fb0 0x7fa8cc19f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:15:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.579+0000 7fa8d1f3f700 1 --2- 172.21.15.113:0/3884044870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8cc1033e0 0x7fa8cc1975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:15:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.579+0000 7fa8d173e700 1 --2- 172.21.15.113:0/3884044870 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8cc197b30 0x7fa8cc19cba0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00b780 tx=0x7fa8bc00ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:15:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.579+0000 7fa8c2ffd700 1 -- 172.21.15.113:0/3884044870 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc004230 con 0x7fa8cc197b30 2024-03-28T05:15:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.579+0000 7fa8c2ffd700 1 -- 172.21.15.113:0/3884044870 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8bc0043d0 con 0x7fa8cc197b30 2024-03-28T05:15:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.579+0000 7fa8d41a3700 1 -- 172.21.15.113:0/3884044870 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8cc074420 con 0x7fa8cc197b30 2024-03-28T05:15:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.579+0000 7fa8c2ffd700 1 -- 172.21.15.113:0/3884044870 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc011c90 con 0x7fa8cc197b30 2024-03-28T05:15:14.578 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.579+0000 7fa8d41a3700 1 -- 172.21.15.113:0/3884044870 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8cc0749e0 con 0x7fa8cc197b30 2024-03-28T05:15:14.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.580+0000 7fa8c2ffd700 1 -- 172.21.15.113:0/3884044870 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fa8bc01a400 con 0x7fa8cc197b30 2024-03-28T05:15:14.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.580+0000 7fa8d41a3700 1 -- 172.21.15.113:0/3884044870 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8cc0637c0 con 0x7fa8cc197b30 2024-03-28T05:15:14.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.580+0000 7fa8c2ffd700 1 --2- 172.21.15.113:0/3884044870 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa8b8039450 0x7fa8b803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:14.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.580+0000 7fa8c2ffd700 1 -- 172.21.15.113:0/3884044870 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa8bc04bb30 con 0x7fa8cc197b30 2024-03-28T05:15:14.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.580+0000 7fa8d1f3f700 1 --2- 172.21.15.113:0/3884044870 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa8b8039450 0x7fa8b803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:14.579 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.581+0000 7fa8d1f3f700 1 --2- 172.21.15.113:0/3884044870 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa8b8039450 0x7fa8b803b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8005200 tx=0x7fa8c801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:15:14.582 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.585+0000 7fa8c2ffd700 1 -- 172.21.15.113:0/3884044870 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa8bc020500 con 0x7fa8cc197b30 2024-03-28T05:15:14.613 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:14 smithi098 ceph-mon[36087]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:14.613 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:14 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:14.613 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:14 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:14.613 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:14 smithi098 ceph-mon[36087]: Reconfiguring mon.c (monmap changed)... 2024-03-28T05:15:14.613 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:14 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T05:15:14.613 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:14 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T05:15:14.613 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:14 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:14.613 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:14 smithi098 ceph-mon[36087]: Reconfiguring daemon mon.c on smithi113 2024-03-28T05:15:14.793 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.796+0000 7fa8d41a3700 1 -- 172.21.15.113:0/3884044870 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi098=y;smithi113=z", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8cc0625d0 con 0x7fa8b8039450 2024-03-28T05:15:14.800 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.803+0000 7fa8c2ffd700 1 -- 172.21.15.113:0/3884044870 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa8cc0625d0 con 0x7fa8b8039450 2024-03-28T05:15:14.800 INFO:teuthology.orchestra.run.smithi113.stdout:Scheduled mgr update... 2024-03-28T05:15:14.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.805+0000 7fa8d41a3700 1 -- 172.21.15.113:0/3884044870 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa8b8039450 msgr2=0x7fa8b803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:14.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.805+0000 7fa8d41a3700 1 --2- 172.21.15.113:0/3884044870 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa8b8039450 0x7fa8b803b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8005200 tx=0x7fa8c801a040 comp rx=0 tx=0).stop 2024-03-28T05:15:14.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.805+0000 7fa8d41a3700 1 -- 172.21.15.113:0/3884044870 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8cc197b30 msgr2=0x7fa8cc19cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:14.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.805+0000 7fa8d41a3700 1 --2- 172.21.15.113:0/3884044870 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8cc197b30 0x7fa8cc19cba0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00b780 tx=0x7fa8bc00ba90 comp rx=0 tx=0).stop 2024-03-28T05:15:14.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.805+0000 7fa8d41a3700 1 -- 172.21.15.113:0/3884044870 shutdown_connections 2024-03-28T05:15:14.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.805+0000 7fa8d41a3700 1 --2- 172.21.15.113:0/3884044870 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa8b8039450 0x7fa8b803b910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:14.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.805+0000 7fa8d41a3700 1 --2- 172.21.15.113:0/3884044870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa8cc197fb0 0x7fa8cc19f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:14.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.805+0000 7fa8d41a3700 1 --2- 172.21.15.113:0/3884044870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8cc1033e0 0x7fa8cc1975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:14.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.805+0000 7fa8d41a3700 1 --2- 172.21.15.113:0/3884044870 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa8cc197b30 0x7fa8cc19cba0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:14.802 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.805+0000 7fa8d41a3700 1 -- 172.21.15.113:0/3884044870 >> 172.21.15.113:0/3884044870 conn(0x7fa8cc0fd280 msgr2=0x7fa8cc0fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:15:14.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.805+0000 7fa8d41a3700 1 -- 172.21.15.113:0/3884044870 shutdown_connections 2024-03-28T05:15:14.803 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:15:14.805+0000 7fa8d41a3700 1 -- 172.21.15.113:0/3884044870 wait complete. 2024-03-28T05:15:15.663 DEBUG:teuthology.orchestra.run.smithi098:mgr.y> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mgr.y.service 2024-03-28T05:15:15.666 DEBUG:teuthology.orchestra.run.smithi113:mgr.z> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mgr.z.service 2024-03-28T05:15:15.668 INFO:tasks.cephadm:Deploying OSDs... 2024-03-28T05:15:15.668 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T05:15:15.668 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T05:15:15.683 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T05:15:15.683 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_1 2024-03-28T05:15:15.738 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T05:15:15.738 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T05:15:15.738 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-28T05:15:15.738 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T05:15:15.738 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T05:15:15.738 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-28 05:14:20.138678507 +0000 2024-03-28T05:15:15.738 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-28 05:09:43.309294680 +0000 2024-03-28T05:15:15.739 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-28 05:09:43.309294680 +0000 2024-03-28T05:15:15.739 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-28 05:09:43.309294680 +0000 2024-03-28T05:15:15.739 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T05:15:15.805 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-28T05:15:15.805 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-28T05:15:15.805 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000276186 s, 1.9 MB/s 2024-03-28T05:15:15.806 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T05:15:15.862 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_2 2024-03-28T05:15:15.916 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T05:15:15.916 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T05:15:15.917 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-28T05:15:15.917 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T05:15:15.917 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T05:15:15.917 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-28 05:14:20.170678223 +0000 2024-03-28T05:15:15.917 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-28 05:09:43.670291564 +0000 2024-03-28T05:15:15.917 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-28 05:09:43.670291564 +0000 2024-03-28T05:15:15.917 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-28 05:09:43.670291564 +0000 2024-03-28T05:15:15.917 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T05:15:15.983 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-28T05:15:15.983 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-28T05:15:15.983 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000316138 s, 1.6 MB/s 2024-03-28T05:15:15.984 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T05:15:15.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:15 smithi113 ceph-mon[37164]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:15.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:15 smithi113 ceph-mon[37164]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi098=y;smithi113=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:15:15.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:15 smithi113 ceph-mon[37164]: Saving service mgr spec with placement smithi023=x;smithi098=y;smithi113=z;count:3 2024-03-28T05:15:15.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:15 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:15.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:15 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:15.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:15 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:15.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:15 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:15:16.039 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_3 2024-03-28T05:15:16.093 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T05:15:16.093 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T05:15:16.093 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-28T05:15:16.093 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T05:15:16.093 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T05:15:16.093 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-28 05:14:20.207677895 +0000 2024-03-28T05:15:16.094 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-28 05:09:44.035288413 +0000 2024-03-28T05:15:16.094 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-28 05:09:44.035288413 +0000 2024-03-28T05:15:16.094 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-28 05:09:44.035288413 +0000 2024-03-28T05:15:16.094 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T05:15:16.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:15 smithi098 ceph-mon[36087]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:16.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:15 smithi098 ceph-mon[36087]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi098=y;smithi113=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:15:16.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:15 smithi098 ceph-mon[36087]: Saving service mgr spec with placement smithi023=x;smithi098=y;smithi113=z;count:3 2024-03-28T05:15:16.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:15 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:16.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:15 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:16.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:15 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:16.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:15 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:15:16.157 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:15 smithi023 ceph-mon[31954]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:16.157 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:15 smithi023 ceph-mon[31954]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi098=y;smithi113=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:15:16.157 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:15 smithi023 ceph-mon[31954]: Saving service mgr spec with placement smithi023=x;smithi098=y;smithi113=z;count:3 2024-03-28T05:15:16.157 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:15 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:16.157 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:15 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:16.157 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:15 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:16.157 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:15 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:15:16.159 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-28T05:15:16.159 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-28T05:15:16.159 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000286991 s, 1.8 MB/s 2024-03-28T05:15:16.160 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T05:15:16.216 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_4 2024-03-28T05:15:16.271 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T05:15:16.271 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T05:15:16.271 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-28T05:15:16.271 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T05:15:16.271 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T05:15:16.271 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-28 05:14:20.233677664 +0000 2024-03-28T05:15:16.271 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-28 05:09:44.381285427 +0000 2024-03-28T05:15:16.271 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-28 05:09:44.381285427 +0000 2024-03-28T05:15:16.272 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-28 05:09:44.381285427 +0000 2024-03-28T05:15:16.272 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T05:15:16.336 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-28T05:15:16.336 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-28T05:15:16.336 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000273387 s, 1.9 MB/s 2024-03-28T05:15:16.337 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T05:15:16.393 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-28T05:15:16.393 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T05:15:16.412 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T05:15:16.412 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-03-28T05:15:16.469 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T05:15:16.469 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T05:15:16.469 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-03-28T05:15:16.469 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T05:15:16.469 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T05:15:16.469 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-28 05:15:00.959681352 +0000 2024-03-28T05:15:16.470 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-28 05:09:53.183416695 +0000 2024-03-28T05:15:16.470 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-28 05:09:53.183416695 +0000 2024-03-28T05:15:16.470 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-28 05:09:53.183416695 +0000 2024-03-28T05:15:16.470 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T05:15:16.540 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-28T05:15:16.540 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-28T05:15:16.540 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000288221 s, 1.8 MB/s 2024-03-28T05:15:16.541 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T05:15:16.600 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-03-28T05:15:16.658 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T05:15:16.658 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T05:15:16.658 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 681 Links: 1 2024-03-28T05:15:16.658 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T05:15:16.658 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T05:15:16.658 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-28 05:15:00.960681339 +0000 2024-03-28T05:15:16.658 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-28 05:09:53.508412307 +0000 2024-03-28T05:15:16.658 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-28 05:09:53.508412307 +0000 2024-03-28T05:15:16.658 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-28 05:09:53.508412307 +0000 2024-03-28T05:15:16.659 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T05:15:16.725 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-28T05:15:16.725 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-28T05:15:16.725 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000224556 s, 2.3 MB/s 2024-03-28T05:15:16.726 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T05:15:16.783 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-03-28T05:15:16.839 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T05:15:16.839 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T05:15:16.839 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-03-28T05:15:16.840 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T05:15:16.840 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T05:15:16.840 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-28 05:15:00.960681339 +0000 2024-03-28T05:15:16.840 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-28 05:09:53.848407717 +0000 2024-03-28T05:15:16.840 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-28 05:09:53.848407717 +0000 2024-03-28T05:15:16.840 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-28 05:09:53.848407717 +0000 2024-03-28T05:15:16.840 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T05:15:16.906 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-28T05:15:16.906 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-28T05:15:16.906 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000242996 s, 2.1 MB/s 2024-03-28T05:15:16.907 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T05:15:16.963 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-03-28T05:15:17.019 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T05:15:17.019 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T05:15:17.019 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-28T05:15:17.019 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T05:15:17.019 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T05:15:17.019 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-28 05:15:00.960681339 +0000 2024-03-28T05:15:17.019 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-28 05:09:54.169403383 +0000 2024-03-28T05:15:17.019 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-28 05:09:54.169403383 +0000 2024-03-28T05:15:17.019 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-28 05:09:54.169403383 +0000 2024-03-28T05:15:17.019 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T05:15:17.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:16 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:17.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:16 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:15:17.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:16 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:17.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:16 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T05:15:17.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:16 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T05:15:17.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:16 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T05:15:17.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:16 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:17.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:16 smithi098 ceph-mon[36087]: Deploying daemon mgr.z on smithi113 2024-03-28T05:15:17.085 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-28T05:15:17.085 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-28T05:15:17.085 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000206187 s, 2.5 MB/s 2024-03-28T05:15:17.086 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T05:15:17.142 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-28T05:15:17.142 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T05:15:17.158 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T05:15:17.158 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_1 2024-03-28T05:15:17.167 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:16 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:17.167 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:16 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:15:17.167 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:16 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:17.168 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:16 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T05:15:17.168 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:16 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T05:15:17.168 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:16 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T05:15:17.168 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:16 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:17.168 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:16 smithi113 ceph-mon[37164]: Deploying daemon mgr.z on smithi113 2024-03-28T05:15:17.183 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T05:15:17.183 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T05:15:17.183 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-03-28T05:15:17.183 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T05:15:17.183 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T05:15:17.183 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-03-28 05:14:51.398715432 +0000 2024-03-28T05:15:17.183 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-03-28 05:10:02.988902924 +0000 2024-03-28T05:15:17.183 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-03-28 05:10:02.988902924 +0000 2024-03-28T05:15:17.183 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2024-03-28 05:10:02.988902924 +0000 2024-03-28T05:15:17.183 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T05:15:17.250 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-03-28T05:15:17.250 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-03-28T05:15:17.250 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000214018 s, 2.4 MB/s 2024-03-28T05:15:17.252 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T05:15:17.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:16 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:17.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:16 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:15:17.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:16 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:17.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:16 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T05:15:17.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:16 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T05:15:17.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:16 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T05:15:17.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:16 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:17.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:16 smithi023 ceph-mon[31954]: Deploying daemon mgr.z on smithi113 2024-03-28T05:15:17.307 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_2 2024-03-28T05:15:17.361 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T05:15:17.361 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T05:15:17.361 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-28T05:15:17.361 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T05:15:17.361 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T05:15:17.362 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-03-28 05:14:51.399715412 +0000 2024-03-28T05:15:17.362 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-03-28 05:10:03.347895341 +0000 2024-03-28T05:15:17.362 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-03-28 05:10:03.347895341 +0000 2024-03-28T05:15:17.362 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2024-03-28 05:10:03.347895341 +0000 2024-03-28T05:15:17.362 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T05:15:17.426 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-03-28T05:15:17.426 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-03-28T05:15:17.426 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000244349 s, 2.1 MB/s 2024-03-28T05:15:17.427 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T05:15:17.482 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_3 2024-03-28T05:15:17.537 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T05:15:17.537 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T05:15:17.537 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-03-28T05:15:17.537 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T05:15:17.537 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T05:15:17.537 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-03-28 05:14:51.399715412 +0000 2024-03-28T05:15:17.538 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-03-28 05:10:03.683888244 +0000 2024-03-28T05:15:17.538 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-03-28 05:10:03.683888244 +0000 2024-03-28T05:15:17.538 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2024-03-28 05:10:03.683888244 +0000 2024-03-28T05:15:17.538 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T05:15:17.768 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-03-28T05:15:17.768 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-03-28T05:15:17.768 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000208804 s, 2.5 MB/s 2024-03-28T05:15:17.769 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T05:15:17.784 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_4 2024-03-28T05:15:17.847 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T05:15:17.847 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T05:15:17.847 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-28T05:15:17.847 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T05:15:17.847 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T05:15:17.847 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-03-28 05:15:17.714174763 +0000 2024-03-28T05:15:17.847 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-03-28 05:10:04.019881147 +0000 2024-03-28T05:15:17.847 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-03-28 05:10:04.019881147 +0000 2024-03-28T05:15:17.847 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2024-03-28 05:10:04.019881147 +0000 2024-03-28T05:15:17.847 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T05:15:17.959 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-03-28T05:15:17.960 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-03-28T05:15:17.960 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.00023271 s, 2.2 MB/s 2024-03-28T05:15:17.961 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T05:15:17.976 INFO:tasks.cephadm:Deploying osd.0 on smithi023 with /dev/vg_nvme/lv_4... 2024-03-28T05:15:17.976 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T05:15:18.181 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:17 smithi113 ceph-mon[37164]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:18.181 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:17 smithi113 systemd[1]: Starting Ceph mgr.z for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:15:18.222 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:15:18.264 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:17 smithi098 ceph-mon[36087]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:18.278 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:17 smithi023 ceph-mon[31954]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:18.474 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:18 smithi113 podman[39014]: 2024-03-28 05:15:18.088326399 +0000 UTC m=+0.017509081 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:15:18.474 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:18 smithi113 podman[39014]: 2024-03-28 05:15:18.200221585 +0000 UTC m=+0.129404258 container create a0459dda73a9ac3a3241ac82047ca0e9cf87db092f67fcfe8ea6b9e45fc231ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T05:15:18.474 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:18 smithi113 podman[39014]: 2024-03-28 05:15:18.402907647 +0000 UTC m=+0.332090339 container init a0459dda73a9ac3a3241ac82047ca0e9cf87db092f67fcfe8ea6b9e45fc231ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T05:15:18.474 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:18 smithi113 podman[39014]: 2024-03-28 05:15:18.406324798 +0000 UTC m=+0.335507481 container start a0459dda73a9ac3a3241ac82047ca0e9cf87db092f67fcfe8ea6b9e45fc231ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-28T05:15:18.474 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:18 smithi113 ceph-mgr[39034]: -- 172.21.15.113:0/1619537171 <== mon.1 v2:172.21.15.113:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d7256b85a0 con 0x55d72568cc00 2024-03-28T05:15:18.474 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:18 smithi113 bash[39014]: a0459dda73a9ac3a3241ac82047ca0e9cf87db092f67fcfe8ea6b9e45fc231ed 2024-03-28T05:15:18.740 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:18 smithi113 systemd[1]: Started Ceph mgr.z for bf37ebc4-ecc1-11ee-95d0-87774f69a715. 2024-03-28T05:15:19.063 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:18 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:18.970+0000 7fc1567be200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T05:15:19.063 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:19 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:19.065+0000 7fc1567be200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T05:15:19.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:19 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:19.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:19 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:19.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:19 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:19.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:19 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T05:15:19.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:19 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T05:15:19.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:19 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T05:15:19.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:19 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:19.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:19 smithi023 ceph-mon[31954]: Deploying daemon mgr.y on smithi098 2024-03-28T05:15:19.811 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:19 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:19.720+0000 7fc1567be200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T05:15:19.811 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:19 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:19.814+0000 7fc1567be200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T05:15:19.811 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:19 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:19.811 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:19 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:19.811 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:19 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:19.811 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:19 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T05:15:19.812 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:19 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T05:15:19.812 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:19 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T05:15:19.812 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:19 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:19.812 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:19 smithi113 ceph-mon[37164]: Deploying daemon mgr.y on smithi098 2024-03-28T05:15:19.812 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:19 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:19.812 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:19 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:19.812 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:19 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:19.812 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:19 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T05:15:19.812 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:19 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T05:15:19.812 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:19 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T05:15:19.812 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:19 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:19.812 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:19 smithi098 ceph-mon[36087]: Deploying daemon mgr.y on smithi098 2024-03-28T05:15:20.071 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T05:15:20.071 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T05:15:20.071 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-28T05:15:20.071 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-28T05:15:20.071 INFO:teuthology.orchestra.run.smithi023.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00710091 s, 1.5 GB/s 2024-03-28T05:15:20.071 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-28T05:15:20.126 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:19 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:19.953+0000 7fc1567be200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T05:15:20.126 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:20 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:20.046+0000 7fc1567be200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T05:15:20.126 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:20 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:20.129+0000 7fc1567be200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T05:15:20.664 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:20 smithi023 ceph-mon[31954]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:20.681 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:20 smithi113 ceph-mon[37164]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:20.682 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:20 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:20.398+0000 7fc1567be200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T05:15:20.682 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:20 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:20.506+0000 7fc1567be200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T05:15:20.682 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:20 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:20.591+0000 7fc1567be200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T05:15:20.682 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:20 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:20.684+0000 7fc1567be200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T05:15:20.691 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_4 2024-03-28T05:15:20.754 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:20 smithi098 ceph-mon[36087]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:20.929 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:15:20.990 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:20 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:20.858+0000 7fc1567be200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T05:15:21.081 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:20 smithi098 ceph-mgr[37483]: -- 172.21.15.98:0/3652928391 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562e618fe5a0 con 0x562e618d2800 2024-03-28T05:15:21.363 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:21 smithi098 bash[37464]: 61998b1d7dd0ea89166036176592634b692e444bed05ef6a4d7e0cbf2f8012f2 2024-03-28T05:15:21.365 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:21 smithi098 systemd[1]: Started Ceph mgr.y for bf37ebc4-ecc1-11ee-95d0-87774f69a715. 2024-03-28T05:15:21.863 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:21 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:21.537+0000 7feaffdfb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T05:15:21.863 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:21 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:21.633+0000 7feaffdfb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T05:15:22.129 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:21 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:21.860+0000 7fc1567be200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T05:15:22.129 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:22 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:22.006+0000 7fc1567be200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T05:15:22.255 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:22 smithi023 ceph-mon[31954]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:22.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:22.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:22.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:22.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:22.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:15:22.397 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:22 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:22.308+0000 7feaffdfb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T05:15:22.397 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:22 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:22.399+0000 7feaffdfb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T05:15:22.397 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:22 smithi098 ceph-mon[36087]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:22.397 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:22 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:22.397 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:22 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:22.397 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:22 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:22.397 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:22 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:22.397 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:22 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:15:22.431 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:22 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:22.347+0000 7fc1567be200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T05:15:22.431 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:22 smithi113 ceph-mon[37164]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:22.431 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:22 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:22.431 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:22 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:22.432 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:22 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:22.432 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:22 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:22.432 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:22 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:15:22.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.456+0000 7f3ea9dfe700 1 -- 172.21.15.23:0/1074244431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea4102c60 msgr2=0x7f3ea4109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:22.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.456+0000 7f3ea9dfe700 1 --2- 172.21.15.23:0/1074244431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea4102c60 0x7f3ea4109510 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c00b0d0 tx=0x7f3e9c00b3e0 comp rx=0 tx=0).stop 2024-03-28T05:15:22.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.456+0000 7f3ea9dfe700 1 -- 172.21.15.23:0/1074244431 shutdown_connections 2024-03-28T05:15:22.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.456+0000 7f3ea9dfe700 1 --2- 172.21.15.23:0/1074244431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea4102c60 0x7f3ea4109510 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:22.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.456+0000 7f3ea9dfe700 1 --2- 172.21.15.23:0/1074244431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ea41010c0 0x7f3ea41014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:22.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.456+0000 7f3ea9dfe700 1 --2- 172.21.15.23:0/1074244431 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ea41022c0 0x7f3ea4102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:22.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.456+0000 7f3ea9dfe700 1 -- 172.21.15.23:0/1074244431 >> 172.21.15.23:0/1074244431 conn(0x7f3ea40fc660 msgr2=0x7f3ea40feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:15:22.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.456+0000 7f3ea9dfe700 1 -- 172.21.15.23:0/1074244431 shutdown_connections 2024-03-28T05:15:22.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.456+0000 7f3ea9dfe700 1 -- 172.21.15.23:0/1074244431 wait complete. 2024-03-28T05:15:22.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.456+0000 7f3ea9dfe700 1 Processor -- start 2024-03-28T05:15:22.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.457+0000 7f3ea9dfe700 1 -- start start 2024-03-28T05:15:22.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.457+0000 7f3ea9dfe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ea41010c0 0x7f3ea406a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:22.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.457+0000 7f3ea9dfe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea41022c0 0x7f3ea419d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:22.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.457+0000 7f3ea9dfe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ea4102c60 0x7f3ea419f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:22.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.457+0000 7f3ea9dfe700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea410cc90 con 0x7f3ea41022c0 2024-03-28T05:15:22.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.457+0000 7f3ea9dfe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea410cb10 con 0x7f3ea4102c60 2024-03-28T05:15:22.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.457+0000 7f3ea9dfe700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea410ce10 con 0x7f3ea41010c0 2024-03-28T05:15:22.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.457+0000 7f3ea2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea41022c0 0x7f3ea419d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:22.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.457+0000 7f3ea2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea41022c0 0x7f3ea419d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55806/0 (socket says 172.21.15.23:55806) 2024-03-28T05:15:22.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.457+0000 7f3ea2ffd700 1 -- 172.21.15.23:0/1315707859 learned_addr learned my addr 172.21.15.23:0/1315707859 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:15:22.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.457+0000 7f3ea37fe700 1 --2- 172.21.15.23:0/1315707859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ea41010c0 0x7f3ea406a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:22.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.457+0000 7f3ea3fff700 1 --2- 172.21.15.23:0/1315707859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ea4102c60 0x7f3ea419f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:22.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.457+0000 7f3ea2ffd700 1 -- 172.21.15.23:0/1315707859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ea41010c0 msgr2=0x7f3ea406a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:22.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.457+0000 7f3ea2ffd700 1 --2- 172.21.15.23:0/1315707859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ea41010c0 0x7f3ea406a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:22.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.457+0000 7f3ea2ffd700 1 -- 172.21.15.23:0/1315707859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ea4102c60 msgr2=0x7f3ea419f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:22.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.457+0000 7f3ea2ffd700 1 --2- 172.21.15.23:0/1315707859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ea4102c60 0x7f3ea419f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:22.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.457+0000 7f3ea2ffd700 1 -- 172.21.15.23:0/1315707859 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea419ffe0 con 0x7f3ea41022c0 2024-03-28T05:15:22.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.458+0000 7f3ea2ffd700 1 --2- 172.21.15.23:0/1315707859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea41022c0 0x7f3ea419d0d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3e9800d980 tx=0x7f3e9800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:15:22.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.458+0000 7f3ea0ff9700 1 -- 172.21.15.23:0/1315707859 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e98016070 con 0x7f3ea41022c0 2024-03-28T05:15:22.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.458+0000 7f3ea0ff9700 1 -- 172.21.15.23:0/1315707859 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e98004d10 con 0x7f3ea41022c0 2024-03-28T05:15:22.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.458+0000 7f3ea9dfe700 1 -- 172.21.15.23:0/1315707859 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea41a02d0 con 0x7f3ea41022c0 2024-03-28T05:15:22.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.458+0000 7f3ea0ff9700 1 -- 172.21.15.23:0/1315707859 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e98005670 con 0x7f3ea41022c0 2024-03-28T05:15:22.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.458+0000 7f3ea9dfe700 1 -- 172.21.15.23:0/1315707859 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea41a0860 con 0x7f3ea41022c0 2024-03-28T05:15:22.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.459+0000 7f3ea0ff9700 1 -- 172.21.15.23:0/1315707859 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f3e9800f690 con 0x7f3ea41022c0 2024-03-28T05:15:22.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.459+0000 7f3ea9dfe700 1 -- 172.21.15.23:0/1315707859 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea4068e50 con 0x7f3ea41022c0 2024-03-28T05:15:22.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.459+0000 7f3ea0ff9700 1 --2- 172.21.15.23:0/1315707859 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3e8c039200 0x7f3e8c03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:22.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.459+0000 7f3ea0ff9700 1 -- 172.21.15.23:0/1315707859 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3e9804bee0 con 0x7f3ea41022c0 2024-03-28T05:15:22.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.459+0000 7f3ea37fe700 1 --2- 172.21.15.23:0/1315707859 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3e8c039200 0x7f3e8c03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:22.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.460+0000 7f3ea37fe700 1 --2- 172.21.15.23:0/1315707859 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3e8c039200 0x7f3e8c03b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4102120 tx=0x7f3e94007400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:15:22.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.464+0000 7f3ea0ff9700 1 -- 172.21.15.23:0/1315707859 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3e9801f020 con 0x7f3ea41022c0 2024-03-28T05:15:22.648 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:22 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:22.547+0000 7feaffdfb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T05:15:22.648 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:22 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:22.651+0000 7feaffdfb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T05:15:22.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:22.686+0000 7f3ea9dfe700 1 -- 172.21.15.23:0/1315707859 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ea4062690 con 0x7f3e8c039200 2024-03-28T05:15:22.740 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:22 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:22.435+0000 7fc1567be200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T05:15:22.740 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:22 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:22.640+0000 7fc1567be200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T05:15:22.999 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:22 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:22.735+0000 7feaffdfb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T05:15:22.999 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:23 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:23.002+0000 7feaffdfb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T05:15:23.101 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:22 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:22.825+0000 7fc1567be200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T05:15:23.285 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:23 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:23.110+0000 7feaffdfb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T05:15:23.285 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:23 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:23.195+0000 7feaffdfb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T05:15:23.285 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:23 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:23.288+0000 7feaffdfb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T05:15:23.358 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:23 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:23.105+0000 7fc1567be200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T05:15:23.358 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:23 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:23.362+0000 7fc1567be200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T05:15:23.560 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:23 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:23.460+0000 7feaffdfb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T05:15:23.740 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:23 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:23.595+0000 7fc1567be200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T05:15:23.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:23.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:23.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:23.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:23.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:23.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:15:23.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:23.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:15:23.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:15:23.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:23.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T05:15:23.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T05:15:23.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:23.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:23.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:23.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:23.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:23.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:23.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:15:23.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:23.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:15:23.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:15:23.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:23.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T05:15:23.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T05:15:23.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:23.963 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:23.963 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:23.963 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:23.964 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:23.964 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:23.964 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:15:23.964 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:23.964 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:15:23.964 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:15:23.964 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:23.964 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T05:15:23.964 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T05:15:23.964 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:24.740 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:24 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:24.391+0000 7fc1567be200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T05:15:24.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:24 smithi113 ceph-mon[37164]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:24.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:24 smithi113 ceph-mon[37164]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:15:24.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:24 smithi113 ceph-mon[37164]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T05:15:24.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:24 smithi113 ceph-mon[37164]: Reconfiguring daemon mgr.x on smithi023 2024-03-28T05:15:24.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:24 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:24.863 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:24 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:24.420+0000 7feaffdfb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T05:15:24.863 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:24 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:24.549+0000 7feaffdfb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T05:15:24.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:24 smithi098 ceph-mon[36087]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:24.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:24 smithi098 ceph-mon[36087]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:15:24.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:24 smithi098 ceph-mon[36087]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T05:15:24.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:24 smithi098 ceph-mon[36087]: Reconfiguring daemon mgr.x on smithi023 2024-03-28T05:15:24.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:24 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:25.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:24 smithi023 ceph-mon[31954]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:25.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:24 smithi023 ceph-mon[31954]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:15:25.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:24 smithi023 ceph-mon[31954]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T05:15:25.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:24 smithi023 ceph-mon[31954]: Reconfiguring daemon mgr.x on smithi023 2024-03-28T05:15:25.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:24 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:25.155 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:24 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:24.878+0000 7feaffdfb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T05:15:25.156 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:24 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:24.966+0000 7feaffdfb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T05:15:25.156 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:25 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:25.158+0000 7feaffdfb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T05:15:25.241 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:24 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:24.959+0000 7fc1567be200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T05:15:25.241 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:25 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:25.056+0000 7fc1567be200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T05:15:25.613 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:25 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:25.340+0000 7feaffdfb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T05:15:25.631 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:15:25 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:15:25.272+0000 7fc1567be200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T05:15:25.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:25.642+0000 7f3ea0ff9700 1 -- 172.21.15.23:0/1315707859 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 14) v1 ==== 91698+0+0 (secure 0 0 0) 0x7f3e9804f030 con 0x7f3ea41022c0 2024-03-28T05:15:25.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:25 smithi023 ceph-mon[31954]: Standby manager daemon z started 2024-03-28T05:15:25.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:25 smithi023 ceph-mon[31954]: from='mgr.? 172.21.15.113:0/2983020111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T05:15:25.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:25 smithi023 ceph-mon[31954]: from='mgr.? 172.21.15.113:0/2983020111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T05:15:25.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:25 smithi023 ceph-mon[31954]: from='mgr.? 172.21.15.113:0/2983020111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T05:15:25.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:25 smithi023 ceph-mon[31954]: from='mgr.? 172.21.15.113:0/2983020111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T05:15:25.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:25 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:25.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:25 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:25.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:25 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:15:25.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:25 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:25.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:25 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:15:25.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:25 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:25.875 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:25 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:25.622+0000 7feaffdfb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T05:15:25.875 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:25 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:25.878+0000 7feaffdfb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T05:15:25.875 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:25 smithi098 ceph-mon[36087]: Standby manager daemon z started 2024-03-28T05:15:25.875 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:25 smithi098 ceph-mon[36087]: from='mgr.? 172.21.15.113:0/2983020111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T05:15:25.875 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:25 smithi098 ceph-mon[36087]: from='mgr.? 172.21.15.113:0/2983020111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T05:15:25.875 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:25 smithi098 ceph-mon[36087]: from='mgr.? 172.21.15.113:0/2983020111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T05:15:25.876 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:25 smithi098 ceph-mon[36087]: from='mgr.? 172.21.15.113:0/2983020111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T05:15:25.876 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:25 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:25.876 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:25 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:25.876 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:25 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:15:25.876 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:25 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:25.876 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:25 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:15:25.876 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:25 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:25.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:25 smithi113 ceph-mon[37164]: Standby manager daemon z started 2024-03-28T05:15:25.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:25 smithi113 ceph-mon[37164]: from='mgr.? 172.21.15.113:0/2983020111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T05:15:25.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:25 smithi113 ceph-mon[37164]: from='mgr.? 172.21.15.113:0/2983020111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T05:15:25.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:25 smithi113 ceph-mon[37164]: from='mgr.? 172.21.15.113:0/2983020111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T05:15:25.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:25 smithi113 ceph-mon[37164]: from='mgr.? 172.21.15.113:0/2983020111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T05:15:25.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:25 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:25.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:25 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:25.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:25 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:15:25.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:25 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:25.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:25 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:15:25.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:25 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:26.363 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:26 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:26.111+0000 7feaffdfb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T05:15:26.895 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:26 smithi098 ceph-mon[36087]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:26.896 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:26 smithi098 ceph-mon[36087]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-28T05:15:26.896 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:26 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T05:15:26.896 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:26 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1494903840' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18150942-59a4-4c3b-83d4-c1fc3ccf1c09"}]: dispatch 2024-03-28T05:15:26.896 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:26 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1494903840' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "18150942-59a4-4c3b-83d4-c1fc3ccf1c09"}]': finished 2024-03-28T05:15:26.896 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:26 smithi098 ceph-mon[36087]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T05:15:26.896 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:26 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T05:15:26.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:26 smithi113 ceph-mon[37164]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:26.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:26 smithi113 ceph-mon[37164]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-28T05:15:26.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:26 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T05:15:26.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:26 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1494903840' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18150942-59a4-4c3b-83d4-c1fc3ccf1c09"}]: dispatch 2024-03-28T05:15:26.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:26 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1494903840' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "18150942-59a4-4c3b-83d4-c1fc3ccf1c09"}]': finished 2024-03-28T05:15:26.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:26 smithi113 ceph-mon[37164]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T05:15:26.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:26 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T05:15:27.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:26 smithi023 ceph-mon[31954]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:27.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:26 smithi023 ceph-mon[31954]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-28T05:15:27.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:26 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T05:15:27.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:26 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1494903840' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18150942-59a4-4c3b-83d4-c1fc3ccf1c09"}]: dispatch 2024-03-28T05:15:27.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:26 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1494903840' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "18150942-59a4-4c3b-83d4-c1fc3ccf1c09"}]': finished 2024-03-28T05:15:27.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:26 smithi023 ceph-mon[31954]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T05:15:27.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:26 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T05:15:27.363 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:26 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:26.898+0000 7feaffdfb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T05:15:27.762 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:27 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1769617045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:15:27.762 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:27 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:27.459+0000 7feaffdfb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T05:15:27.763 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:27 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:27.554+0000 7feaffdfb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T05:15:27.763 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:15:27 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:15:27.765+0000 7feaffdfb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T05:15:27.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:27 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1769617045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:15:28.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:27 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1769617045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:15:28.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:28.652+0000 7f3ea0ff9700 1 -- 172.21.15.23:0/1315707859 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f3e98003680 con 0x7f3ea41022c0 2024-03-28T05:15:28.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:28 smithi113 ceph-mon[37164]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:28.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:28 smithi113 ceph-mon[37164]: Standby manager daemon y started 2024-03-28T05:15:28.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:28 smithi113 ceph-mon[37164]: from='mgr.? 172.21.15.98:0/1128031872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T05:15:28.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:28 smithi113 ceph-mon[37164]: from='mgr.? 172.21.15.98:0/1128031872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T05:15:28.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:28 smithi113 ceph-mon[37164]: from='mgr.? 172.21.15.98:0/1128031872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T05:15:28.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:28 smithi113 ceph-mon[37164]: from='mgr.? 172.21.15.98:0/1128031872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T05:15:29.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:28 smithi023 ceph-mon[31954]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:29.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:28 smithi023 ceph-mon[31954]: Standby manager daemon y started 2024-03-28T05:15:29.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:28 smithi023 ceph-mon[31954]: from='mgr.? 172.21.15.98:0/1128031872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T05:15:29.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:28 smithi023 ceph-mon[31954]: from='mgr.? 172.21.15.98:0/1128031872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T05:15:29.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:28 smithi023 ceph-mon[31954]: from='mgr.? 172.21.15.98:0/1128031872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T05:15:29.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:28 smithi023 ceph-mon[31954]: from='mgr.? 172.21.15.98:0/1128031872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T05:15:29.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:28 smithi098 ceph-mon[36087]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:29.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:28 smithi098 ceph-mon[36087]: Standby manager daemon y started 2024-03-28T05:15:29.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:28 smithi098 ceph-mon[36087]: from='mgr.? 172.21.15.98:0/1128031872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T05:15:29.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:28 smithi098 ceph-mon[36087]: from='mgr.? 172.21.15.98:0/1128031872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T05:15:29.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:28 smithi098 ceph-mon[36087]: from='mgr.? 172.21.15.98:0/1128031872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T05:15:29.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:28 smithi098 ceph-mon[36087]: from='mgr.? 172.21.15.98:0/1128031872' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T05:15:29.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:29 smithi113 ceph-mon[37164]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-28T05:15:29.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:29 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T05:15:30.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:29 smithi023 ceph-mon[31954]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-28T05:15:30.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:29 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T05:15:30.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:29 smithi098 ceph-mon[36087]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-28T05:15:30.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:29 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T05:15:30.677 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:30 smithi023 ceph-mon[31954]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:30.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:30 smithi113 ceph-mon[37164]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:31.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:30 smithi098 ceph-mon[36087]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:31.949 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:31 smithi023 ceph-mon[31954]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:31.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:31 smithi113 ceph-mon[37164]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:32.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:31 smithi098 ceph-mon[36087]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:33.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:33 smithi113 ceph-mon[37164]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:33.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:33 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T05:15:33.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:33 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:33.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:33 smithi113 ceph-mon[37164]: Deploying daemon osd.0 on smithi023 2024-03-28T05:15:34.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:33 smithi023 ceph-mon[31954]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:34.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:33 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T05:15:34.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:33 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:34.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:33 smithi023 ceph-mon[31954]: Deploying daemon osd.0 on smithi023 2024-03-28T05:15:34.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:33 smithi098 ceph-mon[36087]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:34.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:33 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T05:15:34.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:33 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:34.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:33 smithi098 ceph-mon[36087]: Deploying daemon osd.0 on smithi023 2024-03-28T05:15:35.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:35 smithi113 ceph-mon[37164]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:36.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:35 smithi023 ceph-mon[31954]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:36.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:35 smithi098 ceph-mon[36087]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:37.893 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:37 smithi023 ceph-mon[31954]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:37.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:37 smithi113 ceph-mon[37164]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:38.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:37 smithi098 ceph-mon[36087]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:39.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:39 smithi113 ceph-mon[37164]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:39.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:39 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:39.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:39 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:39.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:39 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:15:40.012 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:39 smithi023 ceph-mon[31954]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:40.012 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:39 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:40.012 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:39 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:40.012 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:39 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:15:40.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:39 smithi098 ceph-mon[36087]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:40.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:39 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:40.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:39 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:40.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:39 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:15:41.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:41.798+0000 7f3ea0ff9700 1 -- 172.21.15.23:0/1315707859 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3ea4062690 con 0x7f3e8c039200 2024-03-28T05:15:41.797 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 0 on host 'smithi023' 2024-03-28T05:15:41.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:41.800+0000 7f3ea9dfe700 1 -- 172.21.15.23:0/1315707859 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3e8c039200 msgr2=0x7f3e8c03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:41.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:41.800+0000 7f3ea9dfe700 1 --2- 172.21.15.23:0/1315707859 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3e8c039200 0x7f3e8c03b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4102120 tx=0x7f3e94007400 comp rx=0 tx=0).stop 2024-03-28T05:15:41.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:41.800+0000 7f3ea9dfe700 1 -- 172.21.15.23:0/1315707859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea41022c0 msgr2=0x7f3ea419d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:41.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:41.800+0000 7f3ea9dfe700 1 --2- 172.21.15.23:0/1315707859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea41022c0 0x7f3ea419d0d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3e9800d980 tx=0x7f3e9800dc90 comp rx=0 tx=0).stop 2024-03-28T05:15:41.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:41.800+0000 7f3ea9dfe700 1 -- 172.21.15.23:0/1315707859 shutdown_connections 2024-03-28T05:15:41.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:41.800+0000 7f3ea9dfe700 1 --2- 172.21.15.23:0/1315707859 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3e8c039200 0x7f3e8c03b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:41.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:41.800+0000 7f3ea9dfe700 1 --2- 172.21.15.23:0/1315707859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ea4102c60 0x7f3ea419f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:41.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:41.800+0000 7f3ea9dfe700 1 --2- 172.21.15.23:0/1315707859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3ea41010c0 0x7f3ea406a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:41.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:41.800+0000 7f3ea9dfe700 1 --2- 172.21.15.23:0/1315707859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea41022c0 0x7f3ea419d0d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:41.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:41.800+0000 7f3ea9dfe700 1 -- 172.21.15.23:0/1315707859 >> 172.21.15.23:0/1315707859 conn(0x7f3ea40fc660 msgr2=0x7f3ea41055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:15:41.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:41.801+0000 7f3ea9dfe700 1 -- 172.21.15.23:0/1315707859 shutdown_connections 2024-03-28T05:15:41.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:15:41.801+0000 7f3ea9dfe700 1 -- 172.21.15.23:0/1315707859 wait complete. 2024-03-28T05:15:41.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:41 smithi113 ceph-mon[37164]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:42.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:41 smithi023 ceph-mon[31954]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:42.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:41 smithi098 ceph-mon[36087]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:42.384 DEBUG:teuthology.orchestra.run.smithi023:osd.0> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.0.service 2024-03-28T05:15:42.386 INFO:tasks.cephadm:Deploying osd.1 on smithi098 with /dev/vg_nvme/lv_4... 2024-03-28T05:15:42.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T05:15:42.632 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T05:15:42.778 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:15:42 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-0[46655]: 2024-03-28T05:15:42.561+0000 7fe4a543d7c0 -1 osd.0 0 log_to_monitors true 2024-03-28T05:15:43.012 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:42 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:43.012 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:42 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:43.012 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:42 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:43.012 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:42 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:43.012 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:42 smithi098 ceph-mon[36087]: from='osd.0 [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T05:15:43.049 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:42 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:43.049 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:42 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:43.049 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:42 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:43.049 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:42 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:43.049 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:42 smithi023 ceph-mon[31954]: from='osd.0 [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T05:15:43.240 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:42 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:43.240 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:42 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:43.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:42 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:43.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:42 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:43.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:42 smithi113 ceph-mon[37164]: from='osd.0 [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T05:15:44.417 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T05:15:44.417 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T05:15:44.417 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-28T05:15:44.417 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-28T05:15:44.417 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128208 s, 818 MB/s 2024-03-28T05:15:44.417 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-28T05:15:44.628 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:44 smithi098 ceph-mon[36087]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:44.628 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:44 smithi098 ceph-mon[36087]: from='osd.0 [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T05:15:44.628 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:44 smithi098 ceph-mon[36087]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T05:15:44.628 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:44 smithi098 ceph-mon[36087]: from='osd.0 [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T05:15:44.628 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:44 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T05:15:44.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:44 smithi113 ceph-mon[37164]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:44.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:44 smithi113 ceph-mon[37164]: from='osd.0 [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T05:15:44.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:44 smithi113 ceph-mon[37164]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T05:15:44.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:44 smithi113 ceph-mon[37164]: from='osd.0 [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T05:15:44.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:44 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T05:15:44.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:44 smithi023 ceph-mon[31954]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:44.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:44 smithi023 ceph-mon[31954]: from='osd.0 [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T05:15:44.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:44 smithi023 ceph-mon[31954]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T05:15:44.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:44 smithi023 ceph-mon[31954]: from='osd.0 [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T05:15:44.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:44 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T05:15:44.779 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:15:44 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-0[46655]: 2024-03-28T05:15:44.452+0000 7fe49aaa2700 -1 osd.0 0 waiting for initial osdmap 2024-03-28T05:15:44.779 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:15:44 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-0[46655]: 2024-03-28T05:15:44.455+0000 7fe496895700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T05:15:45.034 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_4 2024-03-28T05:15:45.276 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T05:15:45.656 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:45 smithi023 ceph-mon[31954]: from='osd.0 [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T05:15:45.656 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:45 smithi023 ceph-mon[31954]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T05:15:45.656 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:45 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T05:15:45.656 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:45 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T05:15:45.734 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:45 smithi098 ceph-mon[36087]: from='osd.0 [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T05:15:45.734 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:45 smithi098 ceph-mon[36087]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T05:15:45.734 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:45 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T05:15:45.734 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:45 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T05:15:45.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:45 smithi113 ceph-mon[37164]: from='osd.0 [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T05:15:45.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:45 smithi113 ceph-mon[37164]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T05:15:45.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:45 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T05:15:45.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:45 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T05:15:46.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:46 smithi098 ceph-mon[36087]: purged_snaps scrub starts 2024-03-28T05:15:46.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:46 smithi098 ceph-mon[36087]: purged_snaps scrub ok 2024-03-28T05:15:46.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:46 smithi098 ceph-mon[36087]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:46.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:46 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T05:15:46.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:46 smithi098 ceph-mon[36087]: osd.0 [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881] boot 2024-03-28T05:15:46.724 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:46 smithi098 ceph-mon[36087]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T05:15:46.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:46 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T05:15:46.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:46 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:46.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:46 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:46.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:46 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:15:46.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:46 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:46.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:46 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:46.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:46 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:15:46.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:46 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:46.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:46 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:15:46.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:46 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:46.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:46 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:15:46.725 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:46 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:46.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:46 smithi113 ceph-mon[37164]: purged_snaps scrub starts 2024-03-28T05:15:46.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:46 smithi113 ceph-mon[37164]: purged_snaps scrub ok 2024-03-28T05:15:46.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:46 smithi113 ceph-mon[37164]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:46.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:46 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T05:15:46.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:46 smithi113 ceph-mon[37164]: osd.0 [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881] boot 2024-03-28T05:15:46.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:46 smithi113 ceph-mon[37164]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T05:15:46.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:46 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T05:15:46.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:46 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:46.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:46 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:46.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:46 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:15:46.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:46 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:46.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:46 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:46.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:46 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:15:46.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:46 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:46.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:46 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:15:46.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:46 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:46.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:46 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:15:46.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:46 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:46.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:46 smithi023 ceph-mon[31954]: purged_snaps scrub starts 2024-03-28T05:15:46.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:46 smithi023 ceph-mon[31954]: purged_snaps scrub ok 2024-03-28T05:15:46.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:46 smithi023 ceph-mon[31954]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T05:15:46.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:46 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T05:15:46.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:46 smithi023 ceph-mon[31954]: osd.0 [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881] boot 2024-03-28T05:15:46.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:46 smithi023 ceph-mon[31954]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T05:15:46.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:46 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T05:15:46.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:46 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:46.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:46 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:46.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:46 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:15:46.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:46 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:46.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:46 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:46.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:46 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:15:46.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:46 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:46.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:46 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:15:46.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:46 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:46.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:46 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:15:46.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:46 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:15:46.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.830+0000 7fceab72f700 1 -- 172.21.15.98:0/1679102384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcea41033e0 msgr2=0x7fcea4103840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:46.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.830+0000 7fceab72f700 1 --2- 172.21.15.98:0/1679102384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcea41033e0 0x7fcea4103840 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fce940099c0 tx=0x7fce94009cd0 comp rx=0 tx=0).stop 2024-03-28T05:15:46.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.830+0000 7fceab72f700 1 -- 172.21.15.98:0/1679102384 shutdown_connections 2024-03-28T05:15:46.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.830+0000 7fceab72f700 1 --2- 172.21.15.98:0/1679102384 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcea4103d80 0x7fcea410a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:46.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.830+0000 7fceab72f700 1 --2- 172.21.15.98:0/1679102384 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcea41021e0 0x7fcea4102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:46.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.830+0000 7fceab72f700 1 --2- 172.21.15.98:0/1679102384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcea41033e0 0x7fcea4103840 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.830+0000 7fceab72f700 1 -- 172.21.15.98:0/1679102384 >> 172.21.15.98:0/1679102384 conn(0x7fcea40fd7a0 msgr2=0x7fcea40ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:15:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.830+0000 7fceab72f700 1 -- 172.21.15.98:0/1679102384 shutdown_connections 2024-03-28T05:15:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.830+0000 7fceab72f700 1 -- 172.21.15.98:0/1679102384 wait complete. 2024-03-28T05:15:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.831+0000 7fceab72f700 1 Processor -- start 2024-03-28T05:15:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.831+0000 7fceab72f700 1 -- start start 2024-03-28T05:15:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.831+0000 7fceab72f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcea41021e0 0x7fcea419a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.831+0000 7fceab72f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcea41033e0 0x7fcea419aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.831+0000 7fceab72f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcea4103d80 0x7fcea41a1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.831+0000 7fceab72f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea410dda0 con 0x7fcea41033e0 2024-03-28T05:15:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.831+0000 7fceab72f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea410dc20 con 0x7fcea4103d80 2024-03-28T05:15:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.831+0000 7fceab72f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea410df20 con 0x7fcea41021e0 2024-03-28T05:15:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.831+0000 7fcea9ccc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcea4103d80 0x7fcea41a1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.831+0000 7fcea9ccc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcea4103d80 0x7fcea41a1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53264/0 (socket says 172.21.15.98:53264) 2024-03-28T05:15:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.831+0000 7fcea9ccc700 1 -- 172.21.15.98:0/654370877 learned_addr learned my addr 172.21.15.98:0/654370877 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-28T05:15:46.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.831+0000 7fcea8cca700 1 --2- 172.21.15.98:0/654370877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcea41033e0 0x7fcea419aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.831+0000 7fcea94cb700 1 --2- 172.21.15.98:0/654370877 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcea41021e0 0x7fcea419a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.831+0000 7fcea8cca700 1 -- 172.21.15.98:0/654370877 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcea41021e0 msgr2=0x7fcea419a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.831+0000 7fcea8cca700 1 --2- 172.21.15.98:0/654370877 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcea41021e0 0x7fcea419a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.831+0000 7fcea8cca700 1 -- 172.21.15.98:0/654370877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcea4103d80 msgr2=0x7fcea41a1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:15:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.831+0000 7fcea8cca700 1 --2- 172.21.15.98:0/654370877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcea4103d80 0x7fcea41a1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:15:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.831+0000 7fcea8cca700 1 -- 172.21.15.98:0/654370877 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea41a21f0 con 0x7fcea41033e0 2024-03-28T05:15:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.831+0000 7fcea9ccc700 1 --2- 172.21.15.98:0/654370877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcea4103d80 0x7fcea41a1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:15:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.831+0000 7fcea8cca700 1 --2- 172.21.15.98:0/654370877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcea41033e0 0x7fcea419aad0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fce940099c0 tx=0x7fce94009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:15:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.832+0000 7fce9a7fc700 1 -- 172.21.15.98:0/654370877 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce94027070 con 0x7fcea41033e0 2024-03-28T05:15:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.832+0000 7fceab72f700 1 -- 172.21.15.98:0/654370877 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea41a2480 con 0x7fcea41033e0 2024-03-28T05:15:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.832+0000 7fce9a7fc700 1 -- 172.21.15.98:0/654370877 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce940058e0 con 0x7fcea41033e0 2024-03-28T05:15:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.832+0000 7fce9a7fc700 1 -- 172.21.15.98:0/654370877 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce94022bb0 con 0x7fcea41033e0 2024-03-28T05:15:46.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.832+0000 7fceab72f700 1 -- 172.21.15.98:0/654370877 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea41a2960 con 0x7fcea41033e0 2024-03-28T05:15:46.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.833+0000 7fceab72f700 1 -- 172.21.15.98:0/654370877 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea4068e50 con 0x7fcea41033e0 2024-03-28T05:15:46.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.834+0000 7fce9a7fc700 1 -- 172.21.15.98:0/654370877 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7fce9402a070 con 0x7fcea41033e0 2024-03-28T05:15:46.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.834+0000 7fce9a7fc700 1 --2- 172.21.15.98:0/654370877 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fce900a2800 0x7fce900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:15:46.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.834+0000 7fcea94cb700 1 --2- 172.21.15.98:0/654370877 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fce900a2800 0x7fce900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:15:46.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.834+0000 7fce9a7fc700 1 -- 172.21.15.98:0/654370877 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7fce940ce190 con 0x7fcea41033e0 2024-03-28T05:15:46.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.835+0000 7fcea94cb700 1 --2- 172.21.15.98:0/654370877 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fce900a2800 0x7fce900a4cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcea4103240 tx=0x7fcea00091c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:15:46.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:46.838+0000 7fce9a7fc700 1 -- 172.21.15.98:0/654370877 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fce94013070 con 0x7fcea41033e0 2024-03-28T05:15:47.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:15:47.043+0000 7fceab72f700 1 -- 172.21.15.98:0/654370877 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fcea4075ae0 con 0x7fce900a2800 2024-03-28T05:15:47.737 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:47 smithi098 ceph-mon[36087]: Detected new or changed devices on smithi023 2024-03-28T05:15:47.737 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:47 smithi098 ceph-mon[36087]: Adjusting osd_memory_target on smithi023 to 17088M 2024-03-28T05:15:47.738 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:47 smithi098 ceph-mon[36087]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T05:15:47.738 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:47 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:15:47.738 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:47 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:15:47.738 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:47 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:47.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:47 smithi113 ceph-mon[37164]: Detected new or changed devices on smithi023 2024-03-28T05:15:47.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:47 smithi113 ceph-mon[37164]: Adjusting osd_memory_target on smithi023 to 17088M 2024-03-28T05:15:47.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:47 smithi113 ceph-mon[37164]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T05:15:47.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:47 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:15:47.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:47 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:15:47.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:47 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:47.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:47 smithi023 ceph-mon[31954]: Detected new or changed devices on smithi023 2024-03-28T05:15:47.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:47 smithi023 ceph-mon[31954]: Adjusting osd_memory_target on smithi023 to 17088M 2024-03-28T05:15:47.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:47 smithi023 ceph-mon[31954]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T05:15:47.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:47 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:15:47.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:47 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:15:47.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:47 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:48.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:48 smithi098 ceph-mon[36087]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:15:48.727 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:48 smithi098 ceph-mon[36087]: from='client.14253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:15:48.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:48 smithi113 ceph-mon[37164]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:15:48.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:48 smithi113 ceph-mon[37164]: from='client.14253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:15:48.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:48 smithi023 ceph-mon[31954]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:15:48.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:48 smithi023 ceph-mon[31954]: from='client.14253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:15:50.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:50 smithi113 ceph-mon[37164]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:15:50.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:50 smithi113 ceph-mon[37164]: from='client.? 172.21.15.98:0/4094287232' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba6d1a95-5135-4670-95f9-7878c0e7bd64"}]: dispatch 2024-03-28T05:15:50.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:50 smithi113 ceph-mon[37164]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba6d1a95-5135-4670-95f9-7878c0e7bd64"}]: dispatch 2024-03-28T05:15:50.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:50 smithi113 ceph-mon[37164]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba6d1a95-5135-4670-95f9-7878c0e7bd64"}]': finished 2024-03-28T05:15:50.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:50 smithi113 ceph-mon[37164]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T05:15:50.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:50 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T05:15:50.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:50 smithi113 ceph-mon[37164]: from='client.? 172.21.15.98:0/2827643352' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:15:50.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:50 smithi023 ceph-mon[31954]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:15:50.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:50 smithi023 ceph-mon[31954]: from='client.? 172.21.15.98:0/4094287232' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba6d1a95-5135-4670-95f9-7878c0e7bd64"}]: dispatch 2024-03-28T05:15:50.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:50 smithi023 ceph-mon[31954]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba6d1a95-5135-4670-95f9-7878c0e7bd64"}]: dispatch 2024-03-28T05:15:50.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:50 smithi023 ceph-mon[31954]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba6d1a95-5135-4670-95f9-7878c0e7bd64"}]': finished 2024-03-28T05:15:50.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:50 smithi023 ceph-mon[31954]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T05:15:50.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:50 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T05:15:50.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:50 smithi023 ceph-mon[31954]: from='client.? 172.21.15.98:0/2827643352' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:15:50.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:50 smithi098 ceph-mon[36087]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:15:50.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:50 smithi098 ceph-mon[36087]: from='client.? 172.21.15.98:0/4094287232' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba6d1a95-5135-4670-95f9-7878c0e7bd64"}]: dispatch 2024-03-28T05:15:50.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:50 smithi098 ceph-mon[36087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba6d1a95-5135-4670-95f9-7878c0e7bd64"}]: dispatch 2024-03-28T05:15:50.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:50 smithi098 ceph-mon[36087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba6d1a95-5135-4670-95f9-7878c0e7bd64"}]': finished 2024-03-28T05:15:50.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:50 smithi098 ceph-mon[36087]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T05:15:50.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:50 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T05:15:50.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:50 smithi098 ceph-mon[36087]: from='client.? 172.21.15.98:0/2827643352' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:15:52.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:52 smithi113 ceph-mon[37164]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:15:52.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:52 smithi023 ceph-mon[31954]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:15:52.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:52 smithi098 ceph-mon[36087]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:15:54.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:54 smithi113 ceph-mon[37164]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:15:54.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:54 smithi023 ceph-mon[31954]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:15:54.834 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:54 smithi098 ceph-mon[36087]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:15:56.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:56 smithi023 ceph-mon[31954]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:15:56.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:56 smithi098 ceph-mon[36087]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:15:56.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:56 smithi113 ceph-mon[37164]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:15:57.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:57 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T05:15:57.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:57 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:57.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:57 smithi023 ceph-mon[31954]: Deploying daemon osd.1 on smithi098 2024-03-28T05:15:57.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:57 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T05:15:57.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:57 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:57.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:57 smithi098 ceph-mon[36087]: Deploying daemon osd.1 on smithi098 2024-03-28T05:15:57.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:57 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T05:15:57.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:57 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:15:57.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:57 smithi113 ceph-mon[37164]: Deploying daemon osd.1 on smithi098 2024-03-28T05:15:58.754 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:15:58 smithi098 ceph-mon[36087]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:15:58.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:15:58 smithi023 ceph-mon[31954]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:15:58.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:15:58 smithi113 ceph-mon[37164]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:16:00.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:00 smithi023 ceph-mon[31954]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:16:00.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:00 smithi098 ceph-mon[36087]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:16:00.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:00 smithi113 ceph-mon[37164]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:16:02.772 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:02 smithi098 ceph-mon[36087]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:16:02.773 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:02 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:02.773 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:02 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:02.773 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:02 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:16:02.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:02 smithi023 ceph-mon[31954]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:16:02.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:02 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:02.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:02 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:02.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:02 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:16:02.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:02 smithi113 ceph-mon[37164]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:16:02.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:02 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:02.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:02 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:02.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:02 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:16:04.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:04 smithi023 ceph-mon[31954]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:16:04.822 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:04 smithi098 ceph-mon[36087]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:16:04.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:04 smithi113 ceph-mon[37164]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:16:05.455 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 1 on host 'smithi098' 2024-03-28T05:16:05.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:16:05.458+0000 7fce9a7fc700 1 -- 172.21.15.98:0/654370877 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcea4075ae0 con 0x7fce900a2800 2024-03-28T05:16:05.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:16:05.460+0000 7fceab72f700 1 -- 172.21.15.98:0/654370877 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fce900a2800 msgr2=0x7fce900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:16:05.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:16:05.460+0000 7fceab72f700 1 --2- 172.21.15.98:0/654370877 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fce900a2800 0x7fce900a4cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcea4103240 tx=0x7fcea00091c0 comp rx=0 tx=0).stop 2024-03-28T05:16:05.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:16:05.460+0000 7fceab72f700 1 -- 172.21.15.98:0/654370877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcea41033e0 msgr2=0x7fcea419aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:16:05.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:16:05.460+0000 7fceab72f700 1 --2- 172.21.15.98:0/654370877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcea41033e0 0x7fcea419aad0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fce940099c0 tx=0x7fce94009580 comp rx=0 tx=0).stop 2024-03-28T05:16:05.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:16:05.460+0000 7fceab72f700 1 -- 172.21.15.98:0/654370877 shutdown_connections 2024-03-28T05:16:05.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:16:05.460+0000 7fceab72f700 1 --2- 172.21.15.98:0/654370877 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fce900a2800 0x7fce900a4cc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:16:05.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:16:05.460+0000 7fceab72f700 1 --2- 172.21.15.98:0/654370877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcea4103d80 0x7fcea41a1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:16:05.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:16:05.460+0000 7fceab72f700 1 --2- 172.21.15.98:0/654370877 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcea41021e0 0x7fcea419a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:16:05.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:16:05.460+0000 7fceab72f700 1 --2- 172.21.15.98:0/654370877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcea41033e0 0x7fcea419aad0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:16:05.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:16:05.460+0000 7fceab72f700 1 -- 172.21.15.98:0/654370877 >> 172.21.15.98:0/654370877 conn(0x7fcea40fd7a0 msgr2=0x7fcea40743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:16:05.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:16:05.460+0000 7fceab72f700 1 -- 172.21.15.98:0/654370877 shutdown_connections 2024-03-28T05:16:05.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:16:05.460+0000 7fceab72f700 1 -- 172.21.15.98:0/654370877 wait complete. 2024-03-28T05:16:06.150 DEBUG:teuthology.orchestra.run.smithi098:osd.1> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.1.service 2024-03-28T05:16:06.153 INFO:tasks.cephadm:Deploying osd.2 on smithi113 with /dev/vg_nvme/lv_4... 2024-03-28T05:16:06.153 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T05:16:06.396 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T05:16:06.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:06 smithi113 ceph-mon[37164]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:16:06.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:06.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:06.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:06.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:06.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:06 smithi113 ceph-mon[37164]: from='osd.1 [v2:172.21.15.98:6800/555491234,v1:172.21.15.98:6801/555491234]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T05:16:06.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:06 smithi113 ceph-mon[37164]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T05:16:06.719 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:06 smithi098 ceph-mon[36087]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:16:06.719 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:06 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:06.719 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:06 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:06.719 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:06 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:06.719 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:06 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:06.719 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:06 smithi098 ceph-mon[36087]: from='osd.1 [v2:172.21.15.98:6800/555491234,v1:172.21.15.98:6801/555491234]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T05:16:06.719 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:06 smithi098 ceph-mon[36087]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T05:16:06.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:06 smithi023 ceph-mon[31954]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:16:06.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:06.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:06.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:06.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:06.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:06 smithi023 ceph-mon[31954]: from='osd.1 [v2:172.21.15.98:6800/555491234,v1:172.21.15.98:6801/555491234]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T05:16:06.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:06 smithi023 ceph-mon[31954]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T05:16:07.699 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:07 smithi098 ceph-mon[36087]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T05:16:07.699 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:07 smithi098 ceph-mon[36087]: from='osd.1 [v2:172.21.15.98:6800/555491234,v1:172.21.15.98:6801/555491234]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:16:07.699 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:07 smithi098 ceph-mon[36087]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T05:16:07.699 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:07 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T05:16:07.699 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:07 smithi098 ceph-mon[36087]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:16:07.778 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:07 smithi023 ceph-mon[31954]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T05:16:07.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:07 smithi023 ceph-mon[31954]: from='osd.1 [v2:172.21.15.98:6800/555491234,v1:172.21.15.98:6801/555491234]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:16:07.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:07 smithi023 ceph-mon[31954]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T05:16:07.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:07 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T05:16:07.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:07 smithi023 ceph-mon[31954]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:16:07.928 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:07 smithi113 ceph-mon[37164]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T05:16:07.928 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:07 smithi113 ceph-mon[37164]: from='osd.1 [v2:172.21.15.98:6800/555491234,v1:172.21.15.98:6801/555491234]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:16:07.928 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:07 smithi113 ceph-mon[37164]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T05:16:07.928 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:07 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T05:16:07.928 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:07 smithi113 ceph-mon[37164]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:16:08.530 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:16:08 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-1[40683]: 2024-03-28T05:16:08.227+0000 7f88b73e1700 -1 osd.1 0 waiting for initial osdmap 2024-03-28T05:16:08.530 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:16:08 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-1[40683]: 2024-03-28T05:16:08.231+0000 7f88ae9cb700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T05:16:08.586 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T05:16:08.586 INFO:teuthology.orchestra.run.smithi113.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T05:16:08.586 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10+0 records in 2024-03-28T05:16:08.586 INFO:teuthology.orchestra.run.smithi113.stderr:10+0 records out 2024-03-28T05:16:08.586 INFO:teuthology.orchestra.run.smithi113.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00722184 s, 1.5 GB/s 2024-03-28T05:16:08.586 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping successful for: 2024-03-28T05:16:08.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:08 smithi113 ceph-mon[37164]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:16:08.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:08 smithi113 ceph-mon[37164]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-28T05:16:08.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:08 smithi113 ceph-mon[37164]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T05:16:08.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:08 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T05:16:08.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:08 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T05:16:08.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:08 smithi113 ceph-mon[37164]: from='osd.1 ' entity='osd.1' 2024-03-28T05:16:08.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:08 smithi098 ceph-mon[36087]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:16:08.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:08 smithi098 ceph-mon[36087]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-28T05:16:08.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:08 smithi098 ceph-mon[36087]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T05:16:08.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T05:16:08.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T05:16:08.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:08 smithi098 ceph-mon[36087]: from='osd.1 ' entity='osd.1' 2024-03-28T05:16:08.967 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:08 smithi023 ceph-mon[31954]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:16:08.967 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:08 smithi023 ceph-mon[31954]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-28T05:16:08.967 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:08 smithi023 ceph-mon[31954]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T05:16:08.967 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:08 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T05:16:08.967 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:08 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T05:16:08.967 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:08 smithi023 ceph-mon[31954]: from='osd.1 ' entity='osd.1' 2024-03-28T05:16:09.227 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi113:vg_nvme/lv_4 2024-03-28T05:16:09.462 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T05:16:09.613 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:09 smithi098 ceph-mon[36087]: purged_snaps scrub starts 2024-03-28T05:16:09.613 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:09 smithi098 ceph-mon[36087]: purged_snaps scrub ok 2024-03-28T05:16:09.613 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T05:16:09.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:09 smithi098 ceph-mon[36087]: osd.1 [v2:172.21.15.98:6800/555491234,v1:172.21.15.98:6801/555491234] boot 2024-03-28T05:16:09.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:09 smithi098 ceph-mon[36087]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T05:16:09.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T05:16:09.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:09.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:09.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:16:09.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:09.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:09.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:16:09.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:09.812 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:09 smithi113 ceph-mon[37164]: purged_snaps scrub starts 2024-03-28T05:16:09.812 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:09 smithi113 ceph-mon[37164]: purged_snaps scrub ok 2024-03-28T05:16:09.812 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T05:16:09.812 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:09 smithi113 ceph-mon[37164]: osd.1 [v2:172.21.15.98:6800/555491234,v1:172.21.15.98:6801/555491234] boot 2024-03-28T05:16:09.812 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:09 smithi113 ceph-mon[37164]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T05:16:09.812 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T05:16:09.812 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:09.812 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:09.813 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:16:09.813 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:09.813 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:09.813 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:16:09.813 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:09.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:09 smithi023 ceph-mon[31954]: purged_snaps scrub starts 2024-03-28T05:16:09.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:09 smithi023 ceph-mon[31954]: purged_snaps scrub ok 2024-03-28T05:16:09.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T05:16:09.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:09 smithi023 ceph-mon[31954]: osd.1 [v2:172.21.15.98:6800/555491234,v1:172.21.15.98:6801/555491234] boot 2024-03-28T05:16:09.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:09 smithi023 ceph-mon[31954]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T05:16:09.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T05:16:09.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:09.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:09.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:16:09.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:09.946 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:09.947 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:16:09.947 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:10.740 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:10 smithi113 ceph-mon[37164]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:16:10.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:10 smithi113 ceph-mon[37164]: Detected new or changed devices on smithi098 2024-03-28T05:16:10.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:10 smithi113 ceph-mon[37164]: Adjusting osd_memory_target on smithi098 to 17080M 2024-03-28T05:16:10.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:16:10.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:10.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:16:10.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:10.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:10 smithi098 ceph-mon[36087]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:16:10.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:10 smithi098 ceph-mon[36087]: Detected new or changed devices on smithi098 2024-03-28T05:16:10.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:10 smithi098 ceph-mon[36087]: Adjusting osd_memory_target on smithi098 to 17080M 2024-03-28T05:16:10.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:16:10.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:10.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:16:10.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:10.990 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.991+0000 7fe15d482700 1 -- 172.21.15.113:0/1308387122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe158101100 msgr2=0x7fe158101520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:16:10.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.991+0000 7fe15d482700 1 --2- 172.21.15.113:0/1308387122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe158101100 0x7fe158101520 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe148009990 tx=0x7fe148009ca0 comp rx=0 tx=0).stop 2024-03-28T05:16:10.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.991+0000 7fe15d482700 1 -- 172.21.15.113:0/1308387122 shutdown_connections 2024-03-28T05:16:10.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.991+0000 7fe15d482700 1 --2- 172.21.15.113:0/1308387122 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe158102ca0 0x7fe158109550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:16:10.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.991+0000 7fe15d482700 1 --2- 172.21.15.113:0/1308387122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe158101100 0x7fe158101520 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:16:10.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.991+0000 7fe15d482700 1 --2- 172.21.15.113:0/1308387122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe158102300 0x7fe158102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:16:10.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.991+0000 7fe15d482700 1 -- 172.21.15.113:0/1308387122 >> 172.21.15.113:0/1308387122 conn(0x7fe1580fc680 msgr2=0x7fe1580feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:16:10.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.991+0000 7fe15d482700 1 -- 172.21.15.113:0/1308387122 shutdown_connections 2024-03-28T05:16:10.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.991+0000 7fe15d482700 1 -- 172.21.15.113:0/1308387122 wait complete. 2024-03-28T05:16:10.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.991+0000 7fe15d482700 1 Processor -- start 2024-03-28T05:16:10.991 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.991+0000 7fe15d482700 1 -- start start 2024-03-28T05:16:10.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe15d482700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe158101100 0x7fe15806b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:16:10.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe15d482700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe158102300 0x7fe15819d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:16:10.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe15d482700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe158102ca0 0x7fe15819f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:16:10.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe15d482700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15810cd40 con 0x7fe158102ca0 2024-03-28T05:16:10.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe15d482700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15810cbc0 con 0x7fe158101100 2024-03-28T05:16:10.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe15d482700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15810cec0 con 0x7fe158102300 2024-03-28T05:16:10.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe14ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe158102300 0x7fe15819d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:16:10.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe14ffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe158102300 0x7fe15819d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:36330/0 (socket says 172.21.15.113:36330) 2024-03-28T05:16:10.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe14ffff700 1 -- 172.21.15.113:0/2879797729 learned_addr learned my addr 172.21.15.113:0/2879797729 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-03-28T05:16:10.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe1577fe700 1 --2- 172.21.15.113:0/2879797729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe158102ca0 0x7fe15819f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:16:10.992 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe156ffd700 1 --2- 172.21.15.113:0/2879797729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe158101100 0x7fe15806b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:16:10.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe14ffff700 1 -- 172.21.15.113:0/2879797729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe158101100 msgr2=0x7fe15806b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:16:10.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe14ffff700 1 --2- 172.21.15.113:0/2879797729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe158101100 0x7fe15806b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:16:10.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe14ffff700 1 -- 172.21.15.113:0/2879797729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe158102ca0 msgr2=0x7fe15819f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:16:10.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe14ffff700 1 --2- 172.21.15.113:0/2879797729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe158102ca0 0x7fe15819f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:16:10.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe14ffff700 1 -- 172.21.15.113:0/2879797729 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1581a00e0 con 0x7fe158102300 2024-03-28T05:16:10.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe1577fe700 1 --2- 172.21.15.113:0/2879797729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe158102ca0 0x7fe15819f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:16:10.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe156ffd700 1 --2- 172.21.15.113:0/2879797729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe158101100 0x7fe15806b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:16:10.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe14ffff700 1 --2- 172.21.15.113:0/2879797729 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe158102300 0x7fe15819d1c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe14000b530 tx=0x7fe14000b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:16:10.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe154ff9700 1 -- 172.21.15.113:0/2879797729 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140015020 con 0x7fe158102300 2024-03-28T05:16:10.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe15d482700 1 -- 172.21.15.113:0/2879797729 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1581a03d0 con 0x7fe158102300 2024-03-28T05:16:10.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe154ff9700 1 -- 172.21.15.113:0/2879797729 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe140011d10 con 0x7fe158102300 2024-03-28T05:16:10.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe154ff9700 1 -- 172.21.15.113:0/2879797729 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140010e50 con 0x7fe158102300 2024-03-28T05:16:10.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.992+0000 7fe15d482700 1 -- 172.21.15.113:0/2879797729 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1581a0990 con 0x7fe158102300 2024-03-28T05:16:10.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.993+0000 7fe15d482700 1 -- 172.21.15.113:0/2879797729 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe158068e50 con 0x7fe158102300 2024-03-28T05:16:10.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.994+0000 7fe154ff9700 1 -- 172.21.15.113:0/2879797729 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7fe140018020 con 0x7fe158102300 2024-03-28T05:16:10.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.995+0000 7fe154ff9700 1 --2- 172.21.15.113:0/2879797729 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe1440a29a0 0x7fe1440a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:16:10.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.995+0000 7fe154ff9700 1 -- 172.21.15.113:0/2879797729 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7fe1400cca20 con 0x7fe158102300 2024-03-28T05:16:10.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.995+0000 7fe156ffd700 1 --2- 172.21.15.113:0/2879797729 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe1440a29a0 0x7fe1440a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:16:10.995 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:10.996+0000 7fe156ffd700 1 --2- 172.21.15.113:0/2879797729 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe1440a29a0 0x7fe1440a4e60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe1480039c0 tx=0x7fe148005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:16:10.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:11.000+0000 7fe154ff9700 1 -- 172.21.15.113:0/2879797729 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe14001f020 con 0x7fe158102300 2024-03-28T05:16:11.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:10 smithi023 ceph-mon[31954]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-28T05:16:11.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:10 smithi023 ceph-mon[31954]: Detected new or changed devices on smithi098 2024-03-28T05:16:11.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:10 smithi023 ceph-mon[31954]: Adjusting osd_memory_target on smithi098 to 17080M 2024-03-28T05:16:11.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:16:11.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:11.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:16:11.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:11.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:11.201+0000 7fe15d482700 1 -- 172.21.15.113:0/2879797729 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1580625d0 con 0x7fe1440a29a0 2024-03-28T05:16:11.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:11 smithi098 ceph-mon[36087]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T05:16:11.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:11 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:16:11.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:11 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:16:11.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:11 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:11.872 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:11 smithi113 ceph-mon[37164]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T05:16:11.872 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:11 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:16:11.872 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:11 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:16:11.872 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:11 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:12.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:11 smithi023 ceph-mon[31954]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T05:16:12.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:11 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:16:12.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:11 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:16:12.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:11 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:12.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:12 smithi098 ceph-mon[36087]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:12.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:12 smithi098 ceph-mon[36087]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:16:12.929 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:12 smithi113 ceph-mon[37164]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:12.929 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:12 smithi113 ceph-mon[37164]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:16:13.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:12 smithi023 ceph-mon[31954]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:13.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:12 smithi023 ceph-mon[31954]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:16:14.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:14 smithi098 ceph-mon[36087]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:14.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:14 smithi098 ceph-mon[36087]: from='client.? 172.21.15.113:0/2635757587' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "873bdd60-2a47-403b-b1f4-ee9226a95067"}]: dispatch 2024-03-28T05:16:14.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:14 smithi098 ceph-mon[36087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "873bdd60-2a47-403b-b1f4-ee9226a95067"}]: dispatch 2024-03-28T05:16:14.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:14 smithi098 ceph-mon[36087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "873bdd60-2a47-403b-b1f4-ee9226a95067"}]': finished 2024-03-28T05:16:14.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:14 smithi098 ceph-mon[36087]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T05:16:14.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:14 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T05:16:14.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:14 smithi098 ceph-mon[36087]: from='client.? 172.21.15.113:0/3093501121' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:16:14.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:14 smithi113 ceph-mon[37164]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:14.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:14 smithi113 ceph-mon[37164]: from='client.? 172.21.15.113:0/2635757587' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "873bdd60-2a47-403b-b1f4-ee9226a95067"}]: dispatch 2024-03-28T05:16:14.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:14 smithi113 ceph-mon[37164]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "873bdd60-2a47-403b-b1f4-ee9226a95067"}]: dispatch 2024-03-28T05:16:14.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:14 smithi113 ceph-mon[37164]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "873bdd60-2a47-403b-b1f4-ee9226a95067"}]': finished 2024-03-28T05:16:14.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:14 smithi113 ceph-mon[37164]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T05:16:14.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:14 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T05:16:14.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:14 smithi113 ceph-mon[37164]: from='client.? 172.21.15.113:0/3093501121' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:16:15.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:14 smithi023 ceph-mon[31954]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:15.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:14 smithi023 ceph-mon[31954]: from='client.? 172.21.15.113:0/2635757587' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "873bdd60-2a47-403b-b1f4-ee9226a95067"}]: dispatch 2024-03-28T05:16:15.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:14 smithi023 ceph-mon[31954]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "873bdd60-2a47-403b-b1f4-ee9226a95067"}]: dispatch 2024-03-28T05:16:15.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:14 smithi023 ceph-mon[31954]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "873bdd60-2a47-403b-b1f4-ee9226a95067"}]': finished 2024-03-28T05:16:15.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:14 smithi023 ceph-mon[31954]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T05:16:15.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:14 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T05:16:15.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:14 smithi023 ceph-mon[31954]: from='client.? 172.21.15.113:0/3093501121' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:16:16.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:16 smithi098 ceph-mon[36087]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:16.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:16 smithi113 ceph-mon[37164]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:17.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:16 smithi023 ceph-mon[31954]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:18.804 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:18 smithi113 ceph-mon[37164]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:18.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:18 smithi098 ceph-mon[36087]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:19.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:18 smithi023 ceph-mon[31954]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:20.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:20 smithi113 ceph-mon[37164]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:21.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:20 smithi023 ceph-mon[31954]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:21.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:20 smithi098 ceph-mon[36087]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:21.718 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:21 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T05:16:21.719 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:21 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:22.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:21 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T05:16:22.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:21 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:22.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:21 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T05:16:22.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:21 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:22.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:22 smithi113 ceph-mon[37164]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:22.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:22 smithi113 ceph-mon[37164]: Deploying daemon osd.2 on smithi113 2024-03-28T05:16:23.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:22 smithi023 ceph-mon[31954]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:23.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:22 smithi023 ceph-mon[31954]: Deploying daemon osd.2 on smithi113 2024-03-28T05:16:23.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:22 smithi098 ceph-mon[36087]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:23.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:22 smithi098 ceph-mon[36087]: Deploying daemon osd.2 on smithi113 2024-03-28T05:16:24.977 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:24 smithi113 ceph-mon[37164]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:25.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:24 smithi023 ceph-mon[31954]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:25.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:24 smithi098 ceph-mon[36087]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:26.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:26 smithi113 ceph-mon[37164]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:26.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:26 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:26.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:26 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:26.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:26 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:16:27.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:26 smithi023 ceph-mon[31954]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:27.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:26 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:27.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:26 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:27.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:26 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:16:27.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:26 smithi098 ceph-mon[36087]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:27.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:26 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:27.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:26 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:27.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:26 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:16:28.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:28 smithi113 ceph-mon[37164]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:28.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:28 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:28.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:28 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:28.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:28 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:28.995 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:28 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:16:28.995 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:28 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:29.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:28 smithi023 ceph-mon[31954]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:29.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:28 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:29.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:28 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:29.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:28 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:29.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:28 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:16:29.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:28 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:29.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:28 smithi098 ceph-mon[36087]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:29.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:28 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:29.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:28 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:29.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:28 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:29.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:28 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:16:29.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:28 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:29.262 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:29.265+0000 7fe154ff9700 1 -- 172.21.15.113:0/2879797729 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe1580625d0 con 0x7fe1440a29a0 2024-03-28T05:16:29.262 INFO:teuthology.orchestra.run.smithi113.stdout:Created osd(s) 2 on host 'smithi113' 2024-03-28T05:16:29.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:29.268+0000 7fe15d482700 1 -- 172.21.15.113:0/2879797729 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe1440a29a0 msgr2=0x7fe1440a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:16:29.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:29.268+0000 7fe15d482700 1 --2- 172.21.15.113:0/2879797729 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe1440a29a0 0x7fe1440a4e60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe1480039c0 tx=0x7fe148005fb0 comp rx=0 tx=0).stop 2024-03-28T05:16:29.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:29.268+0000 7fe15d482700 1 -- 172.21.15.113:0/2879797729 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe158102300 msgr2=0x7fe15819d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:16:29.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:29.268+0000 7fe15d482700 1 --2- 172.21.15.113:0/2879797729 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe158102300 0x7fe15819d1c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe14000b530 tx=0x7fe14000b8f0 comp rx=0 tx=0).stop 2024-03-28T05:16:29.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:29.268+0000 7fe15d482700 1 -- 172.21.15.113:0/2879797729 shutdown_connections 2024-03-28T05:16:29.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:29.268+0000 7fe15d482700 1 --2- 172.21.15.113:0/2879797729 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe1440a29a0 0x7fe1440a4e60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:16:29.265 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:29.268+0000 7fe15d482700 1 --2- 172.21.15.113:0/2879797729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe158102ca0 0x7fe15819f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:16:29.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:29.268+0000 7fe15d482700 1 --2- 172.21.15.113:0/2879797729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe158101100 0x7fe15806b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:16:29.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:29.268+0000 7fe15d482700 1 --2- 172.21.15.113:0/2879797729 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe158102300 0x7fe15819d1c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:16:29.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:29.268+0000 7fe15d482700 1 -- 172.21.15.113:0/2879797729 >> 172.21.15.113:0/2879797729 conn(0x7fe1580fc680 msgr2=0x7fe158105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:16:29.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:29.268+0000 7fe15d482700 1 -- 172.21.15.113:0/2879797729 shutdown_connections 2024-03-28T05:16:29.266 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:16:29.268+0000 7fe15d482700 1 -- 172.21.15.113:0/2879797729 wait complete. 2024-03-28T05:16:29.907 DEBUG:teuthology.orchestra.run.smithi113:osd.2> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.2.service 2024-03-28T05:16:29.909 INFO:tasks.cephadm:Deploying osd.3 on smithi023 with /dev/vg_nvme/lv_3... 2024-03-28T05:16:29.910 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T05:16:30.163 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:16:30.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:30 smithi113 ceph-mon[37164]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:30.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:30 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:30.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:30 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:16:30.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:30 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:30.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:30 smithi113 ceph-mon[37164]: from='osd.2 [v2:172.21.15.113:6800/1535856914,v1:172.21.15.113:6801/1535856914]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T05:16:30.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:30 smithi113 ceph-mon[37164]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T05:16:30.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:30 smithi023 ceph-mon[31954]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:30.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:30 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:30.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:30 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:16:30.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:30 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:30.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:30 smithi023 ceph-mon[31954]: from='osd.2 [v2:172.21.15.113:6800/1535856914,v1:172.21.15.113:6801/1535856914]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T05:16:30.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:30 smithi023 ceph-mon[31954]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T05:16:30.613 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:30 smithi098 ceph-mon[36087]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:30.613 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:30 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:30.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:30 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:16:30.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:30 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:30.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:30 smithi098 ceph-mon[36087]: from='osd.2 [v2:172.21.15.113:6800/1535856914,v1:172.21.15.113:6801/1535856914]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T05:16:30.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:30 smithi098 ceph-mon[36087]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T05:16:31.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:31 smithi023 ceph-mon[31954]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T05:16:31.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:31 smithi023 ceph-mon[31954]: from='osd.2 [v2:172.21.15.113:6800/1535856914,v1:172.21.15.113:6801/1535856914]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:16:31.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:31 smithi023 ceph-mon[31954]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T05:16:31.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:31 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T05:16:31.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:31 smithi023 ceph-mon[31954]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:16:31.353 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:31 smithi113 ceph-mon[37164]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T05:16:31.353 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:31 smithi113 ceph-mon[37164]: from='osd.2 [v2:172.21.15.113:6800/1535856914,v1:172.21.15.113:6801/1535856914]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:16:31.353 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:31 smithi113 ceph-mon[37164]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T05:16:31.353 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:31 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T05:16:31.353 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:31 smithi113 ceph-mon[37164]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:16:31.353 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:16:31 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-2[42296]: 2024-03-28T05:16:31.241+0000 7feb81829700 -1 osd.2 0 waiting for initial osdmap 2024-03-28T05:16:31.354 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:16:31 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-2[42296]: 2024-03-28T05:16:31.245+0000 7feb7d61c700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T05:16:31.613 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:31 smithi098 ceph-mon[36087]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T05:16:31.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:31 smithi098 ceph-mon[36087]: from='osd.2 [v2:172.21.15.113:6800/1535856914,v1:172.21.15.113:6801/1535856914]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:16:31.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:31 smithi098 ceph-mon[36087]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T05:16:31.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:31 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T05:16:31.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:31 smithi098 ceph-mon[36087]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:16:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T05:16:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T05:16:32.201 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-28T05:16:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-28T05:16:32.201 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00710479 s, 1.5 GB/s 2024-03-28T05:16:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-28T05:16:32.310 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:32 smithi023 ceph-mon[31954]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:32.310 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:32 smithi023 ceph-mon[31954]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-28T05:16:32.310 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:32 smithi023 ceph-mon[31954]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T05:16:32.310 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:32 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T05:16:32.310 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:32 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T05:16:32.488 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:32 smithi113 ceph-mon[37164]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:32.488 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:32 smithi113 ceph-mon[37164]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-28T05:16:32.488 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:32 smithi113 ceph-mon[37164]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T05:16:32.488 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:32 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T05:16:32.488 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:32 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T05:16:32.613 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:32 smithi098 ceph-mon[36087]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:32.613 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:32 smithi098 ceph-mon[36087]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-28T05:16:32.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:32 smithi098 ceph-mon[36087]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T05:16:32.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:32 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T05:16:32.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:32 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T05:16:32.954 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_3 2024-03-28T05:16:33.201 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:16:33.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:33 smithi023 ceph-mon[31954]: purged_snaps scrub starts 2024-03-28T05:16:33.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:33 smithi023 ceph-mon[31954]: purged_snaps scrub ok 2024-03-28T05:16:33.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:33 smithi023 ceph-mon[31954]: osd.2 [v2:172.21.15.113:6800/1535856914,v1:172.21.15.113:6801/1535856914] boot 2024-03-28T05:16:33.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:33 smithi023 ceph-mon[31954]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T05:16:33.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:33 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T05:16:33.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:33 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T05:16:33.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:33 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:33.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:33 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:33.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:33 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:16:33.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:33 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:33.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:33 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:33.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:33 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:16:33.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:33 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:33.613 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:33 smithi098 ceph-mon[36087]: purged_snaps scrub starts 2024-03-28T05:16:33.613 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:33 smithi098 ceph-mon[36087]: purged_snaps scrub ok 2024-03-28T05:16:33.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:33 smithi098 ceph-mon[36087]: osd.2 [v2:172.21.15.113:6800/1535856914,v1:172.21.15.113:6801/1535856914] boot 2024-03-28T05:16:33.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:33 smithi098 ceph-mon[36087]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T05:16:33.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:33 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T05:16:33.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:33 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T05:16:33.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:33 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:33.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:33 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:33.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:33 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:16:33.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:33 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:33.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:33 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:33.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:33 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:16:33.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:33 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:33.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:33 smithi113 ceph-mon[37164]: purged_snaps scrub starts 2024-03-28T05:16:33.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:33 smithi113 ceph-mon[37164]: purged_snaps scrub ok 2024-03-28T05:16:33.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:33 smithi113 ceph-mon[37164]: osd.2 [v2:172.21.15.113:6800/1535856914,v1:172.21.15.113:6801/1535856914] boot 2024-03-28T05:16:33.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:33 smithi113 ceph-mon[37164]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T05:16:33.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:33 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T05:16:33.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:33 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T05:16:33.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:33 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:33.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:33 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:33.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:33 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:16:33.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:33 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:33.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:33 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:33.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:33 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:16:33.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:33 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:34.394 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:34 smithi023 ceph-mon[31954]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:34.395 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:34 smithi023 ceph-mon[31954]: Detected new or changed devices on smithi113 2024-03-28T05:16:34.395 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:34 smithi023 ceph-mon[31954]: Adjusting osd_memory_target on smithi113 to 17080M 2024-03-28T05:16:34.395 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:34 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T05:16:34.395 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:34 smithi023 ceph-mon[31954]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T05:16:34.395 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:34 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T05:16:34.613 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:34 smithi098 ceph-mon[36087]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:34.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:34 smithi098 ceph-mon[36087]: Detected new or changed devices on smithi113 2024-03-28T05:16:34.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:34 smithi098 ceph-mon[36087]: Adjusting osd_memory_target on smithi113 to 17080M 2024-03-28T05:16:34.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:34 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T05:16:34.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:34 smithi098 ceph-mon[36087]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T05:16:34.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:34 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T05:16:34.645 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:16:34 smithi023 sudo[51021]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T05:16:34.645 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:16:34 smithi023 sudo[51021]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T05:16:34.645 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:16:34 smithi023 sudo[51021]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T05:16:34.645 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:16:34 smithi023 sudo[51021]: pam_unix(sudo:session): session closed for user root 2024-03-28T05:16:34.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:34 smithi113 ceph-mon[37164]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-28T05:16:34.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:34 smithi113 ceph-mon[37164]: Detected new or changed devices on smithi113 2024-03-28T05:16:34.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:34 smithi113 ceph-mon[37164]: Adjusting osd_memory_target on smithi113 to 17080M 2024-03-28T05:16:34.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:34 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T05:16:34.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:34 smithi113 ceph-mon[37164]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T05:16:34.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:34 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T05:16:34.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.888+0000 7f4748b16700 1 -- 172.21.15.23:0/4187111640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4744107f30 msgr2=0x7f474410a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:16:34.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.888+0000 7f4748b16700 1 --2- 172.21.15.23:0/4187111640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4744107f30 0x7f474410a320 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f473c00b0a0 tx=0x7f473c00b3b0 comp rx=0 tx=0).stop 2024-03-28T05:16:34.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.888+0000 7f4748b16700 1 -- 172.21.15.23:0/4187111640 shutdown_connections 2024-03-28T05:16:34.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.888+0000 7f4748b16700 1 --2- 172.21.15.23:0/4187111640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4744107f30 0x7f474410a320 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:16:34.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.888+0000 7f4748b16700 1 --2- 172.21.15.23:0/4187111640 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4744102cd0 0x7f47441050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:16:34.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.888+0000 7f4748b16700 1 --2- 172.21.15.23:0/4187111640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4744105600 0x7f47441079f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:16:34.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.888+0000 7f4748b16700 1 -- 172.21.15.23:0/4187111640 >> 172.21.15.23:0/4187111640 conn(0x7f47440fc680 msgr2=0x7f47440feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:16:34.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.888+0000 7f4748b16700 1 -- 172.21.15.23:0/4187111640 shutdown_connections 2024-03-28T05:16:34.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.888+0000 7f4748b16700 1 -- 172.21.15.23:0/4187111640 wait complete. 2024-03-28T05:16:34.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.889+0000 7f4748b16700 1 Processor -- start 2024-03-28T05:16:34.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.889+0000 7f4748b16700 1 -- start start 2024-03-28T05:16:34.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.889+0000 7f4748b16700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4744102cd0 0x7f4744198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:16:34.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.889+0000 7f4748b16700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4744105600 0x7f47441988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:16:34.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.889+0000 7f4748b16700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4744107f30 0x7f474419f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:16:34.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.889+0000 7f4748b16700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474410ddd0 con 0x7f4744102cd0 2024-03-28T05:16:34.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.889+0000 7f4748b16700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474410dc50 con 0x7f4744107f30 2024-03-28T05:16:34.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.889+0000 7f4748b16700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474410df50 con 0x7f4744105600 2024-03-28T05:16:34.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.889+0000 7f474259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4744102cd0 0x7f4744198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:16:34.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.889+0000 7f474259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4744102cd0 0x7f4744198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50864/0 (socket says 172.21.15.23:50864) 2024-03-28T05:16:34.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.889+0000 7f474259c700 1 -- 172.21.15.23:0/1115696617 learned_addr learned my addr 172.21.15.23:0/1115696617 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:16:34.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.889+0000 7f4742d9d700 1 --2- 172.21.15.23:0/1115696617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4744107f30 0x7f474419f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:16:34.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.889+0000 7f4741d9b700 1 --2- 172.21.15.23:0/1115696617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4744105600 0x7f47441988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:16:34.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.889+0000 7f474259c700 1 -- 172.21.15.23:0/1115696617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4744105600 msgr2=0x7f47441988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:16:34.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.889+0000 7f474259c700 1 --2- 172.21.15.23:0/1115696617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4744105600 0x7f47441988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:16:34.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.889+0000 7f474259c700 1 -- 172.21.15.23:0/1115696617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4744107f30 msgr2=0x7f474419f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:16:34.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.889+0000 7f474259c700 1 --2- 172.21.15.23:0/1115696617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4744107f30 0x7f474419f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:16:34.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.889+0000 7f474259c700 1 -- 172.21.15.23:0/1115696617 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474419ffd0 con 0x7f4744102cd0 2024-03-28T05:16:34.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.890+0000 7f4741d9b700 1 --2- 172.21.15.23:0/1115696617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4744105600 0x7f47441988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:16:34.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.890+0000 7f474259c700 1 --2- 172.21.15.23:0/1115696617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4744102cd0 0x7f4744198370 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f473800c9a0 tx=0x7f473800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:16:34.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.890+0000 7f47337fe700 1 -- 172.21.15.23:0/1115696617 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4738007a60 con 0x7f4744102cd0 2024-03-28T05:16:34.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.890+0000 7f47337fe700 1 -- 172.21.15.23:0/1115696617 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4738004d10 con 0x7f4744102cd0 2024-03-28T05:16:34.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.890+0000 7f4748b16700 1 -- 172.21.15.23:0/1115696617 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47441a02c0 con 0x7f4744102cd0 2024-03-28T05:16:34.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.890+0000 7f4748b16700 1 -- 172.21.15.23:0/1115696617 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47441a07b0 con 0x7f4744102cd0 2024-03-28T05:16:34.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.890+0000 7f47337fe700 1 -- 172.21.15.23:0/1115696617 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4738005690 con 0x7f4744102cd0 2024-03-28T05:16:34.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.891+0000 7f4748b16700 1 -- 172.21.15.23:0/1115696617 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4744068e50 con 0x7f4744102cd0 2024-03-28T05:16:34.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.892+0000 7f47337fe700 1 -- 172.21.15.23:0/1115696617 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f4738007c00 con 0x7f4744102cd0 2024-03-28T05:16:34.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.892+0000 7f47337fe700 1 --2- 172.21.15.23:0/1115696617 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f472c0a2800 0x7f472c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:16:34.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.893+0000 7f4741d9b700 1 --2- 172.21.15.23:0/1115696617 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f472c0a2800 0x7f472c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:16:34.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.893+0000 7f47337fe700 1 -- 172.21.15.23:0/1115696617 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3237+0+0 (secure 0 0 0) 0x7f47380cd320 con 0x7f4744102cd0 2024-03-28T05:16:34.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.893+0000 7f4741d9b700 1 --2- 172.21.15.23:0/1115696617 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f472c0a2800 0x7f472c0a4cc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f473400bc10 tx=0x7f473400b5f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:16:34.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:34.897+0000 7f47337fe700 1 -- 172.21.15.23:0/1115696617 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f473809b900 con 0x7f4744102cd0 2024-03-28T05:16:35.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:35.119+0000 7f4748b16700 1 -- 172.21.15.23:0/1115696617 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f47440625d0 con 0x7f472c0a2800 2024-03-28T05:16:35.278 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:16:34 smithi023 sudo[51050]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T05:16:35.279 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:16:34 smithi023 sudo[51050]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T05:16:35.279 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:16:34 smithi023 sudo[51050]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T05:16:35.279 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:16:34 smithi023 sudo[51050]: pam_unix(sudo:session): session closed for user root 2024-03-28T05:16:35.425 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:16:35 smithi098 sudo[44400]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T05:16:35.425 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:16:35 smithi098 sudo[44400]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T05:16:35.425 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:16:35 smithi098 sudo[44400]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T05:16:35.425 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:16:35 smithi098 sudo[44400]: pam_unix(sudo:session): session closed for user root 2024-03-28T05:16:35.425 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:35 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T05:16:35.426 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:35 smithi098 ceph-mon[36087]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T05:16:35.426 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:35 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:35.426 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:35 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:16:35.426 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:35 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:16:35.426 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:35 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:35.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:35 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T05:16:35.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:35 smithi023 ceph-mon[31954]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T05:16:35.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:35 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:35.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:35 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:16:35.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:35 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:16:35.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:35 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:35.691 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:35 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T05:16:35.691 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:35 smithi113 ceph-mon[37164]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T05:16:35.691 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:35 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:35.692 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:35 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:16:35.692 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:35 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:16:35.692 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:35 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:35.863 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:16:35 smithi098 sudo[44403]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T05:16:35.863 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:16:35 smithi098 sudo[44403]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T05:16:35.863 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:16:35 smithi098 sudo[44403]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T05:16:35.864 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:16:35 smithi098 sudo[44403]: pam_unix(sudo:session): session closed for user root 2024-03-28T05:16:35.941 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:16:35 smithi113 sudo[46078]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T05:16:35.942 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:16:35 smithi113 sudo[46078]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T05:16:35.942 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:16:35 smithi113 sudo[46078]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T05:16:35.942 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:16:35 smithi113 sudo[46078]: pam_unix(sudo:session): session closed for user root 2024-03-28T05:16:35.942 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:16:35 smithi113 sudo[46081]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T05:16:35.942 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:16:35 smithi113 sudo[46081]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T05:16:35.942 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:16:35 smithi113 sudo[46081]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T05:16:36.241 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:16:35 smithi113 sudo[46081]: pam_unix(sudo:session): session closed for user root 2024-03-28T05:16:36.484 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:36 smithi023 sudo[51194]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T05:16:36.484 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:36 smithi023 sudo[51194]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T05:16:36.485 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:36 smithi023 sudo[51194]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T05:16:36.485 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:36 smithi023 sudo[51194]: pam_unix(sudo:session): session closed for user root 2024-03-28T05:16:36.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:36.504+0000 7f47337fe700 1 -- 172.21.15.23:0/1115696617 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4738097870 con 0x7f4744102cd0 2024-03-28T05:16:36.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:36 smithi113 ceph-mon[37164]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T05:16:36.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:36 smithi113 ceph-mon[37164]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:16:36.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:36 smithi113 ceph-mon[37164]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T05:16:36.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:36 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:36.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:36 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:36.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:36 smithi113 ceph-mon[37164]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T05:16:36.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:36 smithi113 sudo[46084]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T05:16:36.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:36 smithi113 sudo[46084]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T05:16:36.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:36 smithi113 sudo[46084]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T05:16:36.760 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:36 smithi023 sudo[51198]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T05:16:36.760 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:36 smithi023 sudo[51198]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T05:16:36.760 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:36 smithi023 sudo[51198]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T05:16:36.760 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:36 smithi023 sudo[51198]: pam_unix(sudo:session): session closed for user root 2024-03-28T05:16:36.760 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:36 smithi023 ceph-mon[31954]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T05:16:36.761 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:36 smithi023 ceph-mon[31954]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:16:36.761 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:36 smithi023 ceph-mon[31954]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T05:16:36.761 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:36 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:36.761 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:36 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:36.761 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:36 smithi023 ceph-mon[31954]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T05:16:36.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:36 smithi098 ceph-mon[36087]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-28T05:16:36.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:36 smithi098 ceph-mon[36087]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:16:36.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:36 smithi098 ceph-mon[36087]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T05:16:36.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:36 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:36.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:36 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:36.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:36 smithi098 ceph-mon[36087]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T05:16:37.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:36 smithi113 sudo[46084]: pam_unix(sudo:session): session closed for user root 2024-03-28T05:16:37.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:36 smithi113 sudo[46087]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T05:16:37.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:36 smithi113 sudo[46087]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T05:16:37.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:36 smithi113 sudo[46087]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T05:16:37.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:36 smithi113 sudo[46087]: pam_unix(sudo:session): session closed for user root 2024-03-28T05:16:37.478 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 sudo[44406]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T05:16:37.478 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 sudo[44406]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T05:16:37.479 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 sudo[44406]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T05:16:37.479 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 sudo[44406]: pam_unix(sudo:session): session closed for user root 2024-03-28T05:16:37.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:37 smithi023 ceph-mon[31954]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T05:16:37.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:37 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T05:16:37.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:37 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:16:37.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:37 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:16:37.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:37 smithi023 ceph-mon[31954]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T05:16:37.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:37 smithi023 ceph-mon[31954]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T05:16:37.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:37 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T05:16:37.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:37 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:16:37.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:37 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:16:37.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:37 smithi023 ceph-mon[31954]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T05:16:37.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:37 smithi023 ceph-mon[31954]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T05:16:37.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:37 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T05:16:37.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:37 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:16:37.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:37 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:16:37.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:37 smithi023 ceph-mon[31954]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T05:16:37.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 sudo[44409]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T05:16:37.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 sudo[44409]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T05:16:37.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 sudo[44409]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T05:16:37.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 sudo[44409]: pam_unix(sudo:session): session closed for user root 2024-03-28T05:16:37.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 ceph-mon[36087]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T05:16:37.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T05:16:37.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:16:37.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:16:37.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 ceph-mon[36087]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T05:16:37.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 ceph-mon[36087]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T05:16:37.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T05:16:37.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:16:37.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:16:37.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 ceph-mon[36087]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T05:16:37.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 ceph-mon[36087]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T05:16:37.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T05:16:37.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:16:37.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:16:37.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:37 smithi098 ceph-mon[36087]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T05:16:37.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:37 smithi113 ceph-mon[37164]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T05:16:37.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:37 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T05:16:37.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:37 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:16:37.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:37 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:16:37.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:37 smithi113 ceph-mon[37164]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T05:16:37.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:37 smithi113 ceph-mon[37164]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T05:16:37.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:37 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T05:16:37.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:37 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:16:37.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:37 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:16:37.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:37 smithi113 ceph-mon[37164]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T05:16:37.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:37 smithi113 ceph-mon[37164]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T05:16:37.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:37 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T05:16:37.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:37 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:16:37.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:37 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:16:37.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:37 smithi113 ceph-mon[37164]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T05:16:38.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:38 smithi023 ceph-mon[31954]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-28T05:16:38.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:38 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/63406494' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6"}]: dispatch 2024-03-28T05:16:38.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:38 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/63406494' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6"}]': finished 2024-03-28T05:16:38.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:38 smithi023 ceph-mon[31954]: osdmap e22: 4 total, 3 up, 4 in 2024-03-28T05:16:38.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:38 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T05:16:38.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:38 smithi098 ceph-mon[36087]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-28T05:16:38.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:38 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/63406494' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6"}]: dispatch 2024-03-28T05:16:38.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:38 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/63406494' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6"}]': finished 2024-03-28T05:16:38.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:38 smithi098 ceph-mon[36087]: osdmap e22: 4 total, 3 up, 4 in 2024-03-28T05:16:38.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:38 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T05:16:38.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:38 smithi113 ceph-mon[37164]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-28T05:16:38.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:38 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/63406494' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6"}]: dispatch 2024-03-28T05:16:38.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:38 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/63406494' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6"}]': finished 2024-03-28T05:16:38.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:38 smithi113 ceph-mon[37164]: osdmap e22: 4 total, 3 up, 4 in 2024-03-28T05:16:38.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:38 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T05:16:39.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:39 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2123344978' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:16:39.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:39 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2123344978' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:16:39.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:39 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2123344978' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:16:40.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:40 smithi023 ceph-mon[31954]: pgmap v87: 1 pgs: 1 active+clean; 449 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-28T05:16:40.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:40 smithi098 ceph-mon[36087]: pgmap v87: 1 pgs: 1 active+clean; 449 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-28T05:16:40.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:40 smithi113 ceph-mon[37164]: pgmap v87: 1 pgs: 1 active+clean; 449 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-28T05:16:42.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:42 smithi023 ceph-mon[31954]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 305 KiB/s wr, 20 op/s 2024-03-28T05:16:42.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:42 smithi098 ceph-mon[36087]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 305 KiB/s wr, 20 op/s 2024-03-28T05:16:42.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:42 smithi113 ceph-mon[37164]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 305 KiB/s wr, 20 op/s 2024-03-28T05:16:44.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:44 smithi023 ceph-mon[31954]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 228 KiB/s wr, 15 op/s 2024-03-28T05:16:44.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:44 smithi098 ceph-mon[36087]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 228 KiB/s wr, 15 op/s 2024-03-28T05:16:44.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:44 smithi113 ceph-mon[37164]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 228 KiB/s wr, 15 op/s 2024-03-28T05:16:45.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:45 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T05:16:45.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:45 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:45.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:45 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T05:16:45.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:45 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:45.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:45 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T05:16:45.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:45 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:16:46.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:46 smithi023 ceph-mon[31954]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.3 KiB/s rd, 195 KiB/s wr, 13 op/s 2024-03-28T05:16:46.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:46 smithi023 ceph-mon[31954]: Deploying daemon osd.3 on smithi023 2024-03-28T05:16:46.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:46 smithi098 ceph-mon[36087]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.3 KiB/s rd, 195 KiB/s wr, 13 op/s 2024-03-28T05:16:46.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:46 smithi098 ceph-mon[36087]: Deploying daemon osd.3 on smithi023 2024-03-28T05:16:46.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:46 smithi113 ceph-mon[37164]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.3 KiB/s rd, 195 KiB/s wr, 13 op/s 2024-03-28T05:16:46.990 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:46 smithi113 ceph-mon[37164]: Deploying daemon osd.3 on smithi023 2024-03-28T05:16:48.766 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:48 smithi023 ceph-mon[31954]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-28T05:16:48.793 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:48 smithi113 ceph-mon[37164]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-28T05:16:48.863 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:48 smithi098 ceph-mon[36087]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-28T05:16:50.820 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:50 smithi023 ceph-mon[31954]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 170 KiB/s wr, 11 op/s 2024-03-28T05:16:50.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:50 smithi098 ceph-mon[36087]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 170 KiB/s wr, 11 op/s 2024-03-28T05:16:50.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:50 smithi113 ceph-mon[37164]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 170 KiB/s wr, 11 op/s 2024-03-28T05:16:52.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:51 smithi113 ceph-mon[37164]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-28T05:16:52.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:51 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:52.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:51 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:52.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:51 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:16:52.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:51 smithi023 ceph-mon[31954]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-28T05:16:52.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:51 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:52.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:51 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:52.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:51 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:16:52.364 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:51 smithi098 ceph-mon[36087]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-28T05:16:52.364 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:51 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:52.364 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:51 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:52.364 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:51 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:16:54.229 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:53 smithi023 ceph-mon[31954]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-28T05:16:54.240 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:53 smithi113 ceph-mon[37164]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-28T05:16:54.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:54.289+0000 7f47337fe700 1 -- 172.21.15.23:0/1115696617 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f47440625d0 con 0x7f472c0a2800 2024-03-28T05:16:54.286 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 3 on host 'smithi023' 2024-03-28T05:16:54.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:54.291+0000 7f4748b16700 1 -- 172.21.15.23:0/1115696617 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f472c0a2800 msgr2=0x7f472c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:16:54.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:54.291+0000 7f4748b16700 1 --2- 172.21.15.23:0/1115696617 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f472c0a2800 0x7f472c0a4cc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f473400bc10 tx=0x7f473400b5f0 comp rx=0 tx=0).stop 2024-03-28T05:16:54.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:54.291+0000 7f4748b16700 1 -- 172.21.15.23:0/1115696617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4744102cd0 msgr2=0x7f4744198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:16:54.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:54.291+0000 7f4748b16700 1 --2- 172.21.15.23:0/1115696617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4744102cd0 0x7f4744198370 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f473800c9a0 tx=0x7f473800ccb0 comp rx=0 tx=0).stop 2024-03-28T05:16:54.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:54.291+0000 7f4748b16700 1 -- 172.21.15.23:0/1115696617 shutdown_connections 2024-03-28T05:16:54.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:54.291+0000 7f4748b16700 1 --2- 172.21.15.23:0/1115696617 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f472c0a2800 0x7f472c0a4cc0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:16:54.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:54.291+0000 7f4748b16700 1 --2- 172.21.15.23:0/1115696617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4744107f30 0x7f474419f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:16:54.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:54.291+0000 7f4748b16700 1 --2- 172.21.15.23:0/1115696617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4744102cd0 0x7f4744198370 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:16:54.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:54.291+0000 7f4748b16700 1 --2- 172.21.15.23:0/1115696617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4744105600 0x7f47441988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:16:54.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:54.291+0000 7f4748b16700 1 -- 172.21.15.23:0/1115696617 >> 172.21.15.23:0/1115696617 conn(0x7f47440fc680 msgr2=0x7f47441035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:16:54.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:54.291+0000 7f4748b16700 1 -- 172.21.15.23:0/1115696617 shutdown_connections 2024-03-28T05:16:54.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:16:54.291+0000 7f4748b16700 1 -- 172.21.15.23:0/1115696617 wait complete. 2024-03-28T05:16:54.364 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:53 smithi098 ceph-mon[36087]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-28T05:16:54.940 DEBUG:teuthology.orchestra.run.smithi023:osd.3> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.3.service 2024-03-28T05:16:54.942 INFO:tasks.cephadm:Deploying osd.4 on smithi098 with /dev/vg_nvme/lv_3... 2024-03-28T05:16:54.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T05:16:55.186 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T05:16:55.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:55 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:55.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:55 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:55.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:55 smithi023 ceph-mon[31954]: from='osd.3 [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T05:16:55.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:55 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:55.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:55 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:55.579 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:55 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:55.579 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:55 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:55.579 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:55 smithi098 ceph-mon[36087]: from='osd.3 [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T05:16:55.579 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:55 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:55.579 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:55 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:55.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:55 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:55.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:55 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:55.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:55 smithi113 ceph-mon[37164]: from='osd.3 [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T05:16:55.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:55 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:55.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:55 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:16:56.569 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:56 smithi098 ceph-mon[36087]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-28T05:16:56.569 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:56 smithi098 ceph-mon[36087]: from='osd.3 [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T05:16:56.569 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:56 smithi098 ceph-mon[36087]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T05:16:56.569 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:56 smithi098 ceph-mon[36087]: from='osd.3 [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T05:16:56.569 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T05:16:56.699 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:56 smithi023 ceph-mon[31954]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-28T05:16:56.699 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:56 smithi023 ceph-mon[31954]: from='osd.3 [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T05:16:56.699 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:56 smithi023 ceph-mon[31954]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T05:16:56.699 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:56 smithi023 ceph-mon[31954]: from='osd.3 [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T05:16:56.699 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T05:16:56.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:56 smithi113 ceph-mon[37164]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-28T05:16:56.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:56 smithi113 ceph-mon[37164]: from='osd.3 [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T05:16:56.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:56 smithi113 ceph-mon[37164]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T05:16:56.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:56 smithi113 ceph-mon[37164]: from='osd.3 [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T05:16:56.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T05:16:57.279 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 05:16:57 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-3[53300]: 2024-03-28T05:16:57.034+0000 7f5c46be2700 -1 osd.3 0 waiting for initial osdmap 2024-03-28T05:16:57.280 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 05:16:57 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-3[53300]: 2024-03-28T05:16:57.039+0000 7f5c3e1cc700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T05:16:57.485 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T05:16:57.485 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T05:16:57.485 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-28T05:16:57.485 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-28T05:16:57.485 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112431 s, 933 MB/s 2024-03-28T05:16:57.485 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-28T05:16:57.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:57 smithi098 ceph-mon[36087]: from='osd.3 [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T05:16:57.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:57 smithi098 ceph-mon[36087]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T05:16:57.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:57 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T05:16:57.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:57 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T05:16:57.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:57 smithi098 ceph-mon[36087]: from='osd.3 [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020]' entity='osd.3' 2024-03-28T05:16:57.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:57 smithi098 ceph-mon[36087]: osd.3 [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020] boot 2024-03-28T05:16:57.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:57 smithi098 ceph-mon[36087]: osdmap e25: 4 total, 4 up, 4 in 2024-03-28T05:16:57.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:57 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T05:16:57.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:57 smithi113 ceph-mon[37164]: from='osd.3 [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T05:16:57.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:57 smithi113 ceph-mon[37164]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T05:16:57.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:57 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T05:16:57.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:57 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T05:16:57.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:57 smithi113 ceph-mon[37164]: from='osd.3 [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020]' entity='osd.3' 2024-03-28T05:16:57.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:57 smithi113 ceph-mon[37164]: osd.3 [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020] boot 2024-03-28T05:16:57.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:57 smithi113 ceph-mon[37164]: osdmap e25: 4 total, 4 up, 4 in 2024-03-28T05:16:57.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:57 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T05:16:57.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:57 smithi023 ceph-mon[31954]: from='osd.3 [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T05:16:57.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:57 smithi023 ceph-mon[31954]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T05:16:57.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:57 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T05:16:57.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:57 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T05:16:57.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:57 smithi023 ceph-mon[31954]: from='osd.3 [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020]' entity='osd.3' 2024-03-28T05:16:57.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:57 smithi023 ceph-mon[31954]: osd.3 [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020] boot 2024-03-28T05:16:57.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:57 smithi023 ceph-mon[31954]: osdmap e25: 4 total, 4 up, 4 in 2024-03-28T05:16:57.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:57 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T05:16:58.173 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_3 2024-03-28T05:16:58.412 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T05:16:58.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:58 smithi098 ceph-mon[36087]: purged_snaps scrub starts 2024-03-28T05:16:58.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:58 smithi098 ceph-mon[36087]: purged_snaps scrub ok 2024-03-28T05:16:58.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:58 smithi098 ceph-mon[36087]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-28T05:16:58.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:58 smithi098 ceph-mon[36087]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T05:16:58.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:58 smithi113 ceph-mon[37164]: purged_snaps scrub starts 2024-03-28T05:16:58.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:58 smithi113 ceph-mon[37164]: purged_snaps scrub ok 2024-03-28T05:16:58.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:58 smithi113 ceph-mon[37164]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-28T05:16:58.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:58 smithi113 ceph-mon[37164]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T05:16:58.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:58 smithi023 ceph-mon[31954]: purged_snaps scrub starts 2024-03-28T05:16:58.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:58 smithi023 ceph-mon[31954]: purged_snaps scrub ok 2024-03-28T05:16:58.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:58 smithi023 ceph-mon[31954]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-28T05:16:58.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:58 smithi023 ceph-mon[31954]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T05:17:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.027+0000 7f6562a07700 1 -- 172.21.15.98:0/68026712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f655c102c60 msgr2=0x7f655c109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:17:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.027+0000 7f6562a07700 1 --2- 172.21.15.98:0/68026712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f655c102c60 0x7f655c109510 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f655400b0d0 tx=0x7f655400b3e0 comp rx=0 tx=0).stop 2024-03-28T05:17:00.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.027+0000 7f6562a07700 1 -- 172.21.15.98:0/68026712 shutdown_connections 2024-03-28T05:17:00.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.027+0000 7f6562a07700 1 --2- 172.21.15.98:0/68026712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f655c102c60 0x7f655c109510 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:00.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.027+0000 7f6562a07700 1 --2- 172.21.15.98:0/68026712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f655c1010c0 0x7f655c1014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:00.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.027+0000 7f6562a07700 1 --2- 172.21.15.98:0/68026712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f655c1022c0 0x7f655c102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:00.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.027+0000 7f6562a07700 1 -- 172.21.15.98:0/68026712 >> 172.21.15.98:0/68026712 conn(0x7f655c0fc660 msgr2=0x7f655c0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:17:00.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.027+0000 7f6562a07700 1 -- 172.21.15.98:0/68026712 shutdown_connections 2024-03-28T05:17:00.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.027+0000 7f6562a07700 1 -- 172.21.15.98:0/68026712 wait complete. 2024-03-28T05:17:00.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.027+0000 7f6562a07700 1 Processor -- start 2024-03-28T05:17:00.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.027+0000 7f6562a07700 1 -- start start 2024-03-28T05:17:00.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.028+0000 7f6562a07700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f655c1010c0 0x7f655c198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:17:00.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.028+0000 7f6562a07700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f655c1022c0 0x7f655c198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:17:00.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.028+0000 7f6562a07700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f655c102c60 0x7f655c19f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:17:00.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.028+0000 7f6562a07700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655c10ccb0 con 0x7f655c1010c0 2024-03-28T05:17:00.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.028+0000 7f6562a07700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655c10cb30 con 0x7f655c1022c0 2024-03-28T05:17:00.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.028+0000 7f6562a07700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655c10ce30 con 0x7f655c102c60 2024-03-28T05:17:00.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.028+0000 7f655b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f655c1022c0 0x7f655c198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:17:00.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.028+0000 7f655b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f655c1022c0 0x7f655c198940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53068/0 (socket says 172.21.15.98:53068) 2024-03-28T05:17:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.028+0000 7f655b7fe700 1 -- 172.21.15.98:0/177329569 learned_addr learned my addr 172.21.15.98:0/177329569 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-28T05:17:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.028+0000 7f6560fa4700 1 --2- 172.21.15.98:0/177329569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f655c102c60 0x7f655c19f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:17:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.028+0000 7f655bfff700 1 --2- 172.21.15.98:0/177329569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f655c1010c0 0x7f655c198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:17:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.028+0000 7f655b7fe700 1 -- 172.21.15.98:0/177329569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f655c102c60 msgr2=0x7f655c19f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:17:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.028+0000 7f655b7fe700 1 --2- 172.21.15.98:0/177329569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f655c102c60 0x7f655c19f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.028+0000 7f655b7fe700 1 -- 172.21.15.98:0/177329569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f655c1010c0 msgr2=0x7f655c198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:17:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.028+0000 7f655b7fe700 1 --2- 172.21.15.98:0/177329569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f655c1010c0 0x7f655c198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.028+0000 7f655b7fe700 1 -- 172.21.15.98:0/177329569 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f655c1a0010 con 0x7f655c1022c0 2024-03-28T05:17:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.028+0000 7f6560fa4700 1 --2- 172.21.15.98:0/177329569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f655c102c60 0x7f655c19f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:17:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.028+0000 7f655bfff700 1 --2- 172.21.15.98:0/177329569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f655c1010c0 0x7f655c198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:17:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.028+0000 7f655b7fe700 1 --2- 172.21.15.98:0/177329569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f655c1022c0 0x7f655c198940 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f655000c970 tx=0x7f655000cd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:17:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.028+0000 7f65597fa700 1 -- 172.21.15.98:0/177329569 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6550007aa0 con 0x7f655c1022c0 2024-03-28T05:17:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.029+0000 7f6562a07700 1 -- 172.21.15.98:0/177329569 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f655c1a0300 con 0x7f655c1022c0 2024-03-28T05:17:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.029+0000 7f65597fa700 1 -- 172.21.15.98:0/177329569 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6550004d10 con 0x7f655c1022c0 2024-03-28T05:17:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.029+0000 7f65597fa700 1 -- 172.21.15.98:0/177329569 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655000f450 con 0x7f655c1022c0 2024-03-28T05:17:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.029+0000 7f6562a07700 1 -- 172.21.15.98:0/177329569 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f655c1a0840 con 0x7f655c1022c0 2024-03-28T05:17:00.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.030+0000 7f6562a07700 1 -- 172.21.15.98:0/177329569 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f655c068e50 con 0x7f655c1022c0 2024-03-28T05:17:00.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.031+0000 7f65597fa700 1 -- 172.21.15.98:0/177329569 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6550004830 con 0x7f655c1022c0 2024-03-28T05:17:00.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.031+0000 7f65597fa700 1 --2- 172.21.15.98:0/177329569 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f65440a2890 0x7f65440a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:17:00.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.031+0000 7f65597fa700 1 -- 172.21.15.98:0/177329569 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 3698+0+0 (secure 0 0 0) 0x7f65500cd6f0 con 0x7f655c1022c0 2024-03-28T05:17:00.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.031+0000 7f655bfff700 1 --2- 172.21.15.98:0/177329569 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f65440a2890 0x7f65440a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:17:00.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.032+0000 7f655bfff700 1 --2- 172.21.15.98:0/177329569 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f65440a2890 0x7f65440a4d50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f654c005950 tx=0x7f654c0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:17:00.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.036+0000 7f65597fa700 1 -- 172.21.15.98:0/177329569 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f655009ba20 con 0x7f655c1022c0 2024-03-28T05:17:00.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:59 smithi098 ceph-mon[36087]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 516 MiB used, 357 GiB / 358 GiB avail 2024-03-28T05:17:00.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:59 smithi098 ceph-mon[36087]: Detected new or changed devices on smithi023 2024-03-28T05:17:00.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:59 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:00.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:59 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:00.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:59 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:17:00.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:59 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:17:00.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:59 smithi098 ceph-mon[36087]: Adjusting osd_memory_target on smithi023 to 8544M 2024-03-28T05:17:00.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:59 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:00.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:59 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:00.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:59 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:17:00.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:59 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:00.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:59 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:17:00.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:59 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:00.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:59 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:17:00.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:59 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:00.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:16:59 smithi098 ceph-mon[36087]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T05:17:00.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:00.238+0000 7f6562a07700 1 -- 172.21.15.98:0/177329569 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f655c0fe850 con 0x7f65440a2890 2024-03-28T05:17:00.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:59 smithi113 ceph-mon[37164]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 516 MiB used, 357 GiB / 358 GiB avail 2024-03-28T05:17:00.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:59 smithi113 ceph-mon[37164]: Detected new or changed devices on smithi023 2024-03-28T05:17:00.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:59 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:00.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:59 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:00.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:59 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:17:00.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:59 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:17:00.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:59 smithi113 ceph-mon[37164]: Adjusting osd_memory_target on smithi023 to 8544M 2024-03-28T05:17:00.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:59 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:00.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:59 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:00.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:59 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:17:00.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:59 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:00.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:59 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:17:00.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:59 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:00.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:59 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:17:00.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:59 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:00.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:16:59 smithi113 ceph-mon[37164]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T05:17:00.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:59 smithi023 ceph-mon[31954]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 516 MiB used, 357 GiB / 358 GiB avail 2024-03-28T05:17:00.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:59 smithi023 ceph-mon[31954]: Detected new or changed devices on smithi023 2024-03-28T05:17:00.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:59 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:00.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:59 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:00.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:59 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:17:00.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:59 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:17:00.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:59 smithi023 ceph-mon[31954]: Adjusting osd_memory_target on smithi023 to 8544M 2024-03-28T05:17:00.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:59 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:00.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:59 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:00.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:59 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:17:00.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:59 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:00.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:59 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:17:00.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:59 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:00.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:59 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:17:00.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:59 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:00.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:16:59 smithi023 ceph-mon[31954]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T05:17:01.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:00 smithi098 ceph-mon[36087]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:17:01.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:00 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:17:01.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:00 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:17:01.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:00 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:01.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:00 smithi098 ceph-mon[36087]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T05:17:01.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:00 smithi113 ceph-mon[37164]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:17:01.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:00 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:17:01.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:00 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:17:01.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:00 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:01.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:00 smithi113 ceph-mon[37164]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T05:17:01.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:00 smithi023 ceph-mon[31954]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:17:01.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:00 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:17:01.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:00 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:17:01.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:00 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:01.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:00 smithi023 ceph-mon[31954]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T05:17:02.095 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:01 smithi098 ceph-mon[36087]: pgmap v104: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 516 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T05:17:02.095 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:01 smithi098 ceph-mon[36087]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-28T05:17:02.095 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:01 smithi098 ceph-mon[36087]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T05:17:02.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:01 smithi113 ceph-mon[37164]: pgmap v104: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 516 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T05:17:02.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:01 smithi113 ceph-mon[37164]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-28T05:17:02.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:01 smithi113 ceph-mon[37164]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T05:17:02.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:01 smithi023 ceph-mon[31954]: pgmap v104: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 516 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T05:17:02.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:01 smithi023 ceph-mon[31954]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-28T05:17:02.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:01 smithi023 ceph-mon[31954]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T05:17:04.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:03 smithi098 ceph-mon[36087]: pgmap v106: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T05:17:04.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:03 smithi098 ceph-mon[36087]: from='client.? 172.21.15.98:0/211309482' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76a90bac-b221-40a3-8238-40e8049cec36"}]: dispatch 2024-03-28T05:17:04.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:03 smithi098 ceph-mon[36087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76a90bac-b221-40a3-8238-40e8049cec36"}]: dispatch 2024-03-28T05:17:04.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:03 smithi098 ceph-mon[36087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76a90bac-b221-40a3-8238-40e8049cec36"}]': finished 2024-03-28T05:17:04.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:03 smithi098 ceph-mon[36087]: osdmap e30: 5 total, 4 up, 5 in 2024-03-28T05:17:04.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:03 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T05:17:04.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:03 smithi098 ceph-mon[36087]: from='client.? 172.21.15.98:0/1444474462' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:17:04.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:03 smithi113 ceph-mon[37164]: pgmap v106: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T05:17:04.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:03 smithi113 ceph-mon[37164]: from='client.? 172.21.15.98:0/211309482' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76a90bac-b221-40a3-8238-40e8049cec36"}]: dispatch 2024-03-28T05:17:04.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:03 smithi113 ceph-mon[37164]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76a90bac-b221-40a3-8238-40e8049cec36"}]: dispatch 2024-03-28T05:17:04.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:03 smithi113 ceph-mon[37164]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76a90bac-b221-40a3-8238-40e8049cec36"}]': finished 2024-03-28T05:17:04.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:03 smithi113 ceph-mon[37164]: osdmap e30: 5 total, 4 up, 5 in 2024-03-28T05:17:04.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:03 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T05:17:04.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:03 smithi113 ceph-mon[37164]: from='client.? 172.21.15.98:0/1444474462' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:17:04.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:03 smithi023 ceph-mon[31954]: pgmap v106: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2024-03-28T05:17:04.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:03 smithi023 ceph-mon[31954]: from='client.? 172.21.15.98:0/211309482' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76a90bac-b221-40a3-8238-40e8049cec36"}]: dispatch 2024-03-28T05:17:04.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:03 smithi023 ceph-mon[31954]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76a90bac-b221-40a3-8238-40e8049cec36"}]: dispatch 2024-03-28T05:17:04.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:03 smithi023 ceph-mon[31954]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76a90bac-b221-40a3-8238-40e8049cec36"}]': finished 2024-03-28T05:17:04.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:03 smithi023 ceph-mon[31954]: osdmap e30: 5 total, 4 up, 5 in 2024-03-28T05:17:04.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:03 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T05:17:04.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:03 smithi023 ceph-mon[31954]: from='client.? 172.21.15.98:0/1444474462' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:17:05.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:04 smithi113 ceph-mon[37164]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-28T05:17:05.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:04 smithi113 ceph-mon[37164]: Cluster is now healthy 2024-03-28T05:17:05.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:04 smithi023 ceph-mon[31954]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-28T05:17:05.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:04 smithi023 ceph-mon[31954]: Cluster is now healthy 2024-03-28T05:17:05.363 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:04 smithi098 ceph-mon[36087]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-28T05:17:05.364 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:04 smithi098 ceph-mon[36087]: Cluster is now healthy 2024-03-28T05:17:06.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:05 smithi113 ceph-mon[37164]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-28T05:17:06.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:05 smithi023 ceph-mon[31954]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-28T05:17:06.363 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:05 smithi098 ceph-mon[36087]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-28T05:17:08.113 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:07 smithi098 ceph-mon[36087]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-28T05:17:08.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:07 smithi113 ceph-mon[37164]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-28T05:17:08.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:07 smithi023 ceph-mon[31954]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-28T05:17:10.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:09 smithi113 ceph-mon[37164]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T05:17:10.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:09 smithi023 ceph-mon[31954]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T05:17:10.363 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:09 smithi098 ceph-mon[36087]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T05:17:11.037 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T05:17:11.037 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:11.037 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:10 smithi098 ceph-mon[36087]: Deploying daemon osd.4 on smithi098 2024-03-28T05:17:11.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T05:17:11.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:11.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:10 smithi113 ceph-mon[37164]: Deploying daemon osd.4 on smithi098 2024-03-28T05:17:11.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T05:17:11.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:11.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:10 smithi023 ceph-mon[31954]: Deploying daemon osd.4 on smithi098 2024-03-28T05:17:12.081 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:11 smithi098 ceph-mon[36087]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-28T05:17:12.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:11 smithi113 ceph-mon[37164]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-28T05:17:12.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:11 smithi023 ceph-mon[31954]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-28T05:17:14.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:13 smithi113 ceph-mon[37164]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T05:17:14.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:13 smithi023 ceph-mon[31954]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T05:17:14.364 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:13 smithi098 ceph-mon[36087]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T05:17:16.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:15 smithi113 ceph-mon[37164]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-28T05:17:16.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:15 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:16.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:15 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:16.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:15 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:17:16.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:15 smithi023 ceph-mon[31954]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-28T05:17:16.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:15 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:16.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:15 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:16.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:15 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:17:16.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:15 smithi098 ceph-mon[36087]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-28T05:17:16.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:15 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:16.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:15 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:16.350 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:15 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:17:18.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:17 smithi113 ceph-mon[37164]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-28T05:17:18.279 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:17 smithi023 ceph-mon[31954]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-28T05:17:18.364 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:17 smithi098 ceph-mon[36087]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-28T05:17:19.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:19.205+0000 7f65597fa700 1 -- 172.21.15.98:0/177329569 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f655c0fe850 con 0x7f65440a2890 2024-03-28T05:17:19.205 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 4 on host 'smithi098' 2024-03-28T05:17:19.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:19.208+0000 7f6562a07700 1 -- 172.21.15.98:0/177329569 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f65440a2890 msgr2=0x7f65440a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:17:19.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:19.208+0000 7f6562a07700 1 --2- 172.21.15.98:0/177329569 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f65440a2890 0x7f65440a4d50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f654c005950 tx=0x7f654c0058e0 comp rx=0 tx=0).stop 2024-03-28T05:17:19.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:19.208+0000 7f6562a07700 1 -- 172.21.15.98:0/177329569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f655c1022c0 msgr2=0x7f655c198940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:17:19.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:19.208+0000 7f6562a07700 1 --2- 172.21.15.98:0/177329569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f655c1022c0 0x7f655c198940 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f655000c970 tx=0x7f655000cd30 comp rx=0 tx=0).stop 2024-03-28T05:17:19.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:19.208+0000 7f6562a07700 1 -- 172.21.15.98:0/177329569 shutdown_connections 2024-03-28T05:17:19.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:19.208+0000 7f6562a07700 1 --2- 172.21.15.98:0/177329569 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f65440a2890 0x7f65440a4d50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:19.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:19.208+0000 7f6562a07700 1 --2- 172.21.15.98:0/177329569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f655c102c60 0x7f655c19f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:19.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:19.208+0000 7f6562a07700 1 --2- 172.21.15.98:0/177329569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f655c1010c0 0x7f655c198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:19.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:19.208+0000 7f6562a07700 1 --2- 172.21.15.98:0/177329569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f655c1022c0 0x7f655c198940 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:19.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:19.208+0000 7f6562a07700 1 -- 172.21.15.98:0/177329569 >> 172.21.15.98:0/177329569 conn(0x7f655c0fc660 msgr2=0x7f655c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:17:19.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:19.208+0000 7f6562a07700 1 -- 172.21.15.98:0/177329569 shutdown_connections 2024-03-28T05:17:19.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:17:19.208+0000 7f6562a07700 1 -- 172.21.15.98:0/177329569 wait complete. 2024-03-28T05:17:19.955 DEBUG:teuthology.orchestra.run.smithi098:osd.4> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.4.service 2024-03-28T05:17:19.958 INFO:tasks.cephadm:Deploying osd.5 on smithi113 with /dev/vg_nvme/lv_3... 2024-03-28T05:17:19.958 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T05:17:20.204 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T05:17:20.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:20 smithi113 ceph-mon[37164]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-28T05:17:20.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:20 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:20.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:20 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:20.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:20 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:20.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:20 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:20.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:20 smithi113 ceph-mon[37164]: from='osd.4 [v2:172.21.15.98:6808/2615902819,v1:172.21.15.98:6809/2615902819]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T05:17:20.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:20 smithi113 ceph-mon[37164]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T05:17:20.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:20 smithi098 ceph-mon[36087]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-28T05:17:20.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:20 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:20.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:20 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:20.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:20 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:20.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:20 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:20.477 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:20 smithi098 ceph-mon[36087]: from='osd.4 [v2:172.21.15.98:6808/2615902819,v1:172.21.15.98:6809/2615902819]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T05:17:20.478 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:20 smithi098 ceph-mon[36087]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T05:17:20.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:20 smithi023 ceph-mon[31954]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-28T05:17:20.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:20 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:20.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:20 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:20.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:20 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:20.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:20 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:20.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:20 smithi023 ceph-mon[31954]: from='osd.4 [v2:172.21.15.98:6808/2615902819,v1:172.21.15.98:6809/2615902819]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T05:17:20.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:20 smithi023 ceph-mon[31954]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T05:17:21.602 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:21 smithi113 ceph-mon[37164]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T05:17:21.602 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:21 smithi113 ceph-mon[37164]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T05:17:21.602 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:21 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T05:17:21.602 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:21 smithi113 ceph-mon[37164]: from='osd.4 [v2:172.21.15.98:6808/2615902819,v1:172.21.15.98:6809/2615902819]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:17:21.602 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:21 smithi113 ceph-mon[37164]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:17:21.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:21 smithi098 ceph-mon[36087]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T05:17:21.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:21 smithi098 ceph-mon[36087]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T05:17:21.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:21 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T05:17:21.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:21 smithi098 ceph-mon[36087]: from='osd.4 [v2:172.21.15.98:6808/2615902819,v1:172.21.15.98:6809/2615902819]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:17:21.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:21 smithi098 ceph-mon[36087]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:17:22.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:21 smithi023 ceph-mon[31954]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T05:17:22.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:21 smithi023 ceph-mon[31954]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T05:17:22.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:21 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T05:17:22.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:21 smithi023 ceph-mon[31954]: from='osd.4 [v2:172.21.15.98:6808/2615902819,v1:172.21.15.98:6809/2615902819]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:17:22.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:21 smithi023 ceph-mon[31954]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:17:22.037 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T05:17:22.037 INFO:teuthology.orchestra.run.smithi113.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T05:17:22.037 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10+0 records in 2024-03-28T05:17:22.037 INFO:teuthology.orchestra.run.smithi113.stderr:10+0 records out 2024-03-28T05:17:22.037 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00806102 s, 1.3 GB/s 2024-03-28T05:17:22.037 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping successful for: 2024-03-28T05:17:22.554 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:22 smithi098 ceph-mon[36087]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-28T05:17:22.554 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:22 smithi098 ceph-mon[36087]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-28T05:17:22.554 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:22 smithi098 ceph-mon[36087]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T05:17:22.555 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:22 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T05:17:22.555 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:22 smithi098 ceph-mon[36087]: from='osd.4 ' entity='osd.4' 2024-03-28T05:17:22.555 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:22 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T05:17:22.555 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 28 05:17:22 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-4[47205]: 2024-03-28T05:17:22.235+0000 7fc8f508f700 -1 osd.4 0 waiting for initial osdmap 2024-03-28T05:17:22.555 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 28 05:17:22 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-4[47205]: 2024-03-28T05:17:22.240+0000 7fc8ece7a700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T05:17:22.697 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:22 smithi113 ceph-mon[37164]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-28T05:17:22.698 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:22 smithi113 ceph-mon[37164]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-28T05:17:22.698 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:22 smithi113 ceph-mon[37164]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T05:17:22.698 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:22 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T05:17:22.698 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:22 smithi113 ceph-mon[37164]: from='osd.4 ' entity='osd.4' 2024-03-28T05:17:22.698 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:22 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T05:17:22.725 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi113:vg_nvme/lv_3 2024-03-28T05:17:22.967 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T05:17:23.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:22 smithi023 ceph-mon[31954]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-28T05:17:23.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:22 smithi023 ceph-mon[31954]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-28T05:17:23.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:22 smithi023 ceph-mon[31954]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T05:17:23.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T05:17:23.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:22 smithi023 ceph-mon[31954]: from='osd.4 ' entity='osd.4' 2024-03-28T05:17:23.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T05:17:23.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:23 smithi113 ceph-mon[37164]: purged_snaps scrub starts 2024-03-28T05:17:23.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:23 smithi113 ceph-mon[37164]: purged_snaps scrub ok 2024-03-28T05:17:23.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:23.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:23.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:17:23.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:17:23.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:23.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:23.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:17:23.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:23.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:23 smithi113 ceph-mon[37164]: osd.4 [v2:172.21.15.98:6808/2615902819,v1:172.21.15.98:6809/2615902819] boot 2024-03-28T05:17:23.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:23 smithi113 ceph-mon[37164]: osdmap e33: 5 total, 5 up, 5 in 2024-03-28T05:17:23.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T05:17:23.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:17:23.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:23.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:17:23.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:23 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:23.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:23 smithi098 ceph-mon[36087]: purged_snaps scrub starts 2024-03-28T05:17:23.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:23 smithi098 ceph-mon[36087]: purged_snaps scrub ok 2024-03-28T05:17:23.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:23.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:23.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:17:23.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:17:23.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:23.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:23.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:17:23.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:23.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:23 smithi098 ceph-mon[36087]: osd.4 [v2:172.21.15.98:6808/2615902819,v1:172.21.15.98:6809/2615902819] boot 2024-03-28T05:17:23.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:23 smithi098 ceph-mon[36087]: osdmap e33: 5 total, 5 up, 5 in 2024-03-28T05:17:23.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T05:17:23.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:17:23.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:23.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:17:23.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:23 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:24.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:23 smithi023 ceph-mon[31954]: purged_snaps scrub starts 2024-03-28T05:17:24.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:23 smithi023 ceph-mon[31954]: purged_snaps scrub ok 2024-03-28T05:17:24.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:24.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:24.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:17:24.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:17:24.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:24.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:24.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:17:24.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:24.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:23 smithi023 ceph-mon[31954]: osd.4 [v2:172.21.15.98:6808/2615902819,v1:172.21.15.98:6809/2615902819] boot 2024-03-28T05:17:24.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:23 smithi023 ceph-mon[31954]: osdmap e33: 5 total, 5 up, 5 in 2024-03-28T05:17:24.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T05:17:24.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:17:24.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:24.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:17:24.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:23 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:24.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.654+0000 7fc698fe9700 1 -- 172.21.15.113:0/2039095775 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc694101d40 msgr2=0x7fc694109600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:17:24.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.654+0000 7fc698fe9700 1 --2- 172.21.15.113:0/2039095775 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc694101d40 0x7fc694109600 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc68c00b0a0 tx=0x7fc68c00b3b0 comp rx=0 tx=0).stop 2024-03-28T05:17:24.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.654+0000 7fc698fe9700 1 -- 172.21.15.113:0/2039095775 shutdown_connections 2024-03-28T05:17:24.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.654+0000 7fc698fe9700 1 --2- 172.21.15.113:0/2039095775 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc694101d40 0x7fc694109600 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:24.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.654+0000 7fc698fe9700 1 --2- 172.21.15.113:0/2039095775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6941026f0 0x7fc694102b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:24.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.654+0000 7fc698fe9700 1 --2- 172.21.15.113:0/2039095775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6941013a0 0x7fc694101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:24.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.654+0000 7fc698fe9700 1 -- 172.21.15.113:0/2039095775 >> 172.21.15.113:0/2039095775 conn(0x7fc6940fbe30 msgr2=0x7fc6940fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:17:24.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.655+0000 7fc698fe9700 1 -- 172.21.15.113:0/2039095775 shutdown_connections 2024-03-28T05:17:24.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.655+0000 7fc698fe9700 1 -- 172.21.15.113:0/2039095775 wait complete. 2024-03-28T05:17:24.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.655+0000 7fc698fe9700 1 Processor -- start 2024-03-28T05:17:24.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.655+0000 7fc698fe9700 1 -- start start 2024-03-28T05:17:24.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.655+0000 7fc698fe9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6941013a0 0x7fc694105140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:17:24.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.655+0000 7fc698fe9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc694101d40 0x7fc694103790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:17:24.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.655+0000 7fc698fe9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6941026f0 0x7fc694103cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:17:24.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.655+0000 7fc698fe9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69410ccf0 con 0x7fc6941013a0 2024-03-28T05:17:24.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.655+0000 7fc698fe9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69410cb70 con 0x7fc6941026f0 2024-03-28T05:17:24.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.656+0000 7fc698fe9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69410ce70 con 0x7fc694101d40 2024-03-28T05:17:24.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.656+0000 7fc691d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc694101d40 0x7fc694103790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:17:24.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.656+0000 7fc691d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc694101d40 0x7fc694103790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:44164/0 (socket says 172.21.15.113:44164) 2024-03-28T05:17:24.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.656+0000 7fc691d9b700 1 -- 172.21.15.113:0/3021154874 learned_addr learned my addr 172.21.15.113:0/3021154874 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-03-28T05:17:24.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.656+0000 7fc692d9d700 1 --2- 172.21.15.113:0/3021154874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6941026f0 0x7fc694103cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:17:24.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.656+0000 7fc69259c700 1 --2- 172.21.15.113:0/3021154874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6941013a0 0x7fc694105140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:17:24.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.656+0000 7fc691d9b700 1 -- 172.21.15.113:0/3021154874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6941026f0 msgr2=0x7fc694103cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:17:24.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.656+0000 7fc691d9b700 1 --2- 172.21.15.113:0/3021154874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6941026f0 0x7fc694103cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:24.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.656+0000 7fc691d9b700 1 -- 172.21.15.113:0/3021154874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6941013a0 msgr2=0x7fc694105140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:17:24.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.656+0000 7fc691d9b700 1 --2- 172.21.15.113:0/3021154874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6941013a0 0x7fc694105140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:24.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.656+0000 7fc691d9b700 1 -- 172.21.15.113:0/3021154874 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc694104580 con 0x7fc694101d40 2024-03-28T05:17:24.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.656+0000 7fc691d9b700 1 --2- 172.21.15.113:0/3021154874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc694101d40 0x7fc694103790 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc68800d980 tx=0x7fc68800dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:17:24.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.656+0000 7fc6837fe700 1 -- 172.21.15.113:0/3021154874 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc688017070 con 0x7fc694101d40 2024-03-28T05:17:24.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.656+0000 7fc698fe9700 1 -- 172.21.15.113:0/3021154874 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6941a24f0 con 0x7fc694101d40 2024-03-28T05:17:24.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.656+0000 7fc6837fe700 1 -- 172.21.15.113:0/3021154874 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc688005020 con 0x7fc694101d40 2024-03-28T05:17:24.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.656+0000 7fc698fe9700 1 -- 172.21.15.113:0/3021154874 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6941a2a00 con 0x7fc694101d40 2024-03-28T05:17:24.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.656+0000 7fc6837fe700 1 -- 172.21.15.113:0/3021154874 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc688009bc0 con 0x7fc694101d40 2024-03-28T05:17:24.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.657+0000 7fc698fe9700 1 -- 172.21.15.113:0/3021154874 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc694068e50 con 0x7fc694101d40 2024-03-28T05:17:24.659 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.658+0000 7fc6837fe700 1 -- 172.21.15.113:0/3021154874 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc6880051d0 con 0x7fc694101d40 2024-03-28T05:17:24.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.659+0000 7fc6837fe700 1 --2- 172.21.15.113:0/3021154874 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc67c0a2890 0x7fc67c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:17:24.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.659+0000 7fc6837fe700 1 -- 172.21.15.113:0/3021154874 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4101+0+0 (secure 0 0 0) 0x7fc6880cdd80 con 0x7fc694101d40 2024-03-28T05:17:24.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.659+0000 7fc69259c700 1 --2- 172.21.15.113:0/3021154874 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc67c0a2890 0x7fc67c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:17:24.660 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.660+0000 7fc69259c700 1 --2- 172.21.15.113:0/3021154874 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc67c0a2890 0x7fc67c0a4d50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc684005fd0 tx=0x7fc684005dd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:17:24.661 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.664+0000 7fc6837fe700 1 -- 172.21.15.113:0/3021154874 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc688020020 con 0x7fc694101d40 2024-03-28T05:17:24.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:24 smithi113 ceph-mon[37164]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-28T05:17:24.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:24 smithi113 ceph-mon[37164]: Detected new or changed devices on smithi098 2024-03-28T05:17:24.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:24 smithi113 ceph-mon[37164]: Adjusting osd_memory_target on smithi098 to 8540M 2024-03-28T05:17:24.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:24 smithi098 ceph-mon[36087]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-28T05:17:24.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:24 smithi098 ceph-mon[36087]: Detected new or changed devices on smithi098 2024-03-28T05:17:24.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:24 smithi098 ceph-mon[36087]: Adjusting osd_memory_target on smithi098 to 8540M 2024-03-28T05:17:24.867 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:24.869+0000 7fc698fe9700 1 -- 172.21.15.113:0/3021154874 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6940fdf60 con 0x7fc67c0a2890 2024-03-28T05:17:25.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:24 smithi023 ceph-mon[31954]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-28T05:17:25.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:24 smithi023 ceph-mon[31954]: Detected new or changed devices on smithi098 2024-03-28T05:17:25.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:24 smithi023 ceph-mon[31954]: Adjusting osd_memory_target on smithi098 to 8540M 2024-03-28T05:17:25.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:25 smithi113 ceph-mon[37164]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T05:17:25.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:25 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:17:25.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:25 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:17:25.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:25 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:25.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:25 smithi098 ceph-mon[36087]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T05:17:25.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:25 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:17:25.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:25 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:17:25.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:25 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:26.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:25 smithi023 ceph-mon[31954]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T05:17:26.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:25 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:17:26.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:25 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:17:26.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:25 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:26.705 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:26 smithi113 ceph-mon[37164]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:26.706 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:26 smithi113 ceph-mon[37164]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:17:26.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:26 smithi098 ceph-mon[36087]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:26.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:26 smithi098 ceph-mon[36087]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:17:27.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:26 smithi023 ceph-mon[31954]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:27.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:26 smithi023 ceph-mon[31954]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:17:27.855 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:27 smithi113 ceph-mon[37164]: from='client.? 172.21.15.113:0/358659983' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b53fdb55-a8df-40a8-bd1b-8bcef634e26a"}]: dispatch 2024-03-28T05:17:27.855 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:27 smithi113 ceph-mon[37164]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b53fdb55-a8df-40a8-bd1b-8bcef634e26a"}]: dispatch 2024-03-28T05:17:27.855 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:27 smithi113 ceph-mon[37164]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b53fdb55-a8df-40a8-bd1b-8bcef634e26a"}]': finished 2024-03-28T05:17:27.855 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:27 smithi113 ceph-mon[37164]: osdmap e35: 6 total, 5 up, 6 in 2024-03-28T05:17:27.855 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:27 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T05:17:27.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:27 smithi098 ceph-mon[36087]: from='client.? 172.21.15.113:0/358659983' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b53fdb55-a8df-40a8-bd1b-8bcef634e26a"}]: dispatch 2024-03-28T05:17:27.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:27 smithi098 ceph-mon[36087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b53fdb55-a8df-40a8-bd1b-8bcef634e26a"}]: dispatch 2024-03-28T05:17:27.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:27 smithi098 ceph-mon[36087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b53fdb55-a8df-40a8-bd1b-8bcef634e26a"}]': finished 2024-03-28T05:17:27.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:27 smithi098 ceph-mon[36087]: osdmap e35: 6 total, 5 up, 6 in 2024-03-28T05:17:27.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:27 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T05:17:28.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:27 smithi023 ceph-mon[31954]: from='client.? 172.21.15.113:0/358659983' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b53fdb55-a8df-40a8-bd1b-8bcef634e26a"}]: dispatch 2024-03-28T05:17:28.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:27 smithi023 ceph-mon[31954]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b53fdb55-a8df-40a8-bd1b-8bcef634e26a"}]: dispatch 2024-03-28T05:17:28.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:27 smithi023 ceph-mon[31954]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b53fdb55-a8df-40a8-bd1b-8bcef634e26a"}]': finished 2024-03-28T05:17:28.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:27 smithi023 ceph-mon[31954]: osdmap e35: 6 total, 5 up, 6 in 2024-03-28T05:17:28.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:27 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T05:17:28.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:28 smithi098 ceph-mon[36087]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:28.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:28 smithi098 ceph-mon[36087]: from='client.? 172.21.15.113:0/888144529' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:17:28.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:28 smithi113 ceph-mon[37164]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:28.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:28 smithi113 ceph-mon[37164]: from='client.? 172.21.15.113:0/888144529' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:17:29.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:28 smithi023 ceph-mon[31954]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:29.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:28 smithi023 ceph-mon[31954]: from='client.? 172.21.15.113:0/888144529' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:17:30.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:30 smithi098 ceph-mon[36087]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:30.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:30 smithi113 ceph-mon[37164]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:31.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:30 smithi023 ceph-mon[31954]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:32.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:32 smithi113 ceph-mon[37164]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:32.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:32 smithi098 ceph-mon[36087]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:33.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:32 smithi023 ceph-mon[31954]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:34.837 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:34 smithi113 ceph-mon[37164]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:34.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:34 smithi098 ceph-mon[36087]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:35.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:34 smithi023 ceph-mon[31954]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:35.723 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:35 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T05:17:35.723 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:35 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:36.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:35 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T05:17:36.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:35 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:36.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:35 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T05:17:36.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:35 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:36.813 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:36 smithi113 ceph-mon[37164]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:36.813 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:36 smithi113 ceph-mon[37164]: Deploying daemon osd.5 on smithi113 2024-03-28T05:17:37.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:36 smithi023 ceph-mon[31954]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:37.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:36 smithi023 ceph-mon[31954]: Deploying daemon osd.5 on smithi113 2024-03-28T05:17:37.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:36 smithi098 ceph-mon[36087]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:37.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:36 smithi098 ceph-mon[36087]: Deploying daemon osd.5 on smithi113 2024-03-28T05:17:38.874 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:38 smithi113 ceph-mon[37164]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:39.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:38 smithi023 ceph-mon[31954]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:39.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:38 smithi098 ceph-mon[36087]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:40.930 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:40 smithi113 ceph-mon[37164]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:40.931 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:40 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:17:40.931 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:40 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:40.931 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:40 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:41.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:40 smithi023 ceph-mon[31954]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:41.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:40 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:17:41.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:40 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:41.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:40 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:41.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:40 smithi098 ceph-mon[36087]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:41.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:40 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:17:41.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:40 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:41.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:40 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:42.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:42 smithi113 ceph-mon[37164]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:43.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:42 smithi023 ceph-mon[31954]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:43.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:42 smithi098 ceph-mon[36087]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:44.285 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:44.287+0000 7fc6837fe700 1 -- 172.21.15.113:0/3021154874 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc6940fdf60 con 0x7fc67c0a2890 2024-03-28T05:17:44.285 INFO:teuthology.orchestra.run.smithi113.stdout:Created osd(s) 5 on host 'smithi113' 2024-03-28T05:17:44.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:44.289+0000 7fc698fe9700 1 -- 172.21.15.113:0/3021154874 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc67c0a2890 msgr2=0x7fc67c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:17:44.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:44.289+0000 7fc698fe9700 1 --2- 172.21.15.113:0/3021154874 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc67c0a2890 0x7fc67c0a4d50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc684005fd0 tx=0x7fc684005dd0 comp rx=0 tx=0).stop 2024-03-28T05:17:44.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:44.289+0000 7fc698fe9700 1 -- 172.21.15.113:0/3021154874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc694101d40 msgr2=0x7fc694103790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:17:44.287 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:44.289+0000 7fc698fe9700 1 --2- 172.21.15.113:0/3021154874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc694101d40 0x7fc694103790 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc68800d980 tx=0x7fc68800dc90 comp rx=0 tx=0).stop 2024-03-28T05:17:44.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:44.289+0000 7fc698fe9700 1 -- 172.21.15.113:0/3021154874 shutdown_connections 2024-03-28T05:17:44.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:44.289+0000 7fc698fe9700 1 --2- 172.21.15.113:0/3021154874 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc67c0a2890 0x7fc67c0a4d50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:44.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:44.289+0000 7fc698fe9700 1 --2- 172.21.15.113:0/3021154874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6941026f0 0x7fc694103cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:44.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:44.289+0000 7fc698fe9700 1 --2- 172.21.15.113:0/3021154874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6941013a0 0x7fc694105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:44.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:44.289+0000 7fc698fe9700 1 --2- 172.21.15.113:0/3021154874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc694101d40 0x7fc694103790 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:44.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:44.289+0000 7fc698fe9700 1 -- 172.21.15.113:0/3021154874 >> 172.21.15.113:0/3021154874 conn(0x7fc6940fbe30 msgr2=0x7fc6940fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:17:44.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:44.290+0000 7fc698fe9700 1 -- 172.21.15.113:0/3021154874 shutdown_connections 2024-03-28T05:17:44.288 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:17:44.290+0000 7fc698fe9700 1 -- 172.21.15.113:0/3021154874 wait complete. 2024-03-28T05:17:44.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:44 smithi113 ceph-mon[37164]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:44.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:44 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:44.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:44 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:44.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:44 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:44.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:44 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:17:44.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:44 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:44.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:44 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:44.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:44 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:44.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:44 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:17:44.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:44 smithi113 ceph-mon[37164]: from='osd.5 [v2:172.21.15.113:6808/625271528,v1:172.21.15.113:6809/625271528]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T05:17:44.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:44 smithi113 ceph-mon[37164]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T05:17:44.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:44 smithi023 ceph-mon[31954]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:44.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:44 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:44.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:44 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:44.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:44 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:44.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:44 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:17:44.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:44 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:44.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:44 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:44.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:44 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:44.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:44 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:17:44.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:44 smithi023 ceph-mon[31954]: from='osd.5 [v2:172.21.15.113:6808/625271528,v1:172.21.15.113:6809/625271528]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T05:17:44.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:44 smithi023 ceph-mon[31954]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T05:17:44.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:44 smithi098 ceph-mon[36087]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:44.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:44 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:44.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:44 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:44.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:44 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:44.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:44 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:17:44.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:44 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:44.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:44 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:44.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:44 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:44.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:44 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:17:44.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:44 smithi098 ceph-mon[36087]: from='osd.5 [v2:172.21.15.113:6808/625271528,v1:172.21.15.113:6809/625271528]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T05:17:44.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:44 smithi098 ceph-mon[36087]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T05:17:44.884 DEBUG:teuthology.orchestra.run.smithi113:osd.5> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.5.service 2024-03-28T05:17:44.887 INFO:tasks.cephadm:Deploying osd.6 on smithi023 with /dev/vg_nvme/lv_2... 2024-03-28T05:17:44.887 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T05:17:45.139 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:17:46.427 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:46 smithi113 ceph-mon[37164]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:46.427 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:46 smithi113 ceph-mon[37164]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T05:17:46.427 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:46 smithi113 ceph-mon[37164]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T05:17:46.427 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:46 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T05:17:46.427 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:46 smithi113 ceph-mon[37164]: from='osd.5 [v2:172.21.15.113:6808/625271528,v1:172.21.15.113:6809/625271528]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:17:46.427 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:46 smithi113 ceph-mon[37164]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:17:46.427 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:17:46 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:17:46.275+0000 7f93acb1f700 -1 osd.5 0 waiting for initial osdmap 2024-03-28T05:17:46.427 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:17:46 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:17:46.283+0000 7f93a8912700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T05:17:46.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:46 smithi023 ceph-mon[31954]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:46.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:46 smithi023 ceph-mon[31954]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T05:17:46.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:46 smithi023 ceph-mon[31954]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T05:17:46.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:46 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T05:17:46.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:46 smithi023 ceph-mon[31954]: from='osd.5 [v2:172.21.15.113:6808/625271528,v1:172.21.15.113:6809/625271528]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:17:46.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:46 smithi023 ceph-mon[31954]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:17:46.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:46 smithi098 ceph-mon[36087]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:46.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:46 smithi098 ceph-mon[36087]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T05:17:46.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:46 smithi098 ceph-mon[36087]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T05:17:46.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:46 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T05:17:46.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:46 smithi098 ceph-mon[36087]: from='osd.5 [v2:172.21.15.113:6808/625271528,v1:172.21.15.113:6809/625271528]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:17:46.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:46 smithi098 ceph-mon[36087]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:17:47.149 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T05:17:47.149 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T05:17:47.149 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-28T05:17:47.149 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-28T05:17:47.149 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.016515 s, 635 MB/s 2024-03-28T05:17:47.149 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-28T05:17:47.469 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:47 smithi023 ceph-mon[31954]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-28T05:17:47.469 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:47 smithi023 ceph-mon[31954]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T05:17:47.469 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:47 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T05:17:47.470 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:47 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T05:17:47.470 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:47 smithi023 ceph-mon[31954]: osd.5 [v2:172.21.15.113:6808/625271528,v1:172.21.15.113:6809/625271528] boot 2024-03-28T05:17:47.470 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:47 smithi023 ceph-mon[31954]: osdmap e38: 6 total, 6 up, 6 in 2024-03-28T05:17:47.470 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:47 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T05:17:47.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:47 smithi113 ceph-mon[37164]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-28T05:17:47.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:47 smithi113 ceph-mon[37164]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T05:17:47.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:47 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T05:17:47.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:47 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T05:17:47.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:47 smithi113 ceph-mon[37164]: osd.5 [v2:172.21.15.113:6808/625271528,v1:172.21.15.113:6809/625271528] boot 2024-03-28T05:17:47.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:47 smithi113 ceph-mon[37164]: osdmap e38: 6 total, 6 up, 6 in 2024-03-28T05:17:47.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:47 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T05:17:47.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:47 smithi098 ceph-mon[36087]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-28T05:17:47.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:47 smithi098 ceph-mon[36087]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T05:17:47.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:47 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T05:17:47.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:47 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T05:17:47.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:47 smithi098 ceph-mon[36087]: osd.5 [v2:172.21.15.113:6808/625271528,v1:172.21.15.113:6809/625271528] boot 2024-03-28T05:17:47.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:47 smithi098 ceph-mon[36087]: osdmap e38: 6 total, 6 up, 6 in 2024-03-28T05:17:47.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:47 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T05:17:47.783 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_2 2024-03-28T05:17:48.036 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:17:48.466 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:48 smithi023 ceph-mon[31954]: purged_snaps scrub starts 2024-03-28T05:17:48.466 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:48 smithi023 ceph-mon[31954]: purged_snaps scrub ok 2024-03-28T05:17:48.466 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:48 smithi023 ceph-mon[31954]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:48.466 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:48.466 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:48.466 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:17:48.466 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:17:48.466 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:48.467 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:48.467 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:17:48.467 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:48.467 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:48 smithi023 ceph-mon[31954]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T05:17:48.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:48 smithi098 ceph-mon[36087]: purged_snaps scrub starts 2024-03-28T05:17:48.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:48 smithi098 ceph-mon[36087]: purged_snaps scrub ok 2024-03-28T05:17:48.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:48 smithi098 ceph-mon[36087]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:48.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:48 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:48.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:48 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:48.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:48 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:17:48.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:48 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:17:48.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:48 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:48.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:48 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:48.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:48 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:17:48.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:48 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:48.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:48 smithi098 ceph-mon[36087]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T05:17:48.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:48 smithi113 ceph-mon[37164]: purged_snaps scrub starts 2024-03-28T05:17:48.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:48 smithi113 ceph-mon[37164]: purged_snaps scrub ok 2024-03-28T05:17:48.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:48 smithi113 ceph-mon[37164]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-28T05:17:48.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:48 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:48.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:48 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:48.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:48 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:17:48.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:48 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:17:48.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:48 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:48.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:48 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:48.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:48 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:17:48.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:48 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:17:48.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:48 smithi113 ceph-mon[37164]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T05:17:49.519 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:49 smithi113 ceph-mon[37164]: Detected new or changed devices on smithi113 2024-03-28T05:17:49.519 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:49 smithi113 ceph-mon[37164]: Adjusting osd_memory_target on smithi113 to 8540M 2024-03-28T05:17:49.519 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:49 smithi113 ceph-mon[37164]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T05:17:49.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:49 smithi023 ceph-mon[31954]: Detected new or changed devices on smithi113 2024-03-28T05:17:49.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:49 smithi023 ceph-mon[31954]: Adjusting osd_memory_target on smithi113 to 8540M 2024-03-28T05:17:49.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:49 smithi023 ceph-mon[31954]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T05:17:49.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:49 smithi098 ceph-mon[36087]: Detected new or changed devices on smithi113 2024-03-28T05:17:49.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:49 smithi098 ceph-mon[36087]: Adjusting osd_memory_target on smithi113 to 8540M 2024-03-28T05:17:49.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:49 smithi098 ceph-mon[36087]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T05:17:49.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.638+0000 7feff5a58700 1 -- 172.21.15.23:0/1549756811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feff0103520 msgr2=0x7feff0103980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:17:49.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.638+0000 7feff5a58700 1 --2- 172.21.15.23:0/1549756811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feff0103520 0x7feff0103980 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fefd80099c0 tx=0x7fefd8009cd0 comp rx=0 tx=0).stop 2024-03-28T05:17:49.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.638+0000 7feff5a58700 1 -- 172.21.15.23:0/1549756811 shutdown_connections 2024-03-28T05:17:49.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.638+0000 7feff5a58700 1 --2- 172.21.15.23:0/1549756811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feff0103ec0 0x7feff010a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:49.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.638+0000 7feff5a58700 1 --2- 172.21.15.23:0/1549756811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feff01004a0 0x7feff01008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:49.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.638+0000 7feff5a58700 1 --2- 172.21.15.23:0/1549756811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feff0103520 0x7feff0103980 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:49.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.638+0000 7feff5a58700 1 -- 172.21.15.23:0/1549756811 >> 172.21.15.23:0/1549756811 conn(0x7feff00fbe00 msgr2=0x7feff00fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:17:49.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.638+0000 7feff5a58700 1 -- 172.21.15.23:0/1549756811 shutdown_connections 2024-03-28T05:17:49.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.638+0000 7feff5a58700 1 -- 172.21.15.23:0/1549756811 wait complete. 2024-03-28T05:17:49.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.638+0000 7feff5a58700 1 Processor -- start 2024-03-28T05:17:49.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.639+0000 7feff5a58700 1 -- start start 2024-03-28T05:17:49.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.639+0000 7feff5a58700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feff01004a0 0x7feff019a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:17:49.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.639+0000 7feff5a58700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feff0103520 0x7feff019ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:17:49.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.639+0000 7feff5a58700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feff0103ec0 0x7feff01a1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:17:49.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.639+0000 7feff5a58700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff010ddb0 con 0x7feff0103520 2024-03-28T05:17:49.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.639+0000 7feff5a58700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff010dc30 con 0x7feff0103ec0 2024-03-28T05:17:49.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.639+0000 7feff5a58700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff010df30 con 0x7feff01004a0 2024-03-28T05:17:49.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.639+0000 7fefee7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feff0103520 0x7feff019ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:17:49.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.639+0000 7fefee7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feff0103520 0x7feff019ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41852/0 (socket says 172.21.15.23:41852) 2024-03-28T05:17:49.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.639+0000 7fefee7fc700 1 -- 172.21.15.23:0/451755315 learned_addr learned my addr 172.21.15.23:0/451755315 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:17:49.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.639+0000 7fefeeffd700 1 --2- 172.21.15.23:0/451755315 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feff01004a0 0x7feff019a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:17:49.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.639+0000 7fefef7fe700 1 --2- 172.21.15.23:0/451755315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feff0103ec0 0x7feff01a1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:17:49.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.639+0000 7fefee7fc700 1 -- 172.21.15.23:0/451755315 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feff01004a0 msgr2=0x7feff019a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:17:49.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.639+0000 7fefee7fc700 1 --2- 172.21.15.23:0/451755315 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feff01004a0 0x7feff019a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:49.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.639+0000 7fefee7fc700 1 -- 172.21.15.23:0/451755315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feff0103ec0 msgr2=0x7feff01a1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:17:49.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.639+0000 7fefee7fc700 1 --2- 172.21.15.23:0/451755315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feff0103ec0 0x7feff01a1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:17:49.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.639+0000 7fefee7fc700 1 -- 172.21.15.23:0/451755315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff01a2260 con 0x7feff0103520 2024-03-28T05:17:49.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.639+0000 7fefef7fe700 1 --2- 172.21.15.23:0/451755315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feff0103ec0 0x7feff01a1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:17:49.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.639+0000 7fefee7fc700 1 --2- 172.21.15.23:0/451755315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feff0103520 0x7feff019ab40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fefd800b580 tx=0x7fefd800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:17:49.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.640+0000 7feff4a56700 1 -- 172.21.15.23:0/451755315 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd8027070 con 0x7feff0103520 2024-03-28T05:17:49.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.640+0000 7feff4a56700 1 -- 172.21.15.23:0/451755315 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fefd8022950 con 0x7feff0103520 2024-03-28T05:17:49.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.640+0000 7feff5a58700 1 -- 172.21.15.23:0/451755315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff01a24f0 con 0x7feff0103520 2024-03-28T05:17:49.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.640+0000 7feff4a56700 1 -- 172.21.15.23:0/451755315 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd8009e60 con 0x7feff0103520 2024-03-28T05:17:49.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.640+0000 7feff5a58700 1 -- 172.21.15.23:0/451755315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff01a2940 con 0x7feff0103520 2024-03-28T05:17:49.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.641+0000 7feff5a58700 1 -- 172.21.15.23:0/451755315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff0068e50 con 0x7feff0103520 2024-03-28T05:17:49.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.642+0000 7feff4a56700 1 -- 172.21.15.23:0/451755315 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fefd802a070 con 0x7feff0103520 2024-03-28T05:17:49.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.642+0000 7feff4a56700 1 --2- 172.21.15.23:0/451755315 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fefdc0a2890 0x7fefdc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:17:49.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.643+0000 7fefeeffd700 1 --2- 172.21.15.23:0/451755315 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fefdc0a2890 0x7fefdc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:17:49.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.643+0000 7feff4a56700 1 -- 172.21.15.23:0/451755315 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4562+0+0 (secure 0 0 0) 0x7fefd80cea10 con 0x7feff0103520 2024-03-28T05:17:49.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.643+0000 7fefeeffd700 1 --2- 172.21.15.23:0/451755315 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fefdc0a2890 0x7fefdc0a4d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7feff0069b80 tx=0x7fefe8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:17:49.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.646+0000 7feff4a56700 1 -- 172.21.15.23:0/451755315 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fefd8013070 con 0x7feff0103520 2024-03-28T05:17:49.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:17:49.862+0000 7feff5a58700 1 -- 172.21.15.23:0/451755315 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7feff0102a30 con 0x7fefdc0a2890 2024-03-28T05:17:50.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:50 smithi098 ceph-mon[36087]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:17:50.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:50 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:17:50.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:50 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:17:50.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:50 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:50.627 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:50 smithi023 ceph-mon[31954]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:17:50.627 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:50 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:17:50.627 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:50 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:17:50.628 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:50 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:50.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:50 smithi113 ceph-mon[37164]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:17:50.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:50 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:17:50.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:50 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:17:50.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:50 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:17:51.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:51 smithi098 ceph-mon[36087]: from='client.14373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:17:51.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:51 smithi098 ceph-mon[36087]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T05:17:51.677 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:51 smithi023 ceph-mon[31954]: from='client.14373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:17:51.677 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:51 smithi023 ceph-mon[31954]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T05:17:51.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:51 smithi113 ceph-mon[37164]: from='client.14373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:17:51.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:51 smithi113 ceph-mon[37164]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T05:17:52.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:52 smithi098 ceph-mon[36087]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T05:17:52.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:52 smithi098 ceph-mon[36087]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T05:17:52.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:52 smithi113 ceph-mon[37164]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T05:17:52.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:52 smithi113 ceph-mon[37164]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T05:17:52.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:52 smithi023 ceph-mon[31954]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T05:17:52.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:52 smithi023 ceph-mon[31954]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T05:17:53.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:53 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2032554254' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f00283d2-8d67-4e97-a575-9a9d3b55d5cc"}]: dispatch 2024-03-28T05:17:53.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:53 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2032554254' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f00283d2-8d67-4e97-a575-9a9d3b55d5cc"}]': finished 2024-03-28T05:17:53.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:53 smithi098 ceph-mon[36087]: osdmap e43: 7 total, 6 up, 7 in 2024-03-28T05:17:53.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:53 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T05:17:53.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:53 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/4196301282' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:17:53.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:53 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2032554254' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f00283d2-8d67-4e97-a575-9a9d3b55d5cc"}]: dispatch 2024-03-28T05:17:53.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:53 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2032554254' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f00283d2-8d67-4e97-a575-9a9d3b55d5cc"}]': finished 2024-03-28T05:17:53.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:53 smithi113 ceph-mon[37164]: osdmap e43: 7 total, 6 up, 7 in 2024-03-28T05:17:53.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:53 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T05:17:53.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:53 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/4196301282' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:17:53.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:53 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2032554254' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f00283d2-8d67-4e97-a575-9a9d3b55d5cc"}]: dispatch 2024-03-28T05:17:53.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:53 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2032554254' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f00283d2-8d67-4e97-a575-9a9d3b55d5cc"}]': finished 2024-03-28T05:17:53.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:53 smithi023 ceph-mon[31954]: osdmap e43: 7 total, 6 up, 7 in 2024-03-28T05:17:53.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:53 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T05:17:53.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:53 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/4196301282' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:17:54.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:54 smithi113 ceph-mon[37164]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T05:17:55.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:54 smithi023 ceph-mon[31954]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T05:17:55.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:54 smithi098 ceph-mon[36087]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T05:17:56.889 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:56 smithi023 ceph-mon[31954]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-28T05:17:56.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:56 smithi113 ceph-mon[37164]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-28T05:17:57.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:56 smithi098 ceph-mon[36087]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-28T05:17:58.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:17:58 smithi023 ceph-mon[31954]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:17:58.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:17:58 smithi113 ceph-mon[37164]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:17:59.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:17:58 smithi098 ceph-mon[36087]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:00.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:00 smithi023 ceph-mon[31954]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:00.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:00 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T05:18:00.774 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:00 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:00.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:00 smithi113 ceph-mon[37164]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:00.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:00 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T05:18:00.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:00 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:01.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:00 smithi098 ceph-mon[36087]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:01.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:00 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T05:18:01.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:00 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:01.685 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:01 smithi023 ceph-mon[31954]: Deploying daemon osd.6 on smithi023 2024-03-28T05:18:01.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:01 smithi113 ceph-mon[37164]: Deploying daemon osd.6 on smithi023 2024-03-28T05:18:02.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:01 smithi098 ceph-mon[36087]: Deploying daemon osd.6 on smithi023 2024-03-28T05:18:02.957 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:02 smithi023 ceph-mon[31954]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:02.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:02 smithi113 ceph-mon[37164]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:03.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:02 smithi098 ceph-mon[36087]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:04.976 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:04 smithi023 ceph-mon[31954]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:04.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:04 smithi113 ceph-mon[37164]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:05.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:04 smithi098 ceph-mon[36087]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:06.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:06 smithi023 ceph-mon[31954]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:06.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:06.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:06.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:18:06.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:06 smithi098 ceph-mon[36087]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:06.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:06 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:06.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:06 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:06.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:06 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:18:06.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:06 smithi113 ceph-mon[37164]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:06.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:06.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:06.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:18:08.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:08 smithi098 ceph-mon[36087]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:08.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:08 smithi113 ceph-mon[37164]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:09.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:08 smithi023 ceph-mon[31954]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:09.216 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 6 on host 'smithi023' 2024-03-28T05:18:09.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:18:09.216+0000 7feff4a56700 1 -- 172.21.15.23:0/451755315 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7feff0102a30 con 0x7fefdc0a2890 2024-03-28T05:18:09.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:18:09.219+0000 7feff5a58700 1 -- 172.21.15.23:0/451755315 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fefdc0a2890 msgr2=0x7fefdc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:18:09.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:18:09.219+0000 7feff5a58700 1 --2- 172.21.15.23:0/451755315 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fefdc0a2890 0x7fefdc0a4d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7feff0069b80 tx=0x7fefe8009260 comp rx=0 tx=0).stop 2024-03-28T05:18:09.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:18:09.219+0000 7feff5a58700 1 -- 172.21.15.23:0/451755315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feff0103520 msgr2=0x7feff019ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:18:09.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:18:09.219+0000 7feff5a58700 1 --2- 172.21.15.23:0/451755315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feff0103520 0x7feff019ab40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fefd800b580 tx=0x7fefd800fa00 comp rx=0 tx=0).stop 2024-03-28T05:18:09.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:18:09.219+0000 7feff5a58700 1 -- 172.21.15.23:0/451755315 shutdown_connections 2024-03-28T05:18:09.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:18:09.219+0000 7feff5a58700 1 --2- 172.21.15.23:0/451755315 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fefdc0a2890 0x7fefdc0a4d50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:18:09.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:18:09.219+0000 7feff5a58700 1 --2- 172.21.15.23:0/451755315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feff0103ec0 0x7feff01a1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:18:09.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:18:09.219+0000 7feff5a58700 1 --2- 172.21.15.23:0/451755315 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feff01004a0 0x7feff019a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:18:09.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:18:09.219+0000 7feff5a58700 1 --2- 172.21.15.23:0/451755315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feff0103520 0x7feff019ab40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:18:09.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:18:09.219+0000 7feff5a58700 1 -- 172.21.15.23:0/451755315 >> 172.21.15.23:0/451755315 conn(0x7feff00fbe00 msgr2=0x7feff0101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:18:09.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:18:09.219+0000 7feff5a58700 1 -- 172.21.15.23:0/451755315 shutdown_connections 2024-03-28T05:18:09.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:18:09.219+0000 7feff5a58700 1 -- 172.21.15.23:0/451755315 wait complete. 2024-03-28T05:18:09.851 DEBUG:teuthology.orchestra.run.smithi023:osd.6> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.6.service 2024-03-28T05:18:09.853 INFO:tasks.cephadm:Deploying osd.7 on smithi098 with /dev/vg_nvme/lv_2... 2024-03-28T05:18:09.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T05:18:10.100 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T05:18:10.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:10 smithi113 ceph-mon[37164]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:10.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:10.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:10.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:10 smithi113 ceph-mon[37164]: from='osd.6 [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T05:18:10.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:10.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:10.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:10 smithi023 ceph-mon[31954]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:10.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:10.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:10.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:10 smithi023 ceph-mon[31954]: from='osd.6 [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T05:18:10.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:10.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:10.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:10 smithi098 ceph-mon[36087]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:10.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:10.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:10.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:10 smithi098 ceph-mon[36087]: from='osd.6 [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T05:18:10.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:10.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:11.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:11 smithi113 ceph-mon[37164]: from='osd.6 [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T05:18:11.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:11 smithi113 ceph-mon[37164]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T05:18:11.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:11 smithi113 ceph-mon[37164]: from='osd.6 [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T05:18:11.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:11 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T05:18:11.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:11 smithi098 ceph-mon[36087]: from='osd.6 [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T05:18:11.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:11 smithi098 ceph-mon[36087]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T05:18:11.494 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:11 smithi098 ceph-mon[36087]: from='osd.6 [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T05:18:11.495 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:11 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T05:18:11.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:11 smithi023 ceph-mon[31954]: from='osd.6 [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T05:18:11.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:11 smithi023 ceph-mon[31954]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T05:18:11.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:11 smithi023 ceph-mon[31954]: from='osd.6 [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T05:18:11.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:11 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T05:18:11.530 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:18:11 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:18:11.215+0000 7f553f674700 -1 osd.6 0 waiting for initial osdmap 2024-03-28T05:18:11.530 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:18:11 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:18:11.224+0000 7f553b467700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T05:18:12.190 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T05:18:12.190 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T05:18:12.190 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-28T05:18:12.190 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-28T05:18:12.190 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110599 s, 948 MB/s 2024-03-28T05:18:12.190 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-28T05:18:12.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:12 smithi113 ceph-mon[37164]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:12.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:12 smithi113 ceph-mon[37164]: from='osd.6 [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T05:18:12.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:12 smithi113 ceph-mon[37164]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T05:18:12.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:12 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T05:18:12.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:12 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T05:18:12.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:12 smithi098 ceph-mon[36087]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:12.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:12 smithi098 ceph-mon[36087]: from='osd.6 [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T05:18:12.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:12 smithi098 ceph-mon[36087]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T05:18:12.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:12 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T05:18:12.500 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:12 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T05:18:12.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:12 smithi023 ceph-mon[31954]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:12.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:12 smithi023 ceph-mon[31954]: from='osd.6 [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T05:18:12.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:12 smithi023 ceph-mon[31954]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T05:18:12.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:12 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T05:18:12.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:12 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T05:18:12.840 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_2 2024-03-28T05:18:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T05:18:13.453 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:13 smithi098 ceph-mon[36087]: purged_snaps scrub starts 2024-03-28T05:18:13.453 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:13 smithi098 ceph-mon[36087]: purged_snaps scrub ok 2024-03-28T05:18:13.453 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:13 smithi098 ceph-mon[36087]: osd.6 [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176] boot 2024-03-28T05:18:13.453 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:13 smithi098 ceph-mon[36087]: osdmap e46: 7 total, 7 up, 7 in 2024-03-28T05:18:13.453 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:13 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T05:18:13.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:13 smithi113 ceph-mon[37164]: purged_snaps scrub starts 2024-03-28T05:18:13.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:13 smithi113 ceph-mon[37164]: purged_snaps scrub ok 2024-03-28T05:18:13.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:13 smithi113 ceph-mon[37164]: osd.6 [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176] boot 2024-03-28T05:18:13.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:13 smithi113 ceph-mon[37164]: osdmap e46: 7 total, 7 up, 7 in 2024-03-28T05:18:13.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:13 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T05:18:13.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:13 smithi023 ceph-mon[31954]: purged_snaps scrub starts 2024-03-28T05:18:13.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:13 smithi023 ceph-mon[31954]: purged_snaps scrub ok 2024-03-28T05:18:13.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:13 smithi023 ceph-mon[31954]: osd.6 [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176] boot 2024-03-28T05:18:13.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:13 smithi023 ceph-mon[31954]: osdmap e46: 7 total, 7 up, 7 in 2024-03-28T05:18:13.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:13 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T05:18:14.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:14 smithi113 ceph-mon[37164]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:14.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:14 smithi113 ceph-mon[37164]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T05:18:14.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:14 smithi113 ceph-mon[37164]: Detected new or changed devices on smithi023 2024-03-28T05:18:14.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:14 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:14.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:14 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:14.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:14 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:18:14.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:14 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:18:14.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:14 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:18:14.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:14 smithi113 ceph-mon[37164]: Adjusting osd_memory_target on smithi023 to 5696M 2024-03-28T05:18:14.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:14 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:14.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:14 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:14.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:14 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:18:14.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:14 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:14.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:14 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:18:14.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:14 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:14.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:14 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:18:14.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:14 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:14.516 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:14 smithi098 ceph-mon[36087]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:14.517 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:14 smithi098 ceph-mon[36087]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T05:18:14.517 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:14 smithi098 ceph-mon[36087]: Detected new or changed devices on smithi023 2024-03-28T05:18:14.517 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:14 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:14.517 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:14 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:14.517 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:14 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:18:14.517 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:14 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:18:14.517 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:14 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:18:14.517 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:14 smithi098 ceph-mon[36087]: Adjusting osd_memory_target on smithi023 to 5696M 2024-03-28T05:18:14.517 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:14 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:14.517 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:14 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:14.517 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:14 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:18:14.517 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:14 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:14.517 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:14 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:18:14.517 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:14 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:14.517 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:14 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:18:14.518 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:14 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:14.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:14 smithi023 ceph-mon[31954]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-28T05:18:14.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:14 smithi023 ceph-mon[31954]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T05:18:14.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:14 smithi023 ceph-mon[31954]: Detected new or changed devices on smithi023 2024-03-28T05:18:14.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:14 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:14.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:14 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:14.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:14 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:18:14.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:14 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:18:14.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:14 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:18:14.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:14 smithi023 ceph-mon[31954]: Adjusting osd_memory_target on smithi023 to 5696M 2024-03-28T05:18:14.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:14 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:14.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:14 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:14.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:14 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:18:14.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:14 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:14.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:14 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:18:14.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:14 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:14.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:14 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:18:14.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:14 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:14.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.735+0000 7efdd367a700 1 -- 172.21.15.98:0/100650281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc1010b0 msgr2=0x7efdcc1014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:18:14.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.735+0000 7efdd367a700 1 --2- 172.21.15.98:0/100650281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc1010b0 0x7efdcc1014d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efdc800b0d0 tx=0x7efdc800b3e0 comp rx=0 tx=0).stop 2024-03-28T05:18:14.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.735+0000 7efdd367a700 1 -- 172.21.15.98:0/100650281 shutdown_connections 2024-03-28T05:18:14.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.735+0000 7efdd367a700 1 --2- 172.21.15.98:0/100650281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdcc102c50 0x7efdcc109500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:18:14.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.735+0000 7efdd367a700 1 --2- 172.21.15.98:0/100650281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc1010b0 0x7efdcc1014d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:18:14.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.735+0000 7efdd367a700 1 --2- 172.21.15.98:0/100650281 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdcc1022b0 0x7efdcc102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:18:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.735+0000 7efdd367a700 1 -- 172.21.15.98:0/100650281 >> 172.21.15.98:0/100650281 conn(0x7efdcc0fc650 msgr2=0x7efdcc0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:18:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.735+0000 7efdd367a700 1 -- 172.21.15.98:0/100650281 shutdown_connections 2024-03-28T05:18:14.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.735+0000 7efdd367a700 1 -- 172.21.15.98:0/100650281 wait complete. 2024-03-28T05:18:14.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.736+0000 7efdd367a700 1 Processor -- start 2024-03-28T05:18:14.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.736+0000 7efdd367a700 1 -- start start 2024-03-28T05:18:14.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.736+0000 7efdd367a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdcc1010b0 0x7efdcc06b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:18:14.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.736+0000 7efdd367a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdcc1022b0 0x7efdcc19d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:18:14.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.736+0000 7efdd367a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc102c50 0x7efdcc19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:18:14.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.736+0000 7efdd367a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc10cc90 con 0x7efdcc1022b0 2024-03-28T05:18:14.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.736+0000 7efdd367a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc10cb10 con 0x7efdcc102c50 2024-03-28T05:18:14.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.736+0000 7efdd367a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc10ce10 con 0x7efdcc1010b0 2024-03-28T05:18:14.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.736+0000 7efdd1c17700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc102c50 0x7efdcc19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:18:14.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.736+0000 7efdd1c17700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc102c50 0x7efdcc19f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36246/0 (socket says 172.21.15.98:36246) 2024-03-28T05:18:14.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.736+0000 7efdd1c17700 1 -- 172.21.15.98:0/485329202 learned_addr learned my addr 172.21.15.98:0/485329202 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-28T05:18:14.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.736+0000 7efdd0c15700 1 --2- 172.21.15.98:0/485329202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdcc1022b0 0x7efdcc19d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:18:14.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.736+0000 7efdd1416700 1 --2- 172.21.15.98:0/485329202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdcc1010b0 0x7efdcc06b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:18:14.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.736+0000 7efdd1c17700 1 -- 172.21.15.98:0/485329202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdcc1010b0 msgr2=0x7efdcc06b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:18:14.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.736+0000 7efdd1c17700 1 --2- 172.21.15.98:0/485329202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdcc1010b0 0x7efdcc06b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:18:14.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.736+0000 7efdd1c17700 1 -- 172.21.15.98:0/485329202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdcc1022b0 msgr2=0x7efdcc19d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:18:14.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.736+0000 7efdd1c17700 1 --2- 172.21.15.98:0/485329202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdcc1022b0 0x7efdcc19d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:18:14.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.736+0000 7efdd1c17700 1 -- 172.21.15.98:0/485329202 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdcc1a0030 con 0x7efdcc102c50 2024-03-28T05:18:14.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.736+0000 7efdd1c17700 1 --2- 172.21.15.98:0/485329202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc102c50 0x7efdcc19f760 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efdc400d980 tx=0x7efdc400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:18:14.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.737+0000 7efdc27fc700 1 -- 172.21.15.98:0/485329202 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc4017070 con 0x7efdcc102c50 2024-03-28T05:18:14.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.737+0000 7efdc27fc700 1 -- 172.21.15.98:0/485329202 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdc4005020 con 0x7efdcc102c50 2024-03-28T05:18:14.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.737+0000 7efdd367a700 1 -- 172.21.15.98:0/485329202 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdcc1a0320 con 0x7efdcc102c50 2024-03-28T05:18:14.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.737+0000 7efdc27fc700 1 -- 172.21.15.98:0/485329202 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc400fbf0 con 0x7efdcc102c50 2024-03-28T05:18:14.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.737+0000 7efdd367a700 1 -- 172.21.15.98:0/485329202 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdcc1a08b0 con 0x7efdcc102c50 2024-03-28T05:18:14.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.738+0000 7efdd367a700 1 -- 172.21.15.98:0/485329202 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdcc068e50 con 0x7efdcc102c50 2024-03-28T05:18:14.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.739+0000 7efdc27fc700 1 -- 172.21.15.98:0/485329202 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7efdc40051d0 con 0x7efdcc102c50 2024-03-28T05:18:14.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.739+0000 7efdc27fc700 1 --2- 172.21.15.98:0/485329202 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7efdb80a2910 0x7efdb80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:18:14.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.739+0000 7efdc27fc700 1 -- 172.21.15.98:0/485329202 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4965+0+0 (secure 0 0 0) 0x7efdc40cde20 con 0x7efdcc102c50 2024-03-28T05:18:14.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.739+0000 7efdd1416700 1 --2- 172.21.15.98:0/485329202 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7efdb80a2910 0x7efdb80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:18:14.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.740+0000 7efdd1416700 1 --2- 172.21.15.98:0/485329202 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7efdb80a2910 0x7efdb80a4dd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efdc800b770 tx=0x7efdc800b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:18:14.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.743+0000 7efdc27fc700 1 -- 172.21.15.98:0/485329202 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efdc4020020 con 0x7efdcc102c50 2024-03-28T05:18:14.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:14.947+0000 7efdd367a700 1 -- 172.21.15.98:0/485329202 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7efdcc0625d0 con 0x7efdb80a2910 2024-03-28T05:18:15.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:15 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:18:15.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:15 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:18:15.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:15 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:15.545 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:15 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:18:15.545 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:15 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:18:15.545 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:15 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:15.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:15 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:18:15.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:15 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:18:15.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:15 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:16.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:16 smithi023 ceph-mon[31954]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:16.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:16 smithi023 ceph-mon[31954]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:18:16.572 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:16 smithi098 ceph-mon[36087]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:16.572 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:16 smithi098 ceph-mon[36087]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:18:16.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:16 smithi113 ceph-mon[37164]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:16.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:16 smithi113 ceph-mon[37164]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:18:18.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:18 smithi023 ceph-mon[31954]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:18.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:18 smithi023 ceph-mon[31954]: from='client.? 172.21.15.98:0/3179161664' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edc008a7-b07b-40a8-88b7-1614ba38a775"}]: dispatch 2024-03-28T05:18:18.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:18 smithi023 ceph-mon[31954]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edc008a7-b07b-40a8-88b7-1614ba38a775"}]: dispatch 2024-03-28T05:18:18.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:18 smithi023 ceph-mon[31954]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "edc008a7-b07b-40a8-88b7-1614ba38a775"}]': finished 2024-03-28T05:18:18.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:18 smithi023 ceph-mon[31954]: osdmap e48: 8 total, 7 up, 8 in 2024-03-28T05:18:18.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:18 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T05:18:18.543 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:18 smithi113 ceph-mon[37164]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:18.543 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:18 smithi113 ceph-mon[37164]: from='client.? 172.21.15.98:0/3179161664' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edc008a7-b07b-40a8-88b7-1614ba38a775"}]: dispatch 2024-03-28T05:18:18.543 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:18 smithi113 ceph-mon[37164]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edc008a7-b07b-40a8-88b7-1614ba38a775"}]: dispatch 2024-03-28T05:18:18.543 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:18 smithi113 ceph-mon[37164]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "edc008a7-b07b-40a8-88b7-1614ba38a775"}]': finished 2024-03-28T05:18:18.543 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:18 smithi113 ceph-mon[37164]: osdmap e48: 8 total, 7 up, 8 in 2024-03-28T05:18:18.543 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:18 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T05:18:18.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:18 smithi098 ceph-mon[36087]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:18.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:18 smithi098 ceph-mon[36087]: from='client.? 172.21.15.98:0/3179161664' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edc008a7-b07b-40a8-88b7-1614ba38a775"}]: dispatch 2024-03-28T05:18:18.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:18 smithi098 ceph-mon[36087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edc008a7-b07b-40a8-88b7-1614ba38a775"}]: dispatch 2024-03-28T05:18:18.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:18 smithi098 ceph-mon[36087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "edc008a7-b07b-40a8-88b7-1614ba38a775"}]': finished 2024-03-28T05:18:18.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:18 smithi098 ceph-mon[36087]: osdmap e48: 8 total, 7 up, 8 in 2024-03-28T05:18:18.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:18 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T05:18:19.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:19 smithi023 ceph-mon[31954]: from='client.? 172.21.15.98:0/1399008093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:18:19.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:19 smithi098 ceph-mon[36087]: from='client.? 172.21.15.98:0/1399008093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:18:19.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:19 smithi113 ceph-mon[37164]: from='client.? 172.21.15.98:0/1399008093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:18:20.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:20 smithi023 ceph-mon[31954]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:20.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:20 smithi098 ceph-mon[36087]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:20.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:20 smithi113 ceph-mon[37164]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:22.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:22 smithi023 ceph-mon[31954]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:22.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:22 smithi098 ceph-mon[36087]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:22.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:22 smithi113 ceph-mon[37164]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:24.364 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:24 smithi098 ceph-mon[36087]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:24.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:24 smithi023 ceph-mon[31954]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:24.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:24 smithi113 ceph-mon[37164]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:25.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:25 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T05:18:25.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:25 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:25.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:25 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T05:18:25.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:25 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:25.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:25 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T05:18:25.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:25 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:26.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:26 smithi098 ceph-mon[36087]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:26.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:26 smithi098 ceph-mon[36087]: Deploying daemon osd.7 on smithi098 2024-03-28T05:18:26.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:26 smithi113 ceph-mon[37164]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:26.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:26 smithi113 ceph-mon[37164]: Deploying daemon osd.7 on smithi098 2024-03-28T05:18:26.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:26 smithi023 ceph-mon[31954]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:26.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:26 smithi023 ceph-mon[31954]: Deploying daemon osd.7 on smithi098 2024-03-28T05:18:28.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:28 smithi098 ceph-mon[36087]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:28.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:28 smithi113 ceph-mon[37164]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:28.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:28 smithi023 ceph-mon[31954]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:30.456 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:30 smithi098 ceph-mon[36087]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:30.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:30 smithi113 ceph-mon[37164]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:30.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:30 smithi023 ceph-mon[31954]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:32.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:31 smithi098 ceph-mon[36087]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:32.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:31 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:32.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:31 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:32.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:31 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:18:32.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:31 smithi113 ceph-mon[37164]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:32.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:31 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:32.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:31 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:32.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:31 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:18:32.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:31 smithi023 ceph-mon[31954]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:32.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:31 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:32.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:31 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:32.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:31 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:18:34.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:33 smithi098 ceph-mon[36087]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:34.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:33 smithi113 ceph-mon[37164]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:34.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:33 smithi023 ceph-mon[31954]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:34.362 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 7 on host 'smithi098' 2024-03-28T05:18:34.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:34.363+0000 7efdc27fc700 1 -- 172.21.15.98:0/485329202 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7efdcc0625d0 con 0x7efdb80a2910 2024-03-28T05:18:34.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:34.367+0000 7efdd367a700 1 -- 172.21.15.98:0/485329202 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7efdb80a2910 msgr2=0x7efdb80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:18:34.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:34.367+0000 7efdd367a700 1 --2- 172.21.15.98:0/485329202 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7efdb80a2910 0x7efdb80a4dd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efdc800b770 tx=0x7efdc800b040 comp rx=0 tx=0).stop 2024-03-28T05:18:34.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:34.367+0000 7efdd367a700 1 -- 172.21.15.98:0/485329202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc102c50 msgr2=0x7efdcc19f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:18:34.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:34.367+0000 7efdd367a700 1 --2- 172.21.15.98:0/485329202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc102c50 0x7efdcc19f760 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efdc400d980 tx=0x7efdc400dc90 comp rx=0 tx=0).stop 2024-03-28T05:18:34.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:34.367+0000 7efdd367a700 1 -- 172.21.15.98:0/485329202 shutdown_connections 2024-03-28T05:18:34.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:34.367+0000 7efdd367a700 1 --2- 172.21.15.98:0/485329202 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7efdb80a2910 0x7efdb80a4dd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:18:34.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:34.367+0000 7efdd367a700 1 --2- 172.21.15.98:0/485329202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdcc102c50 0x7efdcc19f760 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:18:34.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:34.367+0000 7efdd367a700 1 --2- 172.21.15.98:0/485329202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdcc1010b0 0x7efdcc06b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:18:34.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:34.367+0000 7efdd367a700 1 --2- 172.21.15.98:0/485329202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdcc1022b0 0x7efdcc19d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:18:34.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:34.367+0000 7efdd367a700 1 -- 172.21.15.98:0/485329202 >> 172.21.15.98:0/485329202 conn(0x7efdcc0fc650 msgr2=0x7efdcc105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:18:34.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:34.367+0000 7efdd367a700 1 -- 172.21.15.98:0/485329202 shutdown_connections 2024-03-28T05:18:34.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:18:34.367+0000 7efdd367a700 1 -- 172.21.15.98:0/485329202 wait complete. 2024-03-28T05:18:35.038 DEBUG:teuthology.orchestra.run.smithi098:osd.7> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.7.service 2024-03-28T05:18:35.040 INFO:tasks.cephadm:Deploying osd.8 on smithi113 with /dev/vg_nvme/lv_2... 2024-03-28T05:18:35.040 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T05:18:35.287 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T05:18:35.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:35 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:35.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:35 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:35.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:35 smithi098 ceph-mon[36087]: from='osd.7 [v2:172.21.15.98:6816/1985546028,v1:172.21.15.98:6817/1985546028]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T05:18:35.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:35 smithi098 ceph-mon[36087]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T05:18:35.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:35 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:35.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:35 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:35.643 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:35 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:35.643 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:35 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:35.643 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:35 smithi113 ceph-mon[37164]: from='osd.7 [v2:172.21.15.98:6816/1985546028,v1:172.21.15.98:6817/1985546028]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T05:18:35.643 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:35 smithi113 ceph-mon[37164]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T05:18:35.643 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:35 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:35.643 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:35 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:35.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:35 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:35.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:35 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:35.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:35 smithi023 ceph-mon[31954]: from='osd.7 [v2:172.21.15.98:6816/1985546028,v1:172.21.15.98:6817/1985546028]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T05:18:35.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:35 smithi023 ceph-mon[31954]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T05:18:35.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:35 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:35.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:35 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:36.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:36 smithi113 ceph-mon[37164]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:36.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:36 smithi113 ceph-mon[37164]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T05:18:36.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:36 smithi113 ceph-mon[37164]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T05:18:36.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:36 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T05:18:36.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:36 smithi113 ceph-mon[37164]: from='osd.7 [v2:172.21.15.98:6816/1985546028,v1:172.21.15.98:6817/1985546028]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:18:36.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:36 smithi113 ceph-mon[37164]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:18:36.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:36 smithi098 ceph-mon[36087]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:36.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:36 smithi098 ceph-mon[36087]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T05:18:36.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:36 smithi098 ceph-mon[36087]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T05:18:36.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:36 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T05:18:36.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:36 smithi098 ceph-mon[36087]: from='osd.7 [v2:172.21.15.98:6816/1985546028,v1:172.21.15.98:6817/1985546028]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:18:36.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:36 smithi098 ceph-mon[36087]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:18:36.615 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:18:36 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:18:36.365+0000 7fdcce757700 -1 osd.7 0 waiting for initial osdmap 2024-03-28T05:18:36.615 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:18:36 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:18:36.373+0000 7fdcca54a700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T05:18:36.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:36 smithi023 ceph-mon[31954]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:36.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:36 smithi023 ceph-mon[31954]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T05:18:36.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:36 smithi023 ceph-mon[31954]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T05:18:36.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:36 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T05:18:36.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:36 smithi023 ceph-mon[31954]: from='osd.7 [v2:172.21.15.98:6816/1985546028,v1:172.21.15.98:6817/1985546028]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:18:36.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:36 smithi023 ceph-mon[31954]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:18:37.520 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:37 smithi113 ceph-mon[37164]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-28T05:18:37.520 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:37 smithi113 ceph-mon[37164]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T05:18:37.520 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:37 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T05:18:37.520 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:37 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T05:18:37.520 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T05:18:37.521 INFO:teuthology.orchestra.run.smithi113.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T05:18:37.521 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10+0 records in 2024-03-28T05:18:37.521 INFO:teuthology.orchestra.run.smithi113.stderr:10+0 records out 2024-03-28T05:18:37.521 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00868278 s, 1.2 GB/s 2024-03-28T05:18:37.521 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping successful for: 2024-03-28T05:18:37.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:37 smithi098 ceph-mon[36087]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-28T05:18:37.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:37 smithi098 ceph-mon[36087]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T05:18:37.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:37 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T05:18:37.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:37 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T05:18:37.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:37 smithi023 ceph-mon[31954]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-28T05:18:37.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:37 smithi023 ceph-mon[31954]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T05:18:37.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:37 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T05:18:37.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:37 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T05:18:38.193 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi113:vg_nvme/lv_2 2024-03-28T05:18:38.436 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T05:18:38.487 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:38 smithi113 ceph-mon[37164]: purged_snaps scrub starts 2024-03-28T05:18:38.487 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:38 smithi113 ceph-mon[37164]: purged_snaps scrub ok 2024-03-28T05:18:38.487 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:38 smithi113 ceph-mon[37164]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:38.487 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:38 smithi113 ceph-mon[37164]: osd.7 [v2:172.21.15.98:6816/1985546028,v1:172.21.15.98:6817/1985546028] boot 2024-03-28T05:18:38.487 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:38 smithi113 ceph-mon[37164]: osdmap e51: 8 total, 8 up, 8 in 2024-03-28T05:18:38.487 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:38 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T05:18:38.681 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:38 smithi023 ceph-mon[31954]: purged_snaps scrub starts 2024-03-28T05:18:38.681 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:38 smithi023 ceph-mon[31954]: purged_snaps scrub ok 2024-03-28T05:18:38.681 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:38 smithi023 ceph-mon[31954]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:38.681 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:38 smithi023 ceph-mon[31954]: osd.7 [v2:172.21.15.98:6816/1985546028,v1:172.21.15.98:6817/1985546028] boot 2024-03-28T05:18:38.681 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:38 smithi023 ceph-mon[31954]: osdmap e51: 8 total, 8 up, 8 in 2024-03-28T05:18:38.681 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:38 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T05:18:38.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:38 smithi098 ceph-mon[36087]: purged_snaps scrub starts 2024-03-28T05:18:38.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:38 smithi098 ceph-mon[36087]: purged_snaps scrub ok 2024-03-28T05:18:38.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:38 smithi098 ceph-mon[36087]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-28T05:18:38.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:38 smithi098 ceph-mon[36087]: osd.7 [v2:172.21.15.98:6816/1985546028,v1:172.21.15.98:6817/1985546028] boot 2024-03-28T05:18:38.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:38 smithi098 ceph-mon[36087]: osdmap e51: 8 total, 8 up, 8 in 2024-03-28T05:18:38.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:38 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T05:18:39.549 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:39 smithi113 ceph-mon[37164]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T05:18:39.550 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:39 smithi113 ceph-mon[37164]: Detected new or changed devices on smithi098 2024-03-28T05:18:39.550 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:39 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:39.550 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:39 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:39.550 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:39 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:18:39.550 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:39 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:18:39.550 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:39 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:18:39.550 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:39 smithi113 ceph-mon[37164]: Adjusting osd_memory_target on smithi098 to 5693M 2024-03-28T05:18:39.550 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:39 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:39.550 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:39 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:39.550 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:39 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:18:39.550 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:39 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:39.550 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:39 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:18:39.550 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:39 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:39.550 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:39 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:18:39.550 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:39 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:39.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:39 smithi023 ceph-mon[31954]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T05:18:39.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:39 smithi023 ceph-mon[31954]: Detected new or changed devices on smithi098 2024-03-28T05:18:39.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:39 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:39.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:39 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:39.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:39 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:18:39.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:39 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:18:39.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:39 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:18:39.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:39 smithi023 ceph-mon[31954]: Adjusting osd_memory_target on smithi098 to 5693M 2024-03-28T05:18:39.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:39 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:39.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:39 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:39.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:39 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:18:39.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:39 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:39.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:39 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:18:39.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:39 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:39.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:39 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:18:39.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:39 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:39.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:39 smithi098 ceph-mon[36087]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T05:18:39.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:39 smithi098 ceph-mon[36087]: Detected new or changed devices on smithi098 2024-03-28T05:18:39.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:39 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:39.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:39 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:39.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:39 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:18:39.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:39 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:18:39.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:39 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:18:39.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:39 smithi098 ceph-mon[36087]: Adjusting osd_memory_target on smithi098 to 5693M 2024-03-28T05:18:39.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:39 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:39.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:39 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:39.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:39 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:18:39.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:39 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:39.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:39 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:18:39.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:39 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:39.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:39 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:18:39.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:39 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:40.128 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.126+0000 7f7f37b97700 1 -- 172.21.15.113:0/3958864037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f30103830 msgr2=0x7f7f30103c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:18:40.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.126+0000 7f7f37b97700 1 --2- 172.21.15.113:0/3958864037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f30103830 0x7f7f30103c50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009990 tx=0x7f7f2c009ca0 comp rx=0 tx=0).stop 2024-03-28T05:18:40.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.126+0000 7f7f37b97700 1 -- 172.21.15.113:0/3958864037 shutdown_connections 2024-03-28T05:18:40.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.126+0000 7f7f37b97700 1 --2- 172.21.15.113:0/3958864037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f30100c50 0x7f7f3010a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:18:40.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.126+0000 7f7f37b97700 1 --2- 172.21.15.113:0/3958864037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f30103830 0x7f7f30103c50 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:18:40.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.126+0000 7f7f37b97700 1 --2- 172.21.15.113:0/3958864037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f301002b0 0x7f7f30100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:18:40.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.127+0000 7f7f37b97700 1 -- 172.21.15.113:0/3958864037 >> 172.21.15.113:0/3958864037 conn(0x7f7f300fbe30 msgr2=0x7f7f300fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:18:40.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.127+0000 7f7f37b97700 1 -- 172.21.15.113:0/3958864037 shutdown_connections 2024-03-28T05:18:40.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.127+0000 7f7f37b97700 1 -- 172.21.15.113:0/3958864037 wait complete. 2024-03-28T05:18:40.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.127+0000 7f7f37b97700 1 Processor -- start 2024-03-28T05:18:40.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.127+0000 7f7f37b97700 1 -- start start 2024-03-28T05:18:40.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.127+0000 7f7f37b97700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f301002b0 0x7f7f30198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:18:40.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.127+0000 7f7f37b97700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f30100c50 0x7f7f30198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:18:40.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.127+0000 7f7f37b97700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f30103830 0x7f7f3019f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:18:40.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.127+0000 7f7f37b97700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3010ddf0 con 0x7f7f301002b0 2024-03-28T05:18:40.129 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.127+0000 7f7f37b97700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3010dc70 con 0x7f7f30100c50 2024-03-28T05:18:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.127+0000 7f7f37b97700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3010df70 con 0x7f7f30103830 2024-03-28T05:18:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.127+0000 7f7f36134700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f30103830 0x7f7f3019f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:18:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.127+0000 7f7f36134700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f30103830 0x7f7f3019f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:57614/0 (socket says 172.21.15.113:57614) 2024-03-28T05:18:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.127+0000 7f7f36134700 1 -- 172.21.15.113:0/4159458885 learned_addr learned my addr 172.21.15.113:0/4159458885 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-03-28T05:18:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.127+0000 7f7f35933700 1 --2- 172.21.15.113:0/4159458885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f301002b0 0x7f7f30198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:18:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.128+0000 7f7f35132700 1 --2- 172.21.15.113:0/4159458885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f30100c50 0x7f7f30198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:18:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.128+0000 7f7f36134700 1 -- 172.21.15.113:0/4159458885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f30100c50 msgr2=0x7f7f30198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:18:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.128+0000 7f7f36134700 1 --2- 172.21.15.113:0/4159458885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f30100c50 0x7f7f30198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:18:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.128+0000 7f7f36134700 1 -- 172.21.15.113:0/4159458885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f301002b0 msgr2=0x7f7f30198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:18:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.128+0000 7f7f36134700 1 --2- 172.21.15.113:0/4159458885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f301002b0 0x7f7f30198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:18:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.128+0000 7f7f36134700 1 -- 172.21.15.113:0/4159458885 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f301a0070 con 0x7f7f30103830 2024-03-28T05:18:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.128+0000 7f7f36134700 1 --2- 172.21.15.113:0/4159458885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f30103830 0x7f7f3019f9d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7f28007f10 tx=0x7f7f2800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:18:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.128+0000 7f7f26ffd700 1 -- 172.21.15.113:0/4159458885 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f28004bb0 con 0x7f7f30103830 2024-03-28T05:18:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.128+0000 7f7f37b97700 1 -- 172.21.15.113:0/4159458885 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f301a0360 con 0x7f7f30103830 2024-03-28T05:18:40.130 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.128+0000 7f7f26ffd700 1 -- 172.21.15.113:0/4159458885 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7f28004d50 con 0x7f7f30103830 2024-03-28T05:18:40.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.128+0000 7f7f37b97700 1 -- 172.21.15.113:0/4159458885 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f301a08a0 con 0x7f7f30103830 2024-03-28T05:18:40.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.128+0000 7f7f26ffd700 1 -- 172.21.15.113:0/4159458885 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f280059f0 con 0x7f7f30103830 2024-03-28T05:18:40.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.129+0000 7f7f37b97700 1 -- 172.21.15.113:0/4159458885 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f30068e50 con 0x7f7f30103830 2024-03-28T05:18:40.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.130+0000 7f7f26ffd700 1 -- 172.21.15.113:0/4159458885 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7f2801a020 con 0x7f7f30103830 2024-03-28T05:18:40.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.131+0000 7f7f26ffd700 1 --2- 172.21.15.113:0/4159458885 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f7f1c0a2890 0x7f7f1c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:18:40.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.131+0000 7f7f26ffd700 1 -- 172.21.15.113:0/4159458885 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f7f280cd640 con 0x7f7f30103830 2024-03-28T05:18:40.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.131+0000 7f7f35933700 1 --2- 172.21.15.113:0/4159458885 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f7f1c0a2890 0x7f7f1c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:18:40.131 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.131+0000 7f7f35933700 1 --2- 172.21.15.113:0/4159458885 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f7f1c0a2890 0x7f7f1c0a4d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c003890 tx=0x7f7f2c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:18:40.132 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.135+0000 7f7f26ffd700 1 -- 172.21.15.113:0/4159458885 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7f28010070 con 0x7f7f30103830 2024-03-28T05:18:40.337 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:40.340+0000 7f7f37b97700 1 -- 172.21.15.113:0/4159458885 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f300625d0 con 0x7f7f1c0a2890 2024-03-28T05:18:40.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:40 smithi113 ceph-mon[37164]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:40.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:40 smithi113 ceph-mon[37164]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T05:18:40.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:40 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:18:40.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:40 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:18:40.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:40 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:40.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:40 smithi023 ceph-mon[31954]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:40.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:40 smithi023 ceph-mon[31954]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T05:18:40.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:40 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:18:40.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:40 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:18:40.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:40 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:40.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:40 smithi098 ceph-mon[36087]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:40.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:40 smithi098 ceph-mon[36087]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T05:18:40.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:40 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:18:40.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:40 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:18:40.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:40 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:41.648 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:41 smithi113 ceph-mon[37164]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:18:41.648 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:41 smithi113 ceph-mon[37164]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T05:18:41.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:41 smithi023 ceph-mon[31954]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:18:41.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:41 smithi023 ceph-mon[31954]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T05:18:41.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:41 smithi098 ceph-mon[36087]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:18:41.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:41 smithi098 ceph-mon[36087]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T05:18:42.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:42 smithi113 ceph-mon[37164]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:42.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:42 smithi113 ceph-mon[37164]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T05:18:42.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:42 smithi023 ceph-mon[31954]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:42.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:42 smithi023 ceph-mon[31954]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T05:18:42.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:42 smithi098 ceph-mon[36087]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:42.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:42 smithi098 ceph-mon[36087]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T05:18:43.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:43 smithi113 ceph-mon[37164]: from='client.? 172.21.15.113:0/4020416102' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56e7b2d1-81f0-47b2-8638-f29832bed8de"}]: dispatch 2024-03-28T05:18:43.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:43 smithi113 ceph-mon[37164]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56e7b2d1-81f0-47b2-8638-f29832bed8de"}]: dispatch 2024-03-28T05:18:43.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:43 smithi113 ceph-mon[37164]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56e7b2d1-81f0-47b2-8638-f29832bed8de"}]': finished 2024-03-28T05:18:43.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:43 smithi113 ceph-mon[37164]: osdmap e56: 9 total, 8 up, 9 in 2024-03-28T05:18:43.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:43 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T05:18:43.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:43 smithi023 ceph-mon[31954]: from='client.? 172.21.15.113:0/4020416102' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56e7b2d1-81f0-47b2-8638-f29832bed8de"}]: dispatch 2024-03-28T05:18:43.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:43 smithi023 ceph-mon[31954]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56e7b2d1-81f0-47b2-8638-f29832bed8de"}]: dispatch 2024-03-28T05:18:43.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:43 smithi023 ceph-mon[31954]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56e7b2d1-81f0-47b2-8638-f29832bed8de"}]': finished 2024-03-28T05:18:43.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:43 smithi023 ceph-mon[31954]: osdmap e56: 9 total, 8 up, 9 in 2024-03-28T05:18:43.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:43 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T05:18:43.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:43 smithi098 ceph-mon[36087]: from='client.? 172.21.15.113:0/4020416102' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56e7b2d1-81f0-47b2-8638-f29832bed8de"}]: dispatch 2024-03-28T05:18:43.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:43 smithi098 ceph-mon[36087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56e7b2d1-81f0-47b2-8638-f29832bed8de"}]: dispatch 2024-03-28T05:18:43.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:43 smithi098 ceph-mon[36087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56e7b2d1-81f0-47b2-8638-f29832bed8de"}]': finished 2024-03-28T05:18:43.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:43 smithi098 ceph-mon[36087]: osdmap e56: 9 total, 8 up, 9 in 2024-03-28T05:18:43.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:43 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T05:18:44.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:44 smithi113 ceph-mon[37164]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:44.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:44 smithi113 ceph-mon[37164]: from='client.? 172.21.15.113:0/1973956514' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:18:44.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:44 smithi023 ceph-mon[31954]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:44.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:44 smithi023 ceph-mon[31954]: from='client.? 172.21.15.113:0/1973956514' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:18:44.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:44 smithi098 ceph-mon[36087]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:44.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:44 smithi098 ceph-mon[36087]: from='client.? 172.21.15.113:0/1973956514' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:18:46.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:46 smithi113 ceph-mon[37164]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:46.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:46 smithi023 ceph-mon[31954]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:46.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:46 smithi098 ceph-mon[36087]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:48.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:48 smithi113 ceph-mon[37164]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:48.678 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:48 smithi098 ceph-mon[36087]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:48.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:48 smithi023 ceph-mon[31954]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:50.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:50 smithi113 ceph-mon[37164]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:50.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:50 smithi023 ceph-mon[31954]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:50.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:50 smithi098 ceph-mon[36087]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:51.724 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:51 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T05:18:51.724 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:51 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:51.724 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:51 smithi113 ceph-mon[37164]: Deploying daemon osd.8 on smithi113 2024-03-28T05:18:51.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:51 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T05:18:51.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:51 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:51.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:51 smithi023 ceph-mon[31954]: Deploying daemon osd.8 on smithi113 2024-03-28T05:18:51.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:51 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T05:18:51.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:51 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:18:51.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:51 smithi098 ceph-mon[36087]: Deploying daemon osd.8 on smithi113 2024-03-28T05:18:52.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:52 smithi113 ceph-mon[37164]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:52.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:52 smithi023 ceph-mon[31954]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:52.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:52 smithi098 ceph-mon[36087]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:54.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:54 smithi113 ceph-mon[37164]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:54.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:54 smithi023 ceph-mon[31954]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:54.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:54 smithi098 ceph-mon[36087]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:56.732 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:56 smithi113 ceph-mon[37164]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:56.733 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:56.733 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:56.733 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:18:56.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:56 smithi023 ceph-mon[31954]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:56.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:56.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:56.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:18:56.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:56 smithi098 ceph-mon[36087]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:56.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:56.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:18:56.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:18:58.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:18:58 smithi113 ceph-mon[37164]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:58.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:18:58 smithi023 ceph-mon[31954]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:58.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:18:58 smithi098 ceph-mon[36087]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:18:59.656 INFO:teuthology.orchestra.run.smithi113.stdout:Created osd(s) 8 on host 'smithi113' 2024-03-28T05:18:59.656 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:59.657+0000 7f7f26ffd700 1 -- 172.21.15.113:0/4159458885 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7f300625d0 con 0x7f7f1c0a2890 2024-03-28T05:18:59.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:59.659+0000 7f7f37b97700 1 -- 172.21.15.113:0/4159458885 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f7f1c0a2890 msgr2=0x7f7f1c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:18:59.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:59.659+0000 7f7f37b97700 1 --2- 172.21.15.113:0/4159458885 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f7f1c0a2890 0x7f7f1c0a4d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c003890 tx=0x7f7f2c005fd0 comp rx=0 tx=0).stop 2024-03-28T05:18:59.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:59.659+0000 7f7f37b97700 1 -- 172.21.15.113:0/4159458885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f30103830 msgr2=0x7f7f3019f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:18:59.657 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:59.659+0000 7f7f37b97700 1 --2- 172.21.15.113:0/4159458885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f30103830 0x7f7f3019f9d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7f28007f10 tx=0x7f7f2800d3b0 comp rx=0 tx=0).stop 2024-03-28T05:18:59.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:59.659+0000 7f7f37b97700 1 -- 172.21.15.113:0/4159458885 shutdown_connections 2024-03-28T05:18:59.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:59.659+0000 7f7f37b97700 1 --2- 172.21.15.113:0/4159458885 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f7f1c0a2890 0x7f7f1c0a4d50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:18:59.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:59.659+0000 7f7f37b97700 1 --2- 172.21.15.113:0/4159458885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f30103830 0x7f7f3019f9d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:18:59.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:59.659+0000 7f7f37b97700 1 --2- 172.21.15.113:0/4159458885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f301002b0 0x7f7f30198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:18:59.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:59.659+0000 7f7f37b97700 1 --2- 172.21.15.113:0/4159458885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f30100c50 0x7f7f30198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:18:59.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:59.659+0000 7f7f37b97700 1 -- 172.21.15.113:0/4159458885 >> 172.21.15.113:0/4159458885 conn(0x7f7f300fbe30 msgr2=0x7f7f30106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:18:59.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:59.659+0000 7f7f37b97700 1 -- 172.21.15.113:0/4159458885 shutdown_connections 2024-03-28T05:18:59.658 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:18:59.659+0000 7f7f37b97700 1 -- 172.21.15.113:0/4159458885 wait complete. 2024-03-28T05:19:00.297 DEBUG:teuthology.orchestra.run.smithi113:osd.8> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.8.service 2024-03-28T05:19:00.299 INFO:tasks.cephadm:Deploying osd.9 on smithi023 with /dev/vg_nvme/lv_1... 2024-03-28T05:19:00.299 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T05:19:00.559 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:19:00.923 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:00 smithi023 ceph-mon[31954]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:19:00.924 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:00 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:00.924 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:00 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:00.924 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:00 smithi023 ceph-mon[31954]: from='osd.8 [v2:172.21.15.113:6816/3534876081,v1:172.21.15.113:6817/3534876081]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T05:19:00.924 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:00 smithi023 ceph-mon[31954]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T05:19:00.924 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:00 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:00.924 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:00 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:00.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:00 smithi113 ceph-mon[37164]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:19:00.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:00 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:00.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:00 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:00.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:00 smithi113 ceph-mon[37164]: from='osd.8 [v2:172.21.15.113:6816/3534876081,v1:172.21.15.113:6817/3534876081]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T05:19:00.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:00 smithi113 ceph-mon[37164]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T05:19:00.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:00 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:00.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:00 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:01.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:00 smithi098 ceph-mon[36087]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:19:01.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:00 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:01.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:00 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:01.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:00 smithi098 ceph-mon[36087]: from='osd.8 [v2:172.21.15.113:6816/3534876081,v1:172.21.15.113:6817/3534876081]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T05:19:01.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:00 smithi098 ceph-mon[36087]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T05:19:01.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:00 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:01.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:00 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:01.899 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:01 smithi023 ceph-mon[31954]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T05:19:01.899 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:01 smithi023 ceph-mon[31954]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T05:19:01.899 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:01 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T05:19:01.899 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:01 smithi023 ceph-mon[31954]: from='osd.8 [v2:172.21.15.113:6816/3534876081,v1:172.21.15.113:6817/3534876081]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:19:01.899 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:01 smithi023 ceph-mon[31954]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:19:01.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:01 smithi113 ceph-mon[37164]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T05:19:01.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:01 smithi113 ceph-mon[37164]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T05:19:01.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:01 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T05:19:01.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:01 smithi113 ceph-mon[37164]: from='osd.8 [v2:172.21.15.113:6816/3534876081,v1:172.21.15.113:6817/3534876081]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:19:01.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:01 smithi113 ceph-mon[37164]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:19:02.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:01 smithi098 ceph-mon[36087]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T05:19:02.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:01 smithi098 ceph-mon[36087]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T05:19:02.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:01 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T05:19:02.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:01 smithi098 ceph-mon[36087]: from='osd.8 [v2:172.21.15.113:6816/3534876081,v1:172.21.15.113:6817/3534876081]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:19:02.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:01 smithi098 ceph-mon[36087]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:19:02.357 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:19:02 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:19:02.192+0000 7efef53f7700 -1 osd.8 0 waiting for initial osdmap 2024-03-28T05:19:02.357 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:19:02 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:19:02.202+0000 7efeed1e2700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T05:19:02.761 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T05:19:02.761 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T05:19:02.761 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-28T05:19:02.761 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-28T05:19:02.761 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0166029 s, 632 MB/s 2024-03-28T05:19:02.762 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-28T05:19:02.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:02 smithi023 ceph-mon[31954]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:19:02.785 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:02 smithi023 ceph-mon[31954]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-28T05:19:02.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:02 smithi023 ceph-mon[31954]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T05:19:02.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:02 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T05:19:02.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:02 smithi023 ceph-mon[31954]: from='osd.8 ' entity='osd.8' 2024-03-28T05:19:02.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:02 smithi023 ceph-mon[31954]: osd.8 [v2:172.21.15.113:6816/3534876081,v1:172.21.15.113:6817/3534876081] boot 2024-03-28T05:19:02.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:02 smithi023 ceph-mon[31954]: osdmap e59: 9 total, 9 up, 9 in 2024-03-28T05:19:02.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:02 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T05:19:02.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:02 smithi113 ceph-mon[37164]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:19:02.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:02 smithi113 ceph-mon[37164]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-28T05:19:02.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:02 smithi113 ceph-mon[37164]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T05:19:02.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:02 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T05:19:02.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:02 smithi113 ceph-mon[37164]: from='osd.8 ' entity='osd.8' 2024-03-28T05:19:02.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:02 smithi113 ceph-mon[37164]: osd.8 [v2:172.21.15.113:6816/3534876081,v1:172.21.15.113:6817/3534876081] boot 2024-03-28T05:19:02.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:02 smithi113 ceph-mon[37164]: osdmap e59: 9 total, 9 up, 9 in 2024-03-28T05:19:02.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:02 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T05:19:03.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:02 smithi098 ceph-mon[36087]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-28T05:19:03.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:02 smithi098 ceph-mon[36087]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-28T05:19:03.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:02 smithi098 ceph-mon[36087]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T05:19:03.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:02 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T05:19:03.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:02 smithi098 ceph-mon[36087]: from='osd.8 ' entity='osd.8' 2024-03-28T05:19:03.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:02 smithi098 ceph-mon[36087]: osd.8 [v2:172.21.15.113:6816/3534876081,v1:172.21.15.113:6817/3534876081] boot 2024-03-28T05:19:03.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:02 smithi098 ceph-mon[36087]: osdmap e59: 9 total, 9 up, 9 in 2024-03-28T05:19:03.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:02 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T05:19:03.568 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_1 2024-03-28T05:19:03.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:03 smithi113 ceph-mon[37164]: purged_snaps scrub starts 2024-03-28T05:19:03.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:03 smithi113 ceph-mon[37164]: purged_snaps scrub ok 2024-03-28T05:19:03.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:03 smithi113 ceph-mon[37164]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T05:19:03.825 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:19:03.866 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:03 smithi023 ceph-mon[31954]: purged_snaps scrub starts 2024-03-28T05:19:03.866 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:03 smithi023 ceph-mon[31954]: purged_snaps scrub ok 2024-03-28T05:19:03.867 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:03 smithi023 ceph-mon[31954]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T05:19:04.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:03 smithi098 ceph-mon[36087]: purged_snaps scrub starts 2024-03-28T05:19:04.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:03 smithi098 ceph-mon[36087]: purged_snaps scrub ok 2024-03-28T05:19:04.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:03 smithi098 ceph-mon[36087]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T05:19:04.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:04 smithi023 ceph-mon[31954]: pgmap v196: 1 pgs: 1 peering; 577 KiB data, 656 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:04.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:04 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:04.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:04 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:04.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:04 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:04.971 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:04 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:04.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:04 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:04.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:04 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:04.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:04 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:04.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:04 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:19:04.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:04 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:04.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:04 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:19:04.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:04 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:04.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:04 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:19:04.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:04 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:04.972 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:04 smithi023 ceph-mon[31954]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T05:19:04.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:04 smithi113 ceph-mon[37164]: pgmap v196: 1 pgs: 1 peering; 577 KiB data, 656 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:04.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:04 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:04.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:04 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:04.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:04 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:04.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:04 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:04.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:04 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:04.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:04 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:04.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:04 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:04.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:04 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:19:04.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:04 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:04.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:04 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:19:04.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:04 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:04.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:04 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:19:04.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:04 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:04.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:04 smithi113 ceph-mon[37164]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T05:19:05.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:04 smithi098 ceph-mon[36087]: pgmap v196: 1 pgs: 1 peering; 577 KiB data, 656 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:05.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:04 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:05.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:04 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:05.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:04 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:05.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:04 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:05.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:04 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:05.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:04 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:05.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:04 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:05.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:04 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:19:05.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:04 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:05.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:04 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:19:05.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:04 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:05.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:04 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:19:05.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:04 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:05.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:04 smithi098 ceph-mon[36087]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T05:19:05.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.426+0000 7ffb985c1700 1 -- 172.21.15.23:0/1910727387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb901021c0 msgr2=0x7ffb901025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:19:05.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.426+0000 7ffb985c1700 1 --2- 172.21.15.23:0/1910727387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb901021c0 0x7ffb901025e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c0099c0 tx=0x7ffb8c009cd0 comp rx=0 tx=0).stop 2024-03-28T05:19:05.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.426+0000 7ffb985c1700 1 -- 172.21.15.23:0/1910727387 shutdown_connections 2024-03-28T05:19:05.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.426+0000 7ffb985c1700 1 --2- 172.21.15.23:0/1910727387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb90103d60 0x7ffb9010a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:05.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.426+0000 7ffb985c1700 1 --2- 172.21.15.23:0/1910727387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb901021c0 0x7ffb901025e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:05.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.426+0000 7ffb985c1700 1 --2- 172.21.15.23:0/1910727387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb901033c0 0x7ffb90103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:05.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.426+0000 7ffb985c1700 1 -- 172.21.15.23:0/1910727387 >> 172.21.15.23:0/1910727387 conn(0x7ffb900fd7a0 msgr2=0x7ffb900ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:19:05.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.426+0000 7ffb985c1700 1 -- 172.21.15.23:0/1910727387 shutdown_connections 2024-03-28T05:19:05.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.426+0000 7ffb985c1700 1 -- 172.21.15.23:0/1910727387 wait complete. 2024-03-28T05:19:05.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.426+0000 7ffb985c1700 1 Processor -- start 2024-03-28T05:19:05.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.426+0000 7ffb985c1700 1 -- start start 2024-03-28T05:19:05.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.427+0000 7ffb985c1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb901021c0 0x7ffb901983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:19:05.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.427+0000 7ffb985c1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb901033c0 0x7ffb90198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:19:05.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.427+0000 7ffb985c1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb90103d60 0x7ffb9019f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:19:05.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.427+0000 7ffb985c1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9010ddb0 con 0x7ffb901033c0 2024-03-28T05:19:05.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.427+0000 7ffb985c1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9010dc30 con 0x7ffb901021c0 2024-03-28T05:19:05.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.427+0000 7ffb985c1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9010df30 con 0x7ffb90103d60 2024-03-28T05:19:05.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.427+0000 7ffb95b5c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb901033c0 0x7ffb90198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:19:05.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.427+0000 7ffb95b5c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb901033c0 0x7ffb90198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33426/0 (socket says 172.21.15.23:33426) 2024-03-28T05:19:05.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.427+0000 7ffb95b5c700 1 -- 172.21.15.23:0/3466453034 learned_addr learned my addr 172.21.15.23:0/3466453034 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:19:05.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.427+0000 7ffb96b5e700 1 --2- 172.21.15.23:0/3466453034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb90103d60 0x7ffb9019f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:19:05.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.427+0000 7ffb9635d700 1 --2- 172.21.15.23:0/3466453034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb901021c0 0x7ffb901983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:19:05.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.427+0000 7ffb95b5c700 1 -- 172.21.15.23:0/3466453034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb90103d60 msgr2=0x7ffb9019f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:19:05.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.427+0000 7ffb95b5c700 1 --2- 172.21.15.23:0/3466453034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb90103d60 0x7ffb9019f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:05.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.427+0000 7ffb95b5c700 1 -- 172.21.15.23:0/3466453034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb901021c0 msgr2=0x7ffb901983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:19:05.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.427+0000 7ffb95b5c700 1 --2- 172.21.15.23:0/3466453034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb901021c0 0x7ffb901983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:05.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.427+0000 7ffb95b5c700 1 -- 172.21.15.23:0/3466453034 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb901a0050 con 0x7ffb901033c0 2024-03-28T05:19:05.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.427+0000 7ffb96b5e700 1 --2- 172.21.15.23:0/3466453034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb90103d60 0x7ffb9019f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:19:05.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.427+0000 7ffb95b5c700 1 --2- 172.21.15.23:0/3466453034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb901033c0 0x7ffb90198930 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ffb8000d980 tx=0x7ffb8000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:19:05.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.427+0000 7ffb877fe700 1 -- 172.21.15.23:0/3466453034 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb80016070 con 0x7ffb901033c0 2024-03-28T05:19:05.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.427+0000 7ffb877fe700 1 -- 172.21.15.23:0/3466453034 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb80009b40 con 0x7ffb901033c0 2024-03-28T05:19:05.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.427+0000 7ffb985c1700 1 -- 172.21.15.23:0/3466453034 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb901a0340 con 0x7ffb901033c0 2024-03-28T05:19:05.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.427+0000 7ffb877fe700 1 -- 172.21.15.23:0/3466453034 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8000f620 con 0x7ffb901033c0 2024-03-28T05:19:05.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.428+0000 7ffb985c1700 1 -- 172.21.15.23:0/3466453034 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb901a0880 con 0x7ffb901033c0 2024-03-28T05:19:05.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.429+0000 7ffb985c1700 1 -- 172.21.15.23:0/3466453034 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb90068e50 con 0x7ffb901033c0 2024-03-28T05:19:05.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.430+0000 7ffb877fe700 1 -- 172.21.15.23:0/3466453034 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ffb80019070 con 0x7ffb901033c0 2024-03-28T05:19:05.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.430+0000 7ffb877fe700 1 --2- 172.21.15.23:0/3466453034 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ffb7c0a2890 0x7ffb7c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:19:05.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.430+0000 7ffb9635d700 1 --2- 172.21.15.23:0/3466453034 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ffb7c0a2890 0x7ffb7c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:19:05.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.430+0000 7ffb877fe700 1 -- 172.21.15.23:0/3466453034 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5858+0+0 (secure 0 0 0) 0x7ffb800cdcc0 con 0x7ffb901033c0 2024-03-28T05:19:05.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.431+0000 7ffb9635d700 1 --2- 172.21.15.23:0/3466453034 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ffb7c0a2890 0x7ffb7c0a4d50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c00b580 tx=0x7ffb8c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:19:05.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.435+0000 7ffb877fe700 1 -- 172.21.15.23:0/3466453034 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffb8009b6f0 con 0x7ffb901033c0 2024-03-28T05:19:05.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:05.651+0000 7ffb985c1700 1 -- 172.21.15.23:0/3466453034 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb900625d0 con 0x7ffb7c0a2890 2024-03-28T05:19:05.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:05 smithi113 ceph-mon[37164]: Detected new or changed devices on smithi113 2024-03-28T05:19:05.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:05 smithi113 ceph-mon[37164]: Adjusting osd_memory_target on smithi113 to 5693M 2024-03-28T05:19:05.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:05 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:19:06.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:05 smithi023 ceph-mon[31954]: Detected new or changed devices on smithi113 2024-03-28T05:19:06.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:05 smithi023 ceph-mon[31954]: Adjusting osd_memory_target on smithi113 to 5693M 2024-03-28T05:19:06.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:05 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:19:06.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:05 smithi098 ceph-mon[36087]: Detected new or changed devices on smithi113 2024-03-28T05:19:06.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:05 smithi098 ceph-mon[36087]: Adjusting osd_memory_target on smithi113 to 5693M 2024-03-28T05:19:06.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:05 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:19:06.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:06 smithi113 ceph-mon[37164]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 656 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:06.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:06 smithi113 ceph-mon[37164]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:19:06.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:19:06.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:06.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:06 smithi113 ceph-mon[37164]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T05:19:07.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:06 smithi023 ceph-mon[31954]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 656 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:07.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:06 smithi023 ceph-mon[31954]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:19:07.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:19:07.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:07.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:06 smithi023 ceph-mon[31954]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T05:19:07.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:06 smithi098 ceph-mon[36087]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 656 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:07.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:06 smithi098 ceph-mon[36087]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:19:07.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:06 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:19:07.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:06 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:07.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:06 smithi098 ceph-mon[36087]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T05:19:07.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:07 smithi023 ceph-mon[31954]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T05:19:07.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:07 smithi113 ceph-mon[37164]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T05:19:08.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:07 smithi098 ceph-mon[36087]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T05:19:08.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:08 smithi113 ceph-mon[37164]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 656 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:08.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:08 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3984972504' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2e90caa-6e9c-42bf-9f18-81f7f47880cd"}]: dispatch 2024-03-28T05:19:08.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:08 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3984972504' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a2e90caa-6e9c-42bf-9f18-81f7f47880cd"}]': finished 2024-03-28T05:19:08.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:08 smithi113 ceph-mon[37164]: osdmap e64: 10 total, 9 up, 10 in 2024-03-28T05:19:08.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:08 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T05:19:09.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:08 smithi023 ceph-mon[31954]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 656 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:09.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:08 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3984972504' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2e90caa-6e9c-42bf-9f18-81f7f47880cd"}]: dispatch 2024-03-28T05:19:09.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:08 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3984972504' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a2e90caa-6e9c-42bf-9f18-81f7f47880cd"}]': finished 2024-03-28T05:19:09.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:08 smithi023 ceph-mon[31954]: osdmap e64: 10 total, 9 up, 10 in 2024-03-28T05:19:09.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:08 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T05:19:09.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:08 smithi098 ceph-mon[36087]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 656 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:09.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:08 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3984972504' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2e90caa-6e9c-42bf-9f18-81f7f47880cd"}]: dispatch 2024-03-28T05:19:09.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:08 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3984972504' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a2e90caa-6e9c-42bf-9f18-81f7f47880cd"}]': finished 2024-03-28T05:19:09.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:08 smithi098 ceph-mon[36087]: osdmap e64: 10 total, 9 up, 10 in 2024-03-28T05:19:09.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T05:19:09.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:09 smithi113 ceph-mon[37164]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-03-28T05:19:09.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:09 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/19744040' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:19:10.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:09 smithi023 ceph-mon[31954]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-03-28T05:19:10.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:09 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/19744040' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:19:10.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:09 smithi098 ceph-mon[36087]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-03-28T05:19:10.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:09 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/19744040' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:19:12.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:11 smithi023 ceph-mon[31954]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T05:19:12.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:11 smithi098 ceph-mon[36087]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T05:19:12.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:11 smithi113 ceph-mon[37164]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T05:19:13.884 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:13 smithi023 ceph-mon[31954]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-28T05:19:14.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:13 smithi098 ceph-mon[36087]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-28T05:19:14.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:13 smithi113 ceph-mon[37164]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-28T05:19:15.955 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:15 smithi023 ceph-mon[31954]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T05:19:16.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:15 smithi098 ceph-mon[36087]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T05:19:16.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:15 smithi113 ceph-mon[37164]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T05:19:16.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:16 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T05:19:16.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:16 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:16.917 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:16 smithi023 ceph-mon[31954]: Deploying daemon osd.9 on smithi023 2024-03-28T05:19:17.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:16 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T05:19:17.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:16 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:17.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:16 smithi098 ceph-mon[36087]: Deploying daemon osd.9 on smithi023 2024-03-28T05:19:17.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:16 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T05:19:17.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:16 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:17.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:16 smithi113 ceph-mon[37164]: Deploying daemon osd.9 on smithi023 2024-03-28T05:19:17.869 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:17 smithi023 ceph-mon[31954]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T05:19:18.015 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:17 smithi098 ceph-mon[36087]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T05:19:18.143 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:17 smithi113 ceph-mon[37164]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T05:19:20.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:19 smithi023 ceph-mon[31954]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:20.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:19 smithi098 ceph-mon[36087]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:20.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:19 smithi113 ceph-mon[37164]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:22.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:21 smithi023 ceph-mon[31954]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:22.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:21 smithi098 ceph-mon[36087]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:22.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:21 smithi113 ceph-mon[37164]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:23.234 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:23.235 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:23.235 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:19:23.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:22 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:23.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:22 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:23.241 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:22 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:19:23.364 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:22 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:23.364 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:22 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:23.364 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:22 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:19:24.000 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:23 smithi023 ceph-mon[31954]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:24.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:23 smithi113 ceph-mon[37164]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:24.364 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:23 smithi098 ceph-mon[36087]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:26.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:26.014+0000 7ffb877fe700 1 -- 172.21.15.23:0/3466453034 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ffb900625d0 con 0x7ffb7c0a2890 2024-03-28T05:19:26.013 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 9 on host 'smithi023' 2024-03-28T05:19:26.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:26.017+0000 7ffb985c1700 1 -- 172.21.15.23:0/3466453034 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ffb7c0a2890 msgr2=0x7ffb7c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:19:26.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:26.017+0000 7ffb985c1700 1 --2- 172.21.15.23:0/3466453034 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ffb7c0a2890 0x7ffb7c0a4d50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c00b580 tx=0x7ffb8c005fb0 comp rx=0 tx=0).stop 2024-03-28T05:19:26.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:26.017+0000 7ffb985c1700 1 -- 172.21.15.23:0/3466453034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb901033c0 msgr2=0x7ffb90198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:19:26.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:26.017+0000 7ffb985c1700 1 --2- 172.21.15.23:0/3466453034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb901033c0 0x7ffb90198930 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ffb8000d980 tx=0x7ffb8000dd40 comp rx=0 tx=0).stop 2024-03-28T05:19:26.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:26.017+0000 7ffb985c1700 1 -- 172.21.15.23:0/3466453034 shutdown_connections 2024-03-28T05:19:26.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:26.017+0000 7ffb985c1700 1 --2- 172.21.15.23:0/3466453034 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ffb7c0a2890 0x7ffb7c0a4d50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:26.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:26.017+0000 7ffb985c1700 1 --2- 172.21.15.23:0/3466453034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffb90103d60 0x7ffb9019f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:26.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:26.017+0000 7ffb985c1700 1 --2- 172.21.15.23:0/3466453034 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb901021c0 0x7ffb901983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:26.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:26.017+0000 7ffb985c1700 1 --2- 172.21.15.23:0/3466453034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb901033c0 0x7ffb90198930 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:26.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:26.017+0000 7ffb985c1700 1 -- 172.21.15.23:0/3466453034 >> 172.21.15.23:0/3466453034 conn(0x7ffb900fd7a0 msgr2=0x7ffb90106740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:19:26.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:26.017+0000 7ffb985c1700 1 -- 172.21.15.23:0/3466453034 shutdown_connections 2024-03-28T05:19:26.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:19:26.017+0000 7ffb985c1700 1 -- 172.21.15.23:0/3466453034 wait complete. 2024-03-28T05:19:26.180 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:26 smithi023 ceph-mon[31954]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:26.180 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:26 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:26.180 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:26 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:26.180 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:26 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:26.180 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:26 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:19:26.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:26 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:26.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:26 smithi023 ceph-mon[31954]: from='osd.9 [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T05:19:26.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:26 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:26.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:26 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:19:26.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:26 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:26.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:26 smithi113 ceph-mon[37164]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:26.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:26 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:26.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:26 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:26.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:26 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:26.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:26 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:19:26.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:26 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:26.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:26 smithi113 ceph-mon[37164]: from='osd.9 [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T05:19:26.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:26 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:26.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:26 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:19:26.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:26 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:26.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:26 smithi098 ceph-mon[36087]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:26.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:26 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:26.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:26 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:26.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:26 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:26.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:26 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:19:26.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:26 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:26.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:26 smithi098 ceph-mon[36087]: from='osd.9 [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T05:19:26.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:26 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:26.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:26 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:19:26.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:26 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:26.647 DEBUG:teuthology.orchestra.run.smithi023:osd.9> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.9.service 2024-03-28T05:19:26.650 INFO:tasks.cephadm:Deploying osd.10 on smithi098 with /dev/vg_nvme/lv_1... 2024-03-28T05:19:26.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T05:19:26.898 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T05:19:27.278 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:27 smithi098 ceph-mon[36087]: from='osd.9 [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T05:19:27.278 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:27 smithi098 ceph-mon[36087]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T05:19:27.278 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:27 smithi098 ceph-mon[36087]: from='osd.9 [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T05:19:27.278 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:27 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T05:19:27.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:27 smithi113 ceph-mon[37164]: from='osd.9 [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T05:19:27.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:27 smithi113 ceph-mon[37164]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T05:19:27.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:27 smithi113 ceph-mon[37164]: from='osd.9 [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T05:19:27.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:27 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T05:19:27.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:27 smithi023 ceph-mon[31954]: from='osd.9 [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T05:19:27.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:27 smithi023 ceph-mon[31954]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T05:19:27.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:27 smithi023 ceph-mon[31954]: from='osd.9 [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T05:19:27.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:27 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T05:19:27.531 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:19:27 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:19:27.205+0000 7f7252726700 -1 osd.9 0 waiting for initial osdmap 2024-03-28T05:19:27.531 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:19:27 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:19:27.219+0000 7f724e519700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T05:19:28.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:28 smithi023 ceph-mon[31954]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:28.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:28 smithi023 ceph-mon[31954]: from='osd.9 [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T05:19:28.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:28 smithi023 ceph-mon[31954]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T05:19:28.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:28 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T05:19:28.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:28 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T05:19:28.360 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:28 smithi098 ceph-mon[36087]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:28.360 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:28 smithi098 ceph-mon[36087]: from='osd.9 [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T05:19:28.360 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:28 smithi098 ceph-mon[36087]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T05:19:28.360 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:28 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T05:19:28.360 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:28 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T05:19:28.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:28 smithi113 ceph-mon[37164]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:28.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:28 smithi113 ceph-mon[37164]: from='osd.9 [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T05:19:28.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:28 smithi113 ceph-mon[37164]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T05:19:28.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:28 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T05:19:28.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:28 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T05:19:28.928 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T05:19:28.928 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T05:19:28.928 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-28T05:19:28.928 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-28T05:19:28.929 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125479 s, 836 MB/s 2024-03-28T05:19:28.929 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-28T05:19:29.443 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:29 smithi023 ceph-mon[31954]: purged_snaps scrub starts 2024-03-28T05:19:29.444 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:29 smithi023 ceph-mon[31954]: purged_snaps scrub ok 2024-03-28T05:19:29.444 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:29 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T05:19:29.444 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:29 smithi023 ceph-mon[31954]: osd.9 [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207] boot 2024-03-28T05:19:29.444 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:29 smithi023 ceph-mon[31954]: osdmap e67: 10 total, 10 up, 10 in 2024-03-28T05:19:29.444 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:29 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T05:19:29.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:29 smithi113 ceph-mon[37164]: purged_snaps scrub starts 2024-03-28T05:19:29.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:29 smithi113 ceph-mon[37164]: purged_snaps scrub ok 2024-03-28T05:19:29.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:29 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T05:19:29.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:29 smithi113 ceph-mon[37164]: osd.9 [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207] boot 2024-03-28T05:19:29.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:29 smithi113 ceph-mon[37164]: osdmap e67: 10 total, 10 up, 10 in 2024-03-28T05:19:29.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:29 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T05:19:29.573 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:29 smithi098 ceph-mon[36087]: purged_snaps scrub starts 2024-03-28T05:19:29.573 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:29 smithi098 ceph-mon[36087]: purged_snaps scrub ok 2024-03-28T05:19:29.573 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:29 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T05:19:29.573 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:29 smithi098 ceph-mon[36087]: osd.9 [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207] boot 2024-03-28T05:19:29.573 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:29 smithi098 ceph-mon[36087]: osdmap e67: 10 total, 10 up, 10 in 2024-03-28T05:19:29.573 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:29 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T05:19:29.610 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_1 2024-03-28T05:19:29.850 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T05:19:30.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:30 smithi113 ceph-mon[37164]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:30.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:30 smithi113 ceph-mon[37164]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T05:19:30.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:30 smithi113 ceph-mon[37164]: Detected new or changed devices on smithi023 2024-03-28T05:19:30.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:30 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:30.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:30 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:30.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:30 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:30.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:30 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:30.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:30 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:30.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:30 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:30.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:30 smithi113 ceph-mon[37164]: Adjusting osd_memory_target on smithi023 to 4272M 2024-03-28T05:19:30.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:30 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:30.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:30 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:30.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:30 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:19:30.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:30 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:30.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:30 smithi023 ceph-mon[31954]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:30.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:30 smithi023 ceph-mon[31954]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T05:19:30.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:30 smithi023 ceph-mon[31954]: Detected new or changed devices on smithi023 2024-03-28T05:19:30.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:30 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:30.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:30 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:30.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:30 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:30.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:30 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:30.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:30 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:30.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:30 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:30.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:30 smithi023 ceph-mon[31954]: Adjusting osd_memory_target on smithi023 to 4272M 2024-03-28T05:19:30.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:30 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:30.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:30 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:30.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:30 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:19:30.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:30 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:30.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:30 smithi098 ceph-mon[36087]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-28T05:19:30.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:30 smithi098 ceph-mon[36087]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T05:19:30.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:30 smithi098 ceph-mon[36087]: Detected new or changed devices on smithi023 2024-03-28T05:19:30.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:30 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:30.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:30 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:30.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:30 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:30.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:30 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:30.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:30 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:30.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:30 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:30.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:30 smithi098 ceph-mon[36087]: Adjusting osd_memory_target on smithi023 to 4272M 2024-03-28T05:19:30.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:30 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:30.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:30 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:30.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:30 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:19:30.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:30 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:31.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.547+0000 7f38aa7a9700 1 -- 172.21.15.98:0/1391224150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38a4100260 msgr2=0x7f38a41006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:19:31.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.547+0000 7f38aa7a9700 1 --2- 172.21.15.98:0/1391224150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38a4100260 0x7f38a41006c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3894009990 tx=0x7f3894009ca0 comp rx=0 tx=0).stop 2024-03-28T05:19:31.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.547+0000 7f38aa7a9700 1 -- 172.21.15.98:0/1391224150 shutdown_connections 2024-03-28T05:19:31.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.547+0000 7f38aa7a9700 1 --2- 172.21.15.98:0/1391224150 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38a4100c00 0x7f38a410a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:31.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.547+0000 7f38aa7a9700 1 --2- 172.21.15.98:0/1391224150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38a41037e0 0x7f38a4103c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:31.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.547+0000 7f38aa7a9700 1 --2- 172.21.15.98:0/1391224150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38a4100260 0x7f38a41006c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:31.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.547+0000 7f38aa7a9700 1 -- 172.21.15.98:0/1391224150 >> 172.21.15.98:0/1391224150 conn(0x7f38a40fbe20 msgr2=0x7f38a40fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:19:31.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.547+0000 7f38aa7a9700 1 -- 172.21.15.98:0/1391224150 shutdown_connections 2024-03-28T05:19:31.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.547+0000 7f38aa7a9700 1 -- 172.21.15.98:0/1391224150 wait complete. 2024-03-28T05:19:31.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.548+0000 7f38aa7a9700 1 Processor -- start 2024-03-28T05:19:31.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.548+0000 7f38aa7a9700 1 -- start start 2024-03-28T05:19:31.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.548+0000 7f38aa7a9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38a4100260 0x7f38a4198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:19:31.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.548+0000 7f38aa7a9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38a4100c00 0x7f38a4198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:19:31.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.548+0000 7f38aa7a9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38a41037e0 0x7f38a419fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:19:31.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.548+0000 7f38aa7a9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a410dee0 con 0x7f38a4100260 2024-03-28T05:19:31.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.548+0000 7f38aa7a9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a410dd60 con 0x7f38a41037e0 2024-03-28T05:19:31.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.548+0000 7f38aa7a9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a410e060 con 0x7f38a4100c00 2024-03-28T05:19:31.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.548+0000 7f38a8d46700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38a41037e0 0x7f38a419fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:19:31.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.548+0000 7f38a8d46700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38a41037e0 0x7f38a419fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44562/0 (socket says 172.21.15.98:44562) 2024-03-28T05:19:31.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.548+0000 7f38a8d46700 1 -- 172.21.15.98:0/2756063438 learned_addr learned my addr 172.21.15.98:0/2756063438 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-03-28T05:19:31.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.548+0000 7f38a37fe700 1 --2- 172.21.15.98:0/2756063438 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38a4100c00 0x7f38a4198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:19:31.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.548+0000 7f38a3fff700 1 --2- 172.21.15.98:0/2756063438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38a4100260 0x7f38a4198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:19:31.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.548+0000 7f38a8d46700 1 -- 172.21.15.98:0/2756063438 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38a4100c00 msgr2=0x7f38a4198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:19:31.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.549+0000 7f38a8d46700 1 --2- 172.21.15.98:0/2756063438 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38a4100c00 0x7f38a4198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:31.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.549+0000 7f38a8d46700 1 -- 172.21.15.98:0/2756063438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38a4100260 msgr2=0x7f38a4198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:19:31.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.549+0000 7f38a8d46700 1 --2- 172.21.15.98:0/2756063438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38a4100260 0x7f38a4198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:31.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.549+0000 7f38a8d46700 1 -- 172.21.15.98:0/2756063438 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a41a01b0 con 0x7f38a41037e0 2024-03-28T05:19:31.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.549+0000 7f38a37fe700 1 --2- 172.21.15.98:0/2756063438 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38a4100c00 0x7f38a4198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:19:31.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.549+0000 7f38a3fff700 1 --2- 172.21.15.98:0/2756063438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38a4100260 0x7f38a4198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:19:31.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.549+0000 7f38a8d46700 1 --2- 172.21.15.98:0/2756063438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38a41037e0 0x7f38a419fb10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f389c00b970 tx=0x7f389c00bd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:19:31.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.549+0000 7f38a17fa700 1 -- 172.21.15.98:0/2756063438 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389c019070 con 0x7f38a41037e0 2024-03-28T05:19:31.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.549+0000 7f38aa7a9700 1 -- 172.21.15.98:0/2756063438 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a41a04a0 con 0x7f38a41037e0 2024-03-28T05:19:31.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.549+0000 7f38a17fa700 1 -- 172.21.15.98:0/2756063438 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f389c014440 con 0x7f38a41037e0 2024-03-28T05:19:31.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.549+0000 7f38a17fa700 1 -- 172.21.15.98:0/2756063438 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389c0075d0 con 0x7f38a41037e0 2024-03-28T05:19:31.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.549+0000 7f38aa7a9700 1 -- 172.21.15.98:0/2756063438 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a41a0a60 con 0x7f38a41037e0 2024-03-28T05:19:31.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.550+0000 7f38aa7a9700 1 -- 172.21.15.98:0/2756063438 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a4068e50 con 0x7f38a41037e0 2024-03-28T05:19:31.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.551+0000 7f38a17fa700 1 -- 172.21.15.98:0/2756063438 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f389c01c070 con 0x7f38a41037e0 2024-03-28T05:19:31.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.552+0000 7f38a17fa700 1 --2- 172.21.15.98:0/2756063438 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f388c0a2890 0x7f388c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:19:31.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.552+0000 7f38a17fa700 1 -- 172.21.15.98:0/2756063438 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f389c0cd650 con 0x7f38a41037e0 2024-03-28T05:19:31.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.552+0000 7f38a3fff700 1 --2- 172.21.15.98:0/2756063438 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f388c0a2890 0x7f388c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:19:31.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.552+0000 7f38a3fff700 1 --2- 172.21.15.98:0/2756063438 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f388c0a2890 0x7f388c0a4d50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f38a406a5f0 tx=0x7f3898008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:19:31.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.556+0000 7f38a17fa700 1 -- 172.21.15.98:0/2756063438 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f389c09aef0 con 0x7f38a41037e0 2024-03-28T05:19:31.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:31.761+0000 7f38aa7a9700 1 -- 172.21.15.98:0/2756063438 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f38a40625d0 con 0x7f388c0a2890 2024-03-28T05:19:32.488 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:32 smithi098 ceph-mon[36087]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:32.488 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:32 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:19:32.488 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:32 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:19:32.488 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:32 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:32.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:32 smithi113 ceph-mon[37164]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:32.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:32 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:19:32.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:32 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:19:32.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:32 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:32.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:32 smithi023 ceph-mon[31954]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:32.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:32 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:19:32.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:32 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:19:32.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:32 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:33.364 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:33 smithi098 ceph-mon[36087]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:19:33.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:33 smithi023 ceph-mon[31954]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:19:33.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:33 smithi113 ceph-mon[37164]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:19:34.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:34 smithi023 ceph-mon[31954]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:34.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:34 smithi098 ceph-mon[36087]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:34.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:34 smithi113 ceph-mon[37164]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:35.364 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:35 smithi098 ceph-mon[36087]: from='client.? 172.21.15.98:0/1776473666' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0eda917f-020c-437f-95ae-b26203f6c68e"}]: dispatch 2024-03-28T05:19:35.364 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:35 smithi098 ceph-mon[36087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0eda917f-020c-437f-95ae-b26203f6c68e"}]: dispatch 2024-03-28T05:19:35.364 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:35 smithi098 ceph-mon[36087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0eda917f-020c-437f-95ae-b26203f6c68e"}]': finished 2024-03-28T05:19:35.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:35 smithi098 ceph-mon[36087]: osdmap e69: 11 total, 10 up, 11 in 2024-03-28T05:19:35.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:35 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T05:19:35.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:35 smithi023 ceph-mon[31954]: from='client.? 172.21.15.98:0/1776473666' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0eda917f-020c-437f-95ae-b26203f6c68e"}]: dispatch 2024-03-28T05:19:35.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:35 smithi023 ceph-mon[31954]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0eda917f-020c-437f-95ae-b26203f6c68e"}]: dispatch 2024-03-28T05:19:35.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:35 smithi023 ceph-mon[31954]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0eda917f-020c-437f-95ae-b26203f6c68e"}]': finished 2024-03-28T05:19:35.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:35 smithi023 ceph-mon[31954]: osdmap e69: 11 total, 10 up, 11 in 2024-03-28T05:19:35.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:35 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T05:19:35.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:35 smithi113 ceph-mon[37164]: from='client.? 172.21.15.98:0/1776473666' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0eda917f-020c-437f-95ae-b26203f6c68e"}]: dispatch 2024-03-28T05:19:35.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:35 smithi113 ceph-mon[37164]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0eda917f-020c-437f-95ae-b26203f6c68e"}]: dispatch 2024-03-28T05:19:35.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:35 smithi113 ceph-mon[37164]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0eda917f-020c-437f-95ae-b26203f6c68e"}]': finished 2024-03-28T05:19:35.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:35 smithi113 ceph-mon[37164]: osdmap e69: 11 total, 10 up, 11 in 2024-03-28T05:19:35.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:35 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T05:19:36.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:36 smithi023 ceph-mon[31954]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:36.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:36 smithi023 ceph-mon[31954]: from='client.? 172.21.15.98:0/3653146992' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:19:36.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:36 smithi098 ceph-mon[36087]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:36.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:36 smithi098 ceph-mon[36087]: from='client.? 172.21.15.98:0/3653146992' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:19:36.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:36 smithi113 ceph-mon[37164]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:36.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:36 smithi113 ceph-mon[37164]: from='client.? 172.21.15.98:0/3653146992' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:19:38.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:38 smithi023 ceph-mon[31954]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:38.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:38 smithi098 ceph-mon[36087]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:38.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:38 smithi113 ceph-mon[37164]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:40.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:40 smithi023 ceph-mon[31954]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:40.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:40 smithi098 ceph-mon[36087]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:40.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:40 smithi113 ceph-mon[37164]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:42.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:42 smithi023 ceph-mon[31954]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:42.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:42 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T05:19:42.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:42 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:42.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:42 smithi098 ceph-mon[36087]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:42.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:42 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T05:19:42.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:42 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:42.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:42 smithi113 ceph-mon[37164]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:42.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:42 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T05:19:42.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:42 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:43.554 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:43 smithi098 ceph-mon[36087]: Deploying daemon osd.10 on smithi098 2024-03-28T05:19:43.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:43 smithi113 ceph-mon[37164]: Deploying daemon osd.10 on smithi098 2024-03-28T05:19:43.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:43 smithi023 ceph-mon[31954]: Deploying daemon osd.10 on smithi098 2024-03-28T05:19:44.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:44 smithi098 ceph-mon[36087]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:44.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:44 smithi113 ceph-mon[37164]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:44.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:44 smithi023 ceph-mon[31954]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:46.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:46 smithi098 ceph-mon[36087]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:46.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:46 smithi113 ceph-mon[37164]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:46.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:46 smithi023 ceph-mon[31954]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:48.591 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:48 smithi098 ceph-mon[36087]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:48.591 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:48 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:48.591 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:48 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:48.591 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:48 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:19:48.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:48 smithi113 ceph-mon[37164]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:48.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:48 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:48.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:48 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:48.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:48 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:19:48.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:48 smithi023 ceph-mon[31954]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:48.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:48.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:48.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:19:50.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:50 smithi098 ceph-mon[36087]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:50.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:50 smithi023 ceph-mon[31954]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:50.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:50 smithi113 ceph-mon[37164]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:51.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:51.577+0000 7f38a17fa700 1 -- 172.21.15.98:0/2756063438 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f38a40625d0 con 0x7f388c0a2890 2024-03-28T05:19:51.576 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 10 on host 'smithi098' 2024-03-28T05:19:51.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:51.580+0000 7f38aa7a9700 1 -- 172.21.15.98:0/2756063438 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f388c0a2890 msgr2=0x7f388c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:19:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:51.580+0000 7f38aa7a9700 1 --2- 172.21.15.98:0/2756063438 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f388c0a2890 0x7f388c0a4d50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f38a406a5f0 tx=0x7f3898008040 comp rx=0 tx=0).stop 2024-03-28T05:19:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:51.580+0000 7f38aa7a9700 1 -- 172.21.15.98:0/2756063438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38a41037e0 msgr2=0x7f38a419fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:19:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:51.580+0000 7f38aa7a9700 1 --2- 172.21.15.98:0/2756063438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38a41037e0 0x7f38a419fb10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f389c00b970 tx=0x7f389c00bd30 comp rx=0 tx=0).stop 2024-03-28T05:19:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:51.580+0000 7f38aa7a9700 1 -- 172.21.15.98:0/2756063438 shutdown_connections 2024-03-28T05:19:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:51.580+0000 7f38aa7a9700 1 --2- 172.21.15.98:0/2756063438 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f388c0a2890 0x7f388c0a4d50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:51.580+0000 7f38aa7a9700 1 --2- 172.21.15.98:0/2756063438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38a41037e0 0x7f38a419fb10 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:51.580+0000 7f38aa7a9700 1 --2- 172.21.15.98:0/2756063438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38a4100260 0x7f38a4198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:51.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:51.580+0000 7f38aa7a9700 1 --2- 172.21.15.98:0/2756063438 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38a4100c00 0x7f38a4198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:51.580+0000 7f38aa7a9700 1 -- 172.21.15.98:0/2756063438 >> 172.21.15.98:0/2756063438 conn(0x7f38a40fbe20 msgr2=0x7f38a4106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:19:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:51.580+0000 7f38aa7a9700 1 -- 172.21.15.98:0/2756063438 shutdown_connections 2024-03-28T05:19:51.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-28T05:19:51.580+0000 7f38aa7a9700 1 -- 172.21.15.98:0/2756063438 wait complete. 2024-03-28T05:19:52.305 DEBUG:teuthology.orchestra.run.smithi098:osd.10> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.10.service 2024-03-28T05:19:52.307 INFO:tasks.cephadm:Deploying osd.11 on smithi113 with /dev/vg_nvme/lv_1... 2024-03-28T05:19:52.308 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T05:19:52.555 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T05:19:52.673 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:52 smithi113 ceph-mon[37164]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:52.673 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:52 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:52.673 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:52 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:52.673 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:52 smithi113 ceph-mon[37164]: from='osd.10 [v2:172.21.15.98:6824/2699920595,v1:172.21.15.98:6825/2699920595]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T05:19:52.673 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:52 smithi113 ceph-mon[37164]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T05:19:52.674 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:52 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:52.674 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:52 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:52.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:52 smithi098 ceph-mon[36087]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:52.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:52 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:52.864 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:52 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:52.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:52 smithi098 ceph-mon[36087]: from='osd.10 [v2:172.21.15.98:6824/2699920595,v1:172.21.15.98:6825/2699920595]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T05:19:52.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:52 smithi098 ceph-mon[36087]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T05:19:52.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:52 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:52.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:52 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:53.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:52 smithi023 ceph-mon[31954]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:53.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:52 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:53.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:52 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:53.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:52 smithi023 ceph-mon[31954]: from='osd.10 [v2:172.21.15.98:6824/2699920595,v1:172.21.15.98:6825/2699920595]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T05:19:53.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:52 smithi023 ceph-mon[31954]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T05:19:53.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:52 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:53.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:52 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:53.723 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:53 smithi113 ceph-mon[37164]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T05:19:53.723 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:53 smithi113 ceph-mon[37164]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T05:19:53.724 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:53 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T05:19:53.724 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:53 smithi113 ceph-mon[37164]: from='osd.10 [v2:172.21.15.98:6824/2699920595,v1:172.21.15.98:6825/2699920595]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:19:53.724 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:53 smithi113 ceph-mon[37164]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:19:53.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:53 smithi098 ceph-mon[36087]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T05:19:53.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:53 smithi098 ceph-mon[36087]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T05:19:53.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:53 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T05:19:53.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:53 smithi098 ceph-mon[36087]: from='osd.10 [v2:172.21.15.98:6824/2699920595,v1:172.21.15.98:6825/2699920595]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:19:53.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:53 smithi098 ceph-mon[36087]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:19:53.865 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:19:53 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:19:53.553+0000 7fa8fc5f2700 -1 osd.10 0 waiting for initial osdmap 2024-03-28T05:19:53.866 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:19:53 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:19:53.571+0000 7fa8f83e5700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T05:19:54.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:53 smithi023 ceph-mon[31954]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T05:19:54.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:53 smithi023 ceph-mon[31954]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T05:19:54.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:53 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T05:19:54.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:53 smithi023 ceph-mon[31954]: from='osd.10 [v2:172.21.15.98:6824/2699920595,v1:172.21.15.98:6825/2699920595]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:19:54.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:53 smithi023 ceph-mon[31954]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-28T05:19:54.593 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T05:19:54.594 INFO:teuthology.orchestra.run.smithi113.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T05:19:54.594 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10+0 records in 2024-03-28T05:19:54.594 INFO:teuthology.orchestra.run.smithi113.stderr:10+0 records out 2024-03-28T05:19:54.594 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00930094 s, 1.1 GB/s 2024-03-28T05:19:54.594 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping successful for: 2024-03-28T05:19:54.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:54 smithi098 ceph-mon[36087]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:54.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:54 smithi098 ceph-mon[36087]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-28T05:19:54.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:54 smithi098 ceph-mon[36087]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T05:19:54.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:54 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T05:19:54.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:54 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T05:19:54.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:54 smithi098 ceph-mon[36087]: osd.10 [v2:172.21.15.98:6824/2699920595,v1:172.21.15.98:6825/2699920595] boot 2024-03-28T05:19:54.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:54 smithi098 ceph-mon[36087]: osdmap e72: 11 total, 11 up, 11 in 2024-03-28T05:19:54.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:54 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T05:19:54.772 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:54 smithi113 ceph-mon[37164]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:54.772 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:54 smithi113 ceph-mon[37164]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-28T05:19:54.772 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:54 smithi113 ceph-mon[37164]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T05:19:54.772 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:54 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T05:19:54.772 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:54 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T05:19:54.772 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:54 smithi113 ceph-mon[37164]: osd.10 [v2:172.21.15.98:6824/2699920595,v1:172.21.15.98:6825/2699920595] boot 2024-03-28T05:19:54.772 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:54 smithi113 ceph-mon[37164]: osdmap e72: 11 total, 11 up, 11 in 2024-03-28T05:19:54.773 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:54 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T05:19:55.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:54 smithi023 ceph-mon[31954]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-28T05:19:55.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:54 smithi023 ceph-mon[31954]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-28T05:19:55.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:54 smithi023 ceph-mon[31954]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T05:19:55.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:54 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T05:19:55.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:54 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T05:19:55.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:54 smithi023 ceph-mon[31954]: osd.10 [v2:172.21.15.98:6824/2699920595,v1:172.21.15.98:6825/2699920595] boot 2024-03-28T05:19:55.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:54 smithi023 ceph-mon[31954]: osdmap e72: 11 total, 11 up, 11 in 2024-03-28T05:19:55.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:54 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T05:19:55.349 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi113:vg_nvme/lv_1 2024-03-28T05:19:55.591 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T05:19:55.829 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:55 smithi098 ceph-mon[36087]: purged_snaps scrub starts 2024-03-28T05:19:55.830 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:55 smithi098 ceph-mon[36087]: purged_snaps scrub ok 2024-03-28T05:19:55.830 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:55 smithi098 ceph-mon[36087]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T05:19:55.878 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:55 smithi113 ceph-mon[37164]: purged_snaps scrub starts 2024-03-28T05:19:55.878 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:55 smithi113 ceph-mon[37164]: purged_snaps scrub ok 2024-03-28T05:19:55.878 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:55 smithi113 ceph-mon[37164]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T05:19:56.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:55 smithi023 ceph-mon[31954]: purged_snaps scrub starts 2024-03-28T05:19:56.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:55 smithi023 ceph-mon[31954]: purged_snaps scrub ok 2024-03-28T05:19:56.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:55 smithi023 ceph-mon[31954]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T05:19:56.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:56 smithi113 ceph-mon[37164]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:19:56.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:56 smithi113 ceph-mon[37164]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T05:19:56.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:56.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:56.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:56.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:56.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:56.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:56.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:56.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:56.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:19:56.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:56.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:19:56.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:56.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:19:56.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:57.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:56 smithi023 ceph-mon[31954]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:19:57.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:56 smithi023 ceph-mon[31954]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T05:19:57.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:19:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:19:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:57.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:19:57.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:57.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:56 smithi098 ceph-mon[36087]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:19:57.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:56 smithi098 ceph-mon[36087]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T05:19:57.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:57.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:57.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:57.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:57.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:57.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:19:57.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:57.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:57.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:19:57.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:57.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:19:57.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:57.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:19:57.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:19:57.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.254+0000 7fb7d3a30700 1 -- 172.21.15.113:0/3592202037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7cc105650 msgr2=0x7fb7cc107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:19:57.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.254+0000 7fb7d3a30700 1 --2- 172.21.15.113:0/3592202037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7cc105650 0x7fb7cc107a40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8009990 tx=0x7fb7c8009ca0 comp rx=0 tx=0).stop 2024-03-28T05:19:57.255 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.254+0000 7fb7d3a30700 1 -- 172.21.15.113:0/3592202037 shutdown_connections 2024-03-28T05:19:57.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.254+0000 7fb7d3a30700 1 --2- 172.21.15.113:0/3592202037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7cc107f80 0x7fb7cc10a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:57.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.254+0000 7fb7d3a30700 1 --2- 172.21.15.113:0/3592202037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7cc102d20 0x7fb7cc105110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:57.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.254+0000 7fb7d3a30700 1 --2- 172.21.15.113:0/3592202037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7cc105650 0x7fb7cc107a40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:57.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.254+0000 7fb7d3a30700 1 -- 172.21.15.113:0/3592202037 >> 172.21.15.113:0/3592202037 conn(0x7fb7cc0fc6f0 msgr2=0x7fb7cc0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:19:57.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.254+0000 7fb7d3a30700 1 -- 172.21.15.113:0/3592202037 shutdown_connections 2024-03-28T05:19:57.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.254+0000 7fb7d3a30700 1 -- 172.21.15.113:0/3592202037 wait complete. 2024-03-28T05:19:57.256 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.255+0000 7fb7d3a30700 1 Processor -- start 2024-03-28T05:19:57.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.255+0000 7fb7d3a30700 1 -- start start 2024-03-28T05:19:57.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.255+0000 7fb7d3a30700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7cc102d20 0x7fb7cc198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:19:57.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.255+0000 7fb7d3a30700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7cc105650 0x7fb7cc198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:19:57.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.255+0000 7fb7d3a30700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7cc107f80 0x7fb7cc19f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:19:57.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.255+0000 7fb7d3a30700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc10de10 con 0x7fb7cc107f80 2024-03-28T05:19:57.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.255+0000 7fb7d3a30700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc10dc90 con 0x7fb7cc105650 2024-03-28T05:19:57.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.255+0000 7fb7d3a30700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc10df90 con 0x7fb7cc102d20 2024-03-28T05:19:57.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.255+0000 7fb7d17cc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7cc102d20 0x7fb7cc198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:19:57.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.255+0000 7fb7d17cc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7cc102d20 0x7fb7cc198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41796/0 (socket says 172.21.15.113:41796) 2024-03-28T05:19:57.257 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.255+0000 7fb7d17cc700 1 -- 172.21.15.113:0/2245900740 learned_addr learned my addr 172.21.15.113:0/2245900740 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-03-28T05:19:57.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.255+0000 7fb7d17cc700 1 -- 172.21.15.113:0/2245900740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7cc105650 msgr2=0x7fb7cc198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:19:57.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.255+0000 7fb7d17cc700 1 --2- 172.21.15.113:0/2245900740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7cc105650 0x7fb7cc198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:57.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.255+0000 7fb7d17cc700 1 -- 172.21.15.113:0/2245900740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7cc107f80 msgr2=0x7fb7cc19f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:19:57.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.255+0000 7fb7d17cc700 1 --2- 172.21.15.113:0/2245900740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7cc107f80 0x7fb7cc19f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:19:57.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.255+0000 7fb7d17cc700 1 -- 172.21.15.113:0/2245900740 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7cc1a0070 con 0x7fb7cc102d20 2024-03-28T05:19:57.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.256+0000 7fb7d17cc700 1 --2- 172.21.15.113:0/2245900740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7cc102d20 0x7fb7cc198410 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00d980 tx=0x7fb7bc00dd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:19:57.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.256+0000 7fb7c27fc700 1 -- 172.21.15.113:0/2245900740 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc016070 con 0x7fb7cc102d20 2024-03-28T05:19:57.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.256+0000 7fb7c27fc700 1 -- 172.21.15.113:0/2245900740 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7bc009b40 con 0x7fb7cc102d20 2024-03-28T05:19:57.258 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.256+0000 7fb7d3a30700 1 -- 172.21.15.113:0/2245900740 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7cc1a0360 con 0x7fb7cc102d20 2024-03-28T05:19:57.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.256+0000 7fb7c27fc700 1 -- 172.21.15.113:0/2245900740 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc00f620 con 0x7fb7cc102d20 2024-03-28T05:19:57.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.256+0000 7fb7d3a30700 1 -- 172.21.15.113:0/2245900740 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7cc1a0920 con 0x7fb7cc102d20 2024-03-28T05:19:57.259 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.257+0000 7fb7d3a30700 1 -- 172.21.15.113:0/2245900740 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7cc068e50 con 0x7fb7cc102d20 2024-03-28T05:19:57.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.258+0000 7fb7c27fc700 1 -- 172.21.15.113:0/2245900740 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb7bc019070 con 0x7fb7cc102d20 2024-03-28T05:19:57.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.258+0000 7fb7c27fc700 1 --2- 172.21.15.113:0/2245900740 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb7b80a2890 0x7fb7b80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:19:57.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.258+0000 7fb7c27fc700 1 -- 172.21.15.113:0/2245900740 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6694+0+0 (secure 0 0 0) 0x7fb7bc0ce000 con 0x7fb7cc102d20 2024-03-28T05:19:57.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.259+0000 7fb7d0fcb700 1 --2- 172.21.15.113:0/2245900740 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb7b80a2890 0x7fb7b80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:19:57.260 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.259+0000 7fb7d0fcb700 1 --2- 172.21.15.113:0/2245900740 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb7b80a2890 0x7fb7b80a4d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8005f50 tx=0x7fb7c8005e80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:19:57.261 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.263+0000 7fb7c27fc700 1 -- 172.21.15.113:0/2245900740 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb7bc09b6f0 con 0x7fb7cc102d20 2024-03-28T05:19:57.464 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:19:57.466+0000 7fb7d3a30700 1 -- 172.21.15.113:0/2245900740 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7cc0625d0 con 0x7fb7b80a2890 2024-03-28T05:19:57.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:57 smithi113 ceph-mon[37164]: Detected new or changed devices on smithi098 2024-03-28T05:19:57.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:57 smithi113 ceph-mon[37164]: Adjusting osd_memory_target on smithi098 to 4270M 2024-03-28T05:19:57.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:57 smithi113 ceph-mon[37164]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T05:19:57.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:57 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:19:57.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:57 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:19:57.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:57 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:58.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:57 smithi023 ceph-mon[31954]: Detected new or changed devices on smithi098 2024-03-28T05:19:58.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:57 smithi023 ceph-mon[31954]: Adjusting osd_memory_target on smithi098 to 4270M 2024-03-28T05:19:58.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:57 smithi023 ceph-mon[31954]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T05:19:58.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:57 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:19:58.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:57 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:19:58.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:57 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:58.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:57 smithi098 ceph-mon[36087]: Detected new or changed devices on smithi098 2024-03-28T05:19:58.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:57 smithi098 ceph-mon[36087]: Adjusting osd_memory_target on smithi098 to 4270M 2024-03-28T05:19:58.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:57 smithi098 ceph-mon[36087]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T05:19:58.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:57 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T05:19:58.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:57 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T05:19:58.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:57 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:19:58.650 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:58 smithi113 ceph-mon[37164]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:19:58.650 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:58 smithi113 ceph-mon[37164]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:19:58.650 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:19:58 smithi113 ceph-mon[37164]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T05:19:59.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:58 smithi023 ceph-mon[31954]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:19:59.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:58 smithi023 ceph-mon[31954]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:19:59.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:19:58 smithi023 ceph-mon[31954]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T05:19:59.040 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:58 smithi098 ceph-mon[36087]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:19:59.040 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:58 smithi098 ceph-mon[36087]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:19:59.040 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:19:58 smithi098 ceph-mon[36087]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T05:20:00.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:00 smithi113 ceph-mon[37164]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-03-28T05:20:00.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:00 smithi113 ceph-mon[37164]: overall HEALTH_OK 2024-03-28T05:20:00.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:00 smithi113 ceph-mon[37164]: from='client.? 172.21.15.113:0/1758102617' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aae78501-78a8-469a-be20-3006f5ec5037"}]: dispatch 2024-03-28T05:20:00.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:00 smithi113 ceph-mon[37164]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aae78501-78a8-469a-be20-3006f5ec5037"}]: dispatch 2024-03-28T05:20:00.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:00 smithi113 ceph-mon[37164]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aae78501-78a8-469a-be20-3006f5ec5037"}]': finished 2024-03-28T05:20:00.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:00 smithi113 ceph-mon[37164]: osdmap e77: 12 total, 11 up, 12 in 2024-03-28T05:20:00.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:00 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T05:20:01.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:00 smithi023 ceph-mon[31954]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-03-28T05:20:01.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:00 smithi023 ceph-mon[31954]: overall HEALTH_OK 2024-03-28T05:20:01.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:00 smithi023 ceph-mon[31954]: from='client.? 172.21.15.113:0/1758102617' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aae78501-78a8-469a-be20-3006f5ec5037"}]: dispatch 2024-03-28T05:20:01.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:00 smithi023 ceph-mon[31954]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aae78501-78a8-469a-be20-3006f5ec5037"}]: dispatch 2024-03-28T05:20:01.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:00 smithi023 ceph-mon[31954]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aae78501-78a8-469a-be20-3006f5ec5037"}]': finished 2024-03-28T05:20:01.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:00 smithi023 ceph-mon[31954]: osdmap e77: 12 total, 11 up, 12 in 2024-03-28T05:20:01.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:00 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T05:20:01.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:00 smithi098 ceph-mon[36087]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-03-28T05:20:01.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:00 smithi098 ceph-mon[36087]: overall HEALTH_OK 2024-03-28T05:20:01.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:00 smithi098 ceph-mon[36087]: from='client.? 172.21.15.113:0/1758102617' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aae78501-78a8-469a-be20-3006f5ec5037"}]: dispatch 2024-03-28T05:20:01.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:00 smithi098 ceph-mon[36087]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aae78501-78a8-469a-be20-3006f5ec5037"}]: dispatch 2024-03-28T05:20:01.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:00 smithi098 ceph-mon[36087]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aae78501-78a8-469a-be20-3006f5ec5037"}]': finished 2024-03-28T05:20:01.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:00 smithi098 ceph-mon[36087]: osdmap e77: 12 total, 11 up, 12 in 2024-03-28T05:20:01.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:00 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T05:20:01.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:01 smithi113 ceph-mon[37164]: from='client.? 172.21.15.113:0/1396686643' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:20:02.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:01 smithi023 ceph-mon[31954]: from='client.? 172.21.15.113:0/1396686643' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:20:02.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:01 smithi098 ceph-mon[36087]: from='client.? 172.21.15.113:0/1396686643' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T05:20:02.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:02 smithi113 ceph-mon[37164]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-28T05:20:03.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:02 smithi023 ceph-mon[31954]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-28T05:20:03.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:02 smithi098 ceph-mon[36087]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-28T05:20:04.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:04 smithi113 ceph-mon[37164]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-28T05:20:05.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:04 smithi023 ceph-mon[31954]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-28T05:20:05.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:04 smithi098 ceph-mon[36087]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-28T05:20:06.805 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:06 smithi113 ceph-mon[37164]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T05:20:07.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:06 smithi023 ceph-mon[31954]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T05:20:07.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:06 smithi098 ceph-mon[36087]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T05:20:08.816 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:08 smithi113 ceph-mon[37164]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:20:08.817 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:08 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T05:20:08.817 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:08 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:09.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:08 smithi023 ceph-mon[31954]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:20:09.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:08 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T05:20:09.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:08 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:09.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:08 smithi098 ceph-mon[36087]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:20:09.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T05:20:09.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:09.810 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:09 smithi113 ceph-mon[37164]: Deploying daemon osd.11 on smithi113 2024-03-28T05:20:10.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:09 smithi023 ceph-mon[31954]: Deploying daemon osd.11 on smithi113 2024-03-28T05:20:10.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:09 smithi098 ceph-mon[36087]: Deploying daemon osd.11 on smithi113 2024-03-28T05:20:10.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:10 smithi113 ceph-mon[37164]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:20:11.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:10 smithi023 ceph-mon[31954]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:20:11.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:10 smithi098 ceph-mon[36087]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:20:12.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:12 smithi113 ceph-mon[37164]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:20:13.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:12 smithi023 ceph-mon[31954]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:20:13.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:12 smithi098 ceph-mon[36087]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:20:14.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:14 smithi113 ceph-mon[37164]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:20:14.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:14 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:14.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:14 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:14.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:14 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:20:15.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:14 smithi023 ceph-mon[31954]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:20:15.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:14 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:15.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:14 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:15.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:14 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:20:15.114 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:14 smithi098 ceph-mon[36087]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:20:15.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:14 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:15.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:14 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:15.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:14 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:20:16.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:16 smithi113 ceph-mon[37164]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:20:16.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:16 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:16.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:16 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:16.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:16 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:16.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:16 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:20:16.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:16 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:17.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:16 smithi023 ceph-mon[31954]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:20:17.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:16 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:17.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:16 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:17.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:16 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:17.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:16 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:20:17.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:16 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:17.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:16 smithi098 ceph-mon[36087]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:20:17.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:16 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:17.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:16 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:17.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:16 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:17.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:16 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:20:17.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:16 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:17.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:17.691+0000 7fb7c27fc700 1 -- 172.21.15.113:0/2245900740 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fb7cc0625d0 con 0x7fb7b80a2890 2024-03-28T05:20:17.689 INFO:teuthology.orchestra.run.smithi113.stdout:Created osd(s) 11 on host 'smithi113' 2024-03-28T05:20:17.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:17.693+0000 7fb7d3a30700 1 -- 172.21.15.113:0/2245900740 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb7b80a2890 msgr2=0x7fb7b80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:17.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:17.693+0000 7fb7d3a30700 1 --2- 172.21.15.113:0/2245900740 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb7b80a2890 0x7fb7b80a4d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8005f50 tx=0x7fb7c8005e80 comp rx=0 tx=0).stop 2024-03-28T05:20:17.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:17.693+0000 7fb7d3a30700 1 -- 172.21.15.113:0/2245900740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7cc102d20 msgr2=0x7fb7cc198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:17.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:17.693+0000 7fb7d3a30700 1 --2- 172.21.15.113:0/2245900740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7cc102d20 0x7fb7cc198410 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00d980 tx=0x7fb7bc00dd40 comp rx=0 tx=0).stop 2024-03-28T05:20:17.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:17.693+0000 7fb7d3a30700 1 -- 172.21.15.113:0/2245900740 shutdown_connections 2024-03-28T05:20:17.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:17.693+0000 7fb7d3a30700 1 --2- 172.21.15.113:0/2245900740 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb7b80a2890 0x7fb7b80a4d50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:17.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:17.693+0000 7fb7d3a30700 1 --2- 172.21.15.113:0/2245900740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7cc107f80 0x7fb7cc19f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:17.691 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:17.693+0000 7fb7d3a30700 1 --2- 172.21.15.113:0/2245900740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7cc102d20 0x7fb7cc198410 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:17.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:17.693+0000 7fb7d3a30700 1 --2- 172.21.15.113:0/2245900740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7cc105650 0x7fb7cc198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:17.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:17.693+0000 7fb7d3a30700 1 -- 172.21.15.113:0/2245900740 >> 172.21.15.113:0/2245900740 conn(0x7fb7cc0fc6f0 msgr2=0x7fb7cc103680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:17.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:17.693+0000 7fb7d3a30700 1 -- 172.21.15.113:0/2245900740 shutdown_connections 2024-03-28T05:20:17.692 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:17.693+0000 7fb7d3a30700 1 -- 172.21.15.113:0/2245900740 wait complete. 2024-03-28T05:20:17.991 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:17 smithi113 ceph-mon[37164]: from='osd.11 [v2:172.21.15.113:6824/2113644020,v1:172.21.15.113:6825/2113644020]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T05:20:17.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:17 smithi113 ceph-mon[37164]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T05:20:17.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:17 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:17.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:17 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:17.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:17 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:20:18.016 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:17 smithi098 ceph-mon[36087]: from='osd.11 [v2:172.21.15.113:6824/2113644020,v1:172.21.15.113:6825/2113644020]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T05:20:18.016 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:17 smithi098 ceph-mon[36087]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T05:20:18.016 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:17 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:18.016 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:17 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:18.016 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:17 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:20:18.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:17 smithi023 ceph-mon[31954]: from='osd.11 [v2:172.21.15.113:6824/2113644020,v1:172.21.15.113:6825/2113644020]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T05:20:18.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:17 smithi023 ceph-mon[31954]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T05:20:18.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:17 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:18.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:17 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:18.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:17 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:20:18.426 DEBUG:teuthology.orchestra.run.smithi113:osd.11> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.11.service 2024-03-28T05:20:18.428 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-28T05:20:18.429 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-28T05:20:18.682 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:20:18.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:18 smithi023 ceph-mon[31954]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:20:18.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:18 smithi023 ceph-mon[31954]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T05:20:18.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:18 smithi023 ceph-mon[31954]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T05:20:18.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:18 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T05:20:18.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:18 smithi023 ceph-mon[31954]: from='osd.11 [v2:172.21.15.113:6824/2113644020,v1:172.21.15.113:6825/2113644020]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:20:18.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:18 smithi023 ceph-mon[31954]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:20:18.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:18 smithi113 ceph-mon[37164]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:20:18.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:18 smithi113 ceph-mon[37164]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T05:20:18.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:18 smithi113 ceph-mon[37164]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T05:20:18.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:18 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T05:20:18.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:18 smithi113 ceph-mon[37164]: from='osd.11 [v2:172.21.15.113:6824/2113644020,v1:172.21.15.113:6825/2113644020]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:20:18.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:18 smithi113 ceph-mon[37164]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:20:19.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:18 smithi098 ceph-mon[36087]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:20:19.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:18 smithi098 ceph-mon[36087]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T05:20:19.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:18 smithi098 ceph-mon[36087]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T05:20:19.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:18 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T05:20:19.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:18 smithi098 ceph-mon[36087]: from='osd.11 [v2:172.21.15.113:6824/2113644020,v1:172.21.15.113:6825/2113644020]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:20:19.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:18 smithi098 ceph-mon[36087]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-28T05:20:19.492 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:20:19 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:20:19.218+0000 7fc01610b700 -1 osd.11 0 waiting for initial osdmap 2024-03-28T05:20:19.492 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:20:19 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:20:19.231+0000 7fc00def6700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T05:20:19.822 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:19 smithi023 ceph-mon[31954]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-28T05:20:19.822 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:19 smithi023 ceph-mon[31954]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T05:20:19.822 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:19 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T05:20:19.822 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:19 smithi023 ceph-mon[31954]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:20:19.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:19 smithi023 ceph-mon[31954]: from='osd.11 ' entity='osd.11' 2024-03-28T05:20:19.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:19 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T05:20:19.882 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:19 smithi113 ceph-mon[37164]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-28T05:20:19.882 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:19 smithi113 ceph-mon[37164]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T05:20:19.882 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:19 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T05:20:19.883 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:19 smithi113 ceph-mon[37164]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:20:19.883 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:19 smithi113 ceph-mon[37164]: from='osd.11 ' entity='osd.11' 2024-03-28T05:20:19.883 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:19 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T05:20:20.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:19 smithi098 ceph-mon[36087]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-28T05:20:20.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:19 smithi098 ceph-mon[36087]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T05:20:20.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:19 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T05:20:20.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:19 smithi098 ceph-mon[36087]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-03-28T05:20:20.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:19 smithi098 ceph-mon[36087]: from='osd.11 ' entity='osd.11' 2024-03-28T05:20:20.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:19 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T05:20:20.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.391+0000 7fd687031700 1 -- 172.21.15.23:0/1983828935 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6801033c0 msgr2=0x7fd680103820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:20.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.391+0000 7fd687031700 1 --2- 172.21.15.23:0/1983828935 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6801033c0 0x7fd680103820 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd66c0099c0 tx=0x7fd66c009cd0 comp rx=0 tx=0).stop 2024-03-28T05:20:20.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.391+0000 7fd687031700 1 -- 172.21.15.23:0/1983828935 shutdown_connections 2024-03-28T05:20:20.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.391+0000 7fd687031700 1 --2- 172.21.15.23:0/1983828935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd680103d60 0x7fd68010a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:20.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.391+0000 7fd687031700 1 --2- 172.21.15.23:0/1983828935 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6801021c0 0x7fd6801025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:20.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.391+0000 7fd687031700 1 --2- 172.21.15.23:0/1983828935 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6801033c0 0x7fd680103820 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:20.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.391+0000 7fd687031700 1 -- 172.21.15.23:0/1983828935 >> 172.21.15.23:0/1983828935 conn(0x7fd6800fd7a0 msgr2=0x7fd6800ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:20.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.391+0000 7fd687031700 1 -- 172.21.15.23:0/1983828935 shutdown_connections 2024-03-28T05:20:20.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.391+0000 7fd687031700 1 -- 172.21.15.23:0/1983828935 wait complete. 2024-03-28T05:20:20.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.392+0000 7fd687031700 1 Processor -- start 2024-03-28T05:20:20.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.392+0000 7fd687031700 1 -- start start 2024-03-28T05:20:20.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.392+0000 7fd687031700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6801021c0 0x7fd680075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:20.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.392+0000 7fd687031700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6801033c0 0x7fd680074580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:20.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.392+0000 7fd687031700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd680103d60 0x7fd680074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:20.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.392+0000 7fd687031700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68010dd80 con 0x7fd680103d60 2024-03-28T05:20:20.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.392+0000 7fd687031700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68010dc00 con 0x7fd6801021c0 2024-03-28T05:20:20.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.392+0000 7fd687031700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68010df00 con 0x7fd6801033c0 2024-03-28T05:20:20.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.392+0000 7fd6855ce700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd680103d60 0x7fd680074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:20.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.392+0000 7fd6855ce700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd680103d60 0x7fd680074ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36724/0 (socket says 172.21.15.23:36724) 2024-03-28T05:20:20.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.392+0000 7fd6855ce700 1 -- 172.21.15.23:0/2741101392 learned_addr learned my addr 172.21.15.23:0/2741101392 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:20:20.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.392+0000 7fd67ffff700 1 --2- 172.21.15.23:0/2741101392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6801033c0 0x7fd680074580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:20.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.392+0000 7fd684dcd700 1 --2- 172.21.15.23:0/2741101392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6801021c0 0x7fd680075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:20.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.392+0000 7fd6855ce700 1 -- 172.21.15.23:0/2741101392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6801033c0 msgr2=0x7fd680074580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:20.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.392+0000 7fd6855ce700 1 --2- 172.21.15.23:0/2741101392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6801033c0 0x7fd680074580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:20.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.392+0000 7fd6855ce700 1 -- 172.21.15.23:0/2741101392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6801021c0 msgr2=0x7fd680075f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:20.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.392+0000 7fd6855ce700 1 --2- 172.21.15.23:0/2741101392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6801021c0 0x7fd680075f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:20.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.392+0000 7fd6855ce700 1 -- 172.21.15.23:0/2741101392 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd680075370 con 0x7fd680103d60 2024-03-28T05:20:20.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.393+0000 7fd6855ce700 1 --2- 172.21.15.23:0/2741101392 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd680103d60 0x7fd680074ac0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd67800ada0 tx=0x7fd67800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:20.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.393+0000 7fd67dffb700 1 -- 172.21.15.23:0/2741101392 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd678017020 con 0x7fd680103d60 2024-03-28T05:20:20.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.393+0000 7fd67dffb700 1 -- 172.21.15.23:0/2741101392 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd67800a0d0 con 0x7fd680103d60 2024-03-28T05:20:20.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.393+0000 7fd687031700 1 -- 172.21.15.23:0/2741101392 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6801a2580 con 0x7fd680103d60 2024-03-28T05:20:20.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.393+0000 7fd67dffb700 1 -- 172.21.15.23:0/2741101392 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd678005480 con 0x7fd680103d60 2024-03-28T05:20:20.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.393+0000 7fd687031700 1 -- 172.21.15.23:0/2741101392 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6801a2a90 con 0x7fd680103d60 2024-03-28T05:20:20.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.394+0000 7fd687031700 1 -- 172.21.15.23:0/2741101392 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd680068e50 con 0x7fd680103d60 2024-03-28T05:20:20.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.395+0000 7fd67dffb700 1 -- 172.21.15.23:0/2741101392 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd678005620 con 0x7fd680103d60 2024-03-28T05:20:20.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.395+0000 7fd67dffb700 1 --2- 172.21.15.23:0/2741101392 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd6700a2910 0x7fd6700a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:20.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.395+0000 7fd684dcd700 1 --2- 172.21.15.23:0/2741101392 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd6700a2910 0x7fd6700a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:20.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.395+0000 7fd67dffb700 1 -- 172.21.15.23:0/2741101392 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd6780cdc00 con 0x7fd680103d60 2024-03-28T05:20:20.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.396+0000 7fd684dcd700 1 --2- 172.21.15.23:0/2741101392 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd6700a2910 0x7fd6700a4dd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd674005fd0 tx=0x7fd674005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:20.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.400+0000 7fd67dffb700 1 -- 172.21.15.23:0/2741101392 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd6780d4080 con 0x7fd680103d60 2024-03-28T05:20:20.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.606+0000 7fd687031700 1 -- 172.21.15.23:0/2741101392 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fd68002da30 con 0x7fd680103d60 2024-03-28T05:20:20.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.606+0000 7fd67dffb700 1 -- 172.21.15.23:0/2741101392 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) v1 ==== 74+0+133 (secure 0 0 0) 0x7fd678021020 con 0x7fd680103d60 2024-03-28T05:20:20.605 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:20:20.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.608+0000 7fd687031700 1 -- 172.21.15.23:0/2741101392 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd6700a2910 msgr2=0x7fd6700a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:20.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.608+0000 7fd687031700 1 --2- 172.21.15.23:0/2741101392 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd6700a2910 0x7fd6700a4dd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd674005fd0 tx=0x7fd674005f60 comp rx=0 tx=0).stop 2024-03-28T05:20:20.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.608+0000 7fd687031700 1 -- 172.21.15.23:0/2741101392 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd680103d60 msgr2=0x7fd680074ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:20.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.608+0000 7fd687031700 1 --2- 172.21.15.23:0/2741101392 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd680103d60 0x7fd680074ac0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd67800ada0 tx=0x7fd67800c330 comp rx=0 tx=0).stop 2024-03-28T05:20:20.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.608+0000 7fd687031700 1 -- 172.21.15.23:0/2741101392 shutdown_connections 2024-03-28T05:20:20.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.608+0000 7fd687031700 1 --2- 172.21.15.23:0/2741101392 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd6700a2910 0x7fd6700a4dd0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:20.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.608+0000 7fd687031700 1 --2- 172.21.15.23:0/2741101392 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd680103d60 0x7fd680074ac0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:20.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.608+0000 7fd687031700 1 --2- 172.21.15.23:0/2741101392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6801021c0 0x7fd680075f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:20.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.608+0000 7fd687031700 1 --2- 172.21.15.23:0/2741101392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6801033c0 0x7fd680074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:20.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.608+0000 7fd687031700 1 -- 172.21.15.23:0/2741101392 >> 172.21.15.23:0/2741101392 conn(0x7fd6800fd7a0 msgr2=0x7fd680106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:20.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.608+0000 7fd687031700 1 -- 172.21.15.23:0/2741101392 shutdown_connections 2024-03-28T05:20:20.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:20.608+0000 7fd687031700 1 -- 172.21.15.23:0/2741101392 wait complete. 2024-03-28T05:20:21.331 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1711603220,"num_in_osds":12,"osd_in_since":1711603200,"num_remapped_pgs":0} 2024-03-28T05:20:21.332 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T05:20:21.432 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:21 smithi113 ceph-mon[37164]: purged_snaps scrub starts 2024-03-28T05:20:21.433 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:21 smithi113 ceph-mon[37164]: purged_snaps scrub ok 2024-03-28T05:20:21.433 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:21 smithi113 ceph-mon[37164]: osd.11 [v2:172.21.15.113:6824/2113644020,v1:172.21.15.113:6825/2113644020] boot 2024-03-28T05:20:21.433 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:21 smithi113 ceph-mon[37164]: osdmap e80: 12 total, 12 up, 12 in 2024-03-28T05:20:21.433 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:21 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T05:20:21.433 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:21 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2741101392' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T05:20:21.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:21 smithi023 ceph-mon[31954]: purged_snaps scrub starts 2024-03-28T05:20:21.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:21 smithi023 ceph-mon[31954]: purged_snaps scrub ok 2024-03-28T05:20:21.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:21 smithi023 ceph-mon[31954]: osd.11 [v2:172.21.15.113:6824/2113644020,v1:172.21.15.113:6825/2113644020] boot 2024-03-28T05:20:21.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:21 smithi023 ceph-mon[31954]: osdmap e80: 12 total, 12 up, 12 in 2024-03-28T05:20:21.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:21 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T05:20:21.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:21 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2741101392' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T05:20:21.585 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:20:21.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:21 smithi098 ceph-mon[36087]: purged_snaps scrub starts 2024-03-28T05:20:21.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:21 smithi098 ceph-mon[36087]: purged_snaps scrub ok 2024-03-28T05:20:21.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:21 smithi098 ceph-mon[36087]: osd.11 [v2:172.21.15.113:6824/2113644020,v1:172.21.15.113:6825/2113644020] boot 2024-03-28T05:20:21.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:21 smithi098 ceph-mon[36087]: osdmap e80: 12 total, 12 up, 12 in 2024-03-28T05:20:21.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:21 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T05:20:21.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:21 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2741101392' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T05:20:22.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:22 smithi113 ceph-mon[37164]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:22.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:22 smithi113 ceph-mon[37164]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T05:20:22.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:22 smithi113 ceph-mon[37164]: Detected new or changed devices on smithi113 2024-03-28T05:20:22.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:22 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:22.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:22 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:22.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:22 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:20:22.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:22 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:20:22.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:22 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:20:22.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:22 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:20:22.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:22 smithi113 ceph-mon[37164]: Adjusting osd_memory_target on smithi113 to 4270M 2024-03-28T05:20:22.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:22 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:22.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:22 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:22.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:22 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:20:22.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:22 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:22.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:22 smithi023 ceph-mon[31954]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:22.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:22 smithi023 ceph-mon[31954]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T05:20:22.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:22 smithi023 ceph-mon[31954]: Detected new or changed devices on smithi113 2024-03-28T05:20:22.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:22.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:22.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:20:22.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:20:22.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:20:22.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:20:22.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:22 smithi023 ceph-mon[31954]: Adjusting osd_memory_target on smithi113 to 4270M 2024-03-28T05:20:22.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:22.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:22.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:20:22.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:22 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:22.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:22 smithi098 ceph-mon[36087]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:22.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:22 smithi098 ceph-mon[36087]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T05:20:22.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:22 smithi098 ceph-mon[36087]: Detected new or changed devices on smithi113 2024-03-28T05:20:22.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:22 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:22.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:22 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:22.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:22 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:20:22.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:22 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:20:22.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:22 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:20:22.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:22 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:20:22.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:22 smithi098 ceph-mon[36087]: Adjusting osd_memory_target on smithi113 to 4270M 2024-03-28T05:20:22.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:22 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:22.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:22 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:22.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:22 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:20:22.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:22 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:23.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.250+0000 7fcdfc3b7700 1 -- 172.21.15.23:0/1574709896 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdf4100d30 msgr2=0x7fcdf4101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:23.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.250+0000 7fcdfc3b7700 1 --2- 172.21.15.23:0/1574709896 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdf4100d30 0x7fcdf4101110 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00b0d0 tx=0x7fcdec00b3e0 comp rx=0 tx=0).stop 2024-03-28T05:20:23.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.250+0000 7fcdfc3b7700 1 -- 172.21.15.23:0/1574709896 shutdown_connections 2024-03-28T05:20:23.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.250+0000 7fcdfc3b7700 1 --2- 172.21.15.23:0/1574709896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdf4107550 0x7fcdf410f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:23.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.250+0000 7fcdfc3b7700 1 --2- 172.21.15.23:0/1574709896 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdf4100d30 0x7fcdf4101110 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:23.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.250+0000 7fcdfc3b7700 1 --2- 172.21.15.23:0/1574709896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdf41016e0 0x7fcdf4106e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:23.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.250+0000 7fcdfc3b7700 1 -- 172.21.15.23:0/1574709896 >> 172.21.15.23:0/1574709896 conn(0x7fcdf40763d0 msgr2=0x7fcdf40767e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:23.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.250+0000 7fcdfc3b7700 1 -- 172.21.15.23:0/1574709896 shutdown_connections 2024-03-28T05:20:23.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.250+0000 7fcdfc3b7700 1 -- 172.21.15.23:0/1574709896 wait complete. 2024-03-28T05:20:23.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.251+0000 7fcdfc3b7700 1 Processor -- start 2024-03-28T05:20:23.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.251+0000 7fcdfc3b7700 1 -- start start 2024-03-28T05:20:23.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.251+0000 7fcdfc3b7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdf4100d30 0x7fcdf41a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:23.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.251+0000 7fcdfc3b7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdf41016e0 0x7fcdf41a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:23.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.251+0000 7fcdfc3b7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdf4107550 0x7fcdf419aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:23.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.251+0000 7fcdfc3b7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf41125c0 con 0x7fcdf41016e0 2024-03-28T05:20:23.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.251+0000 7fcdfc3b7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf4112440 con 0x7fcdf4107550 2024-03-28T05:20:23.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.251+0000 7fcdfc3b7700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf4112740 con 0x7fcdf4100d30 2024-03-28T05:20:23.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.251+0000 7fcdf9952700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdf41016e0 0x7fcdf41a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:23.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.251+0000 7fcdf9952700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdf41016e0 0x7fcdf41a0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36500/0 (socket says 172.21.15.23:36500) 2024-03-28T05:20:23.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.251+0000 7fcdf9952700 1 -- 172.21.15.23:0/1663728691 learned_addr learned my addr 172.21.15.23:0/1663728691 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:20:23.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.252+0000 7fcdfa153700 1 --2- 172.21.15.23:0/1663728691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdf4100d30 0x7fcdf41a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:23.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.252+0000 7fcdfa954700 1 --2- 172.21.15.23:0/1663728691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdf4107550 0x7fcdf419aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:23.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.252+0000 7fcdf9952700 1 -- 172.21.15.23:0/1663728691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdf4100d30 msgr2=0x7fcdf41a08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:23.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.252+0000 7fcdf9952700 1 --2- 172.21.15.23:0/1663728691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdf4100d30 0x7fcdf41a08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:23.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.252+0000 7fcdf9952700 1 -- 172.21.15.23:0/1663728691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdf4107550 msgr2=0x7fcdf419aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:23.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.252+0000 7fcdf9952700 1 --2- 172.21.15.23:0/1663728691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdf4107550 0x7fcdf419aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:23.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.252+0000 7fcdf9952700 1 -- 172.21.15.23:0/1663728691 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf419b200 con 0x7fcdf41016e0 2024-03-28T05:20:23.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.252+0000 7fcdf9952700 1 --2- 172.21.15.23:0/1663728691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdf41016e0 0x7fcdf41a0e20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcde800b4d0 tx=0x7fcde800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:23.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.252+0000 7fcdf37fe700 1 -- 172.21.15.23:0/1663728691 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde8015020 con 0x7fcdf41016e0 2024-03-28T05:20:23.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.252+0000 7fcdfc3b7700 1 -- 172.21.15.23:0/1663728691 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf419b4f0 con 0x7fcdf41016e0 2024-03-28T05:20:23.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.252+0000 7fcdf37fe700 1 -- 172.21.15.23:0/1663728691 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcde8011d10 con 0x7fcdf41016e0 2024-03-28T05:20:23.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.252+0000 7fcdf37fe700 1 -- 172.21.15.23:0/1663728691 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde8010e00 con 0x7fcdf41016e0 2024-03-28T05:20:23.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.252+0000 7fcdfc3b7700 1 -- 172.21.15.23:0/1663728691 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf41a77b0 con 0x7fcdf41016e0 2024-03-28T05:20:23.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.253+0000 7fcdfc3b7700 1 -- 172.21.15.23:0/1663728691 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf4068e50 con 0x7fcdf41016e0 2024-03-28T05:20:23.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.254+0000 7fcdf37fe700 1 -- 172.21.15.23:0/1663728691 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcde8018020 con 0x7fcdf41016e0 2024-03-28T05:20:23.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.255+0000 7fcdf37fe700 1 --2- 172.21.15.23:0/1663728691 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fcddc0a2890 0x7fcddc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:23.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.255+0000 7fcdfa153700 1 --2- 172.21.15.23:0/1663728691 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fcddc0a2890 0x7fcddc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:23.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.255+0000 7fcdf37fe700 1 -- 172.21.15.23:0/1663728691 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcde80ce370 con 0x7fcdf41016e0 2024-03-28T05:20:23.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.255+0000 7fcdfa153700 1 --2- 172.21.15.23:0/1663728691 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fcddc0a2890 0x7fcddc0a4d50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcdec000f80 tx=0x7fcdec00b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:23.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.258+0000 7fcdf37fe700 1 -- 172.21.15.23:0/1663728691 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcde801f020 con 0x7fcdf41016e0 2024-03-28T05:20:23.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.462+0000 7fcdfc3b7700 1 -- 172.21.15.23:0/1663728691 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcdf419bfd0 con 0x7fcdf41016e0 2024-03-28T05:20:23.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.463+0000 7fcdf37fe700 1 -- 172.21.15.23:0/1663728691 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18410 (secure 0 0 0) 0x7fcde809b8b0 con 0x7fcdf41016e0 2024-03-28T05:20:23.463 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:20:23.463 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":81,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","created":"2024-03-28T05:13:13.453653+0000","modified":"2024-03-28T05:20:21.227349+0000","last_up_change":"2024-03-28T05:20:20.221527+0000","last_in_change":"2024-03-28T05:20:00.365478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T05:16:32.986974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18150942-59a4-4c3b-83d4-c1fc3ccf1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6803","nonce":1219956881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6805","nonce":1219956881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6809","nonce":1219956881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6807","nonce":1219956881}]},"public_addr":"172.21.15.23:6803/1219956881","cluster_addr":"172.21.15.23:6805/1219956881","heartbeat_back_addr":"172.21.15.23:6809/1219956881","heartbeat_front_addr":"172.21.15.23:6807/1219956881","state":["exists","up"]},{"osd":1,"uuid":"ba6d1a95-5135-4670-95f9-7878c0e7bd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6801","nonce":555491234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6803","nonce":555491234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6807","nonce":555491234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6805","nonce":555491234}]},"public_addr":"172.21.15.98:6801/555491234","cluster_addr":"172.21.15.98:6803/555491234","heartbeat_back_addr":"172.21.15.98:6807/555491234","heartbeat_front_addr":"172.21.15.98:6805/555491234","state":["exists","up"]},{"osd":2,"uuid":"873bdd60-2a47-403b-b1f4-ee9226a95067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6801","nonce":1535856914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6803","nonce":1535856914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6807","nonce":1535856914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6805","nonce":1535856914}]},"public_addr":"172.21.15.113:6801/1535856914","cluster_addr":"172.21.15.113:6803/1535856914","heartbeat_back_addr":"172.21.15.113:6807/1535856914","heartbeat_front_addr":"172.21.15.113:6805/1535856914","state":["exists","up"]},{"osd":3,"uuid":"83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6811","nonce":2949415020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6813","nonce":2949415020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6817","nonce":2949415020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6815","nonce":2949415020}]},"public_addr":"172.21.15.23:6811/2949415020","cluster_addr":"172.21.15.23:6813/2949415020","heartbeat_back_addr":"172.21.15.23:6817/2949415020","heartbeat_front_addr":"172.21.15.23:6815/2949415020","state":["exists","up"]},{"osd":4,"uuid":"76a90bac-b221-40a3-8238-40e8049cec36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6809","nonce":2615902819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6811","nonce":2615902819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6815","nonce":2615902819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6813","nonce":2615902819}]},"public_addr":"172.21.15.98:6809/2615902819","cluster_addr":"172.21.15.98:6811/2615902819","heartbeat_back_addr":"172.21.15.98:6815/2615902819","heartbeat_front_addr":"172.21.15.98:6813/2615902819","state":["exists","up"]},{"osd":5,"uuid":"b53fdb55-a8df-40a8-bd1b-8bcef634e26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6809","nonce":625271528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6811","nonce":625271528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6815","nonce":625271528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6813","nonce":625271528}]},"public_addr":"172.21.15.113:6809/625271528","cluster_addr":"172.21.15.113:6811/625271528","heartbeat_back_addr":"172.21.15.113:6815/625271528","heartbeat_front_addr":"172.21.15.113:6813/625271528","state":["exists","up"]},{"osd":6,"uuid":"f00283d2-8d67-4e97-a575-9a9d3b55d5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6819","nonce":4188464176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6821","nonce":4188464176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6825","nonce":4188464176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6823","nonce":4188464176}]},"public_addr":"172.21.15.23:6819/4188464176","cluster_addr":"172.21.15.23:6821/4188464176","heartbeat_back_addr":"172.21.15.23:6825/4188464176","heartbeat_front_addr":"172.21.15.23:6823/4188464176","state":["exists","up"]},{"osd":7,"uuid":"edc008a7-b07b-40a8-88b7-1614ba38a775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6817","nonce":1985546028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6819","nonce":1985546028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6823","nonce":1985546028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6821","nonce":1985546028}]},"public_addr":"172.21.15.98:6817/1985546028","cluster_addr":"172.21.15.98:6819/1985546028","heartbeat_back_addr":"172.21.15.98:6823/1985546028","heartbeat_front_addr":"172.21.15.98:6821/1985546028","state":["exists","up"]},{"osd":8,"uuid":"56e7b2d1-81f0-47b2-8638-f29832bed8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6817","nonce":3534876081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6819","nonce":3534876081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6823","nonce":3534876081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6821","nonce":3534876081}]},"public_addr":"172.21.15.113:6817/3534876081","cluster_addr":"172.21.15.113:6819/3534876081","heartbeat_back_addr":"172.21.15.113:6823/3534876081","heartbeat_front_addr":"172.21.15.113:6821/3534876081","state":["exists","up"]},{"osd":9,"uuid":"a2e90caa-6e9c-42bf-9f18-81f7f47880cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6827","nonce":672264207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6829","nonce":672264207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6833","nonce":672264207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6831","nonce":672264207}]},"public_addr":"172.21.15.23:6827/672264207","cluster_addr":"172.21.15.23:6829/672264207","heartbeat_back_addr":"172.21.15.23:6833/672264207","heartbeat_front_addr":"172.21.15.23:6831/672264207","state":["exists","up"]},{"osd":10,"uuid":"0eda917f-020c-437f-95ae-b26203f6c68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6825","nonce":2699920595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6827","nonce":2699920595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6831","nonce":2699920595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6829","nonce":2699920595}]},"public_addr":"172.21.15.98:6825/2699920595","cluster_addr":"172.21.15.98:6827/2699920595","heartbeat_back_addr":"172.21.15.98:6831/2699920595","heartbeat_front_addr":"172.21.15.98:6829/2699920595","state":["exists","up"]},{"osd":11,"uuid":"aae78501-78a8-469a-be20-3006f5ec5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6825","nonce":2113644020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6827","nonce":2113644020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6831","nonce":2113644020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6829","nonce":2113644020}]},"public_addr":"172.21.15.113:6825/2113644020","cluster_addr":"172.21.15.113:6827/2113644020","heartbeat_back_addr":"172.21.15.113:6831/2113644020","heartbeat_front_addr":"172.21.15.113:6829/2113644020","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:15:43.527475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:07.006538+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:30.594176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:55.810999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:20.595811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:45.474138+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:10.500409+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:35.787638+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:01.001188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:26.761115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:52.952752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:20:18.559832+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:6800/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1876451381":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2253559862":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2653562762":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/3337463830":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6800/1735221490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/2784771635":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2389902490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6801/1735221490":"2024-03-29T05:13:42.579607+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T05:20:23.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.466+0000 7fcdfc3b7700 1 -- 172.21.15.23:0/1663728691 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fcddc0a2890 msgr2=0x7fcddc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:23.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.466+0000 7fcdfc3b7700 1 --2- 172.21.15.23:0/1663728691 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fcddc0a2890 0x7fcddc0a4d50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcdec000f80 tx=0x7fcdec00b040 comp rx=0 tx=0).stop 2024-03-28T05:20:23.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.466+0000 7fcdfc3b7700 1 -- 172.21.15.23:0/1663728691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdf41016e0 msgr2=0x7fcdf41a0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:23.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.466+0000 7fcdfc3b7700 1 --2- 172.21.15.23:0/1663728691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdf41016e0 0x7fcdf41a0e20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcde800b4d0 tx=0x7fcde800b890 comp rx=0 tx=0).stop 2024-03-28T05:20:23.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.466+0000 7fcdfc3b7700 1 -- 172.21.15.23:0/1663728691 shutdown_connections 2024-03-28T05:20:23.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.466+0000 7fcdfc3b7700 1 --2- 172.21.15.23:0/1663728691 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fcddc0a2890 0x7fcddc0a4d50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:23.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.466+0000 7fcdfc3b7700 1 --2- 172.21.15.23:0/1663728691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcdf4107550 0x7fcdf419aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:23.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.466+0000 7fcdfc3b7700 1 --2- 172.21.15.23:0/1663728691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdf4100d30 0x7fcdf41a08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:23.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.466+0000 7fcdfc3b7700 1 --2- 172.21.15.23:0/1663728691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcdf41016e0 0x7fcdf41a0e20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:23.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.466+0000 7fcdfc3b7700 1 -- 172.21.15.23:0/1663728691 >> 172.21.15.23:0/1663728691 conn(0x7fcdf40763d0 msgr2=0x7fcdf40ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:23.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.466+0000 7fcdfc3b7700 1 -- 172.21.15.23:0/1663728691 shutdown_connections 2024-03-28T05:20:23.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:23.466+0000 7fcdfc3b7700 1 -- 172.21.15.23:0/1663728691 wait complete. 2024-03-28T05:20:24.206 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T05:16:32.986974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T05:20:24.207 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-28T05:20:24.449 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:20:24.491 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:24 smithi113 ceph-mon[37164]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:24.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:24 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1663728691' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:20:24.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:24 smithi023 ceph-mon[31954]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:24.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:24 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1663728691' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:20:24.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:24 smithi098 ceph-mon[36087]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:24.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:24 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1663728691' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:20:26.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.223+0000 7fd766ce3700 1 -- 172.21.15.23:0/2856652282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd760101320 msgr2=0x7fd760101700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:26.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.223+0000 7fd766ce3700 1 --2- 172.21.15.23:0/2856652282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd760101320 0x7fd760101700 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd754009990 tx=0x7fd754009ca0 comp rx=0 tx=0).stop 2024-03-28T05:20:26.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.223+0000 7fd766ce3700 1 -- 172.21.15.23:0/2856652282 shutdown_connections 2024-03-28T05:20:26.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.223+0000 7fd766ce3700 1 --2- 172.21.15.23:0/2856652282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd7601063b0 0x7fd76010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:26.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.223+0000 7fd766ce3700 1 --2- 172.21.15.23:0/2856652282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd760101320 0x7fd760101700 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:26.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.223+0000 7fd766ce3700 1 --2- 172.21.15.23:0/2856652282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd760101cd0 0x7fd760105cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:26.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.223+0000 7fd766ce3700 1 -- 172.21.15.23:0/2856652282 >> 172.21.15.23:0/2856652282 conn(0x7fd7600770b0 msgr2=0x7fd7600774c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:26.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.223+0000 7fd766ce3700 1 -- 172.21.15.23:0/2856652282 shutdown_connections 2024-03-28T05:20:26.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.223+0000 7fd766ce3700 1 -- 172.21.15.23:0/2856652282 wait complete. 2024-03-28T05:20:26.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.223+0000 7fd766ce3700 1 Processor -- start 2024-03-28T05:20:26.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.223+0000 7fd766ce3700 1 -- start start 2024-03-28T05:20:26.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.224+0000 7fd766ce3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd760101320 0x7fd76010e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:26.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.224+0000 7fd766ce3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd760101cd0 0x7fd76010ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:26.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.224+0000 7fd766ce3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7601063b0 0x7fd76010f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:26.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.224+0000 7fd766ce3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd760112510 con 0x7fd7601063b0 2024-03-28T05:20:26.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.224+0000 7fd766ce3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd760112390 con 0x7fd760101cd0 2024-03-28T05:20:26.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.224+0000 7fd766ce3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd760112690 con 0x7fd760101320 2024-03-28T05:20:26.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.224+0000 7fd765280700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7601063b0 0x7fd76010f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:26.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.224+0000 7fd765280700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7601063b0 0x7fd76010f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36512/0 (socket says 172.21.15.23:36512) 2024-03-28T05:20:26.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.224+0000 7fd765280700 1 -- 172.21.15.23:0/3067844200 learned_addr learned my addr 172.21.15.23:0/3067844200 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:20:26.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.224+0000 7fd764a7f700 1 --2- 172.21.15.23:0/3067844200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd760101320 0x7fd76010e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:26.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.224+0000 7fd75ffff700 1 --2- 172.21.15.23:0/3067844200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd760101cd0 0x7fd76010ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:26.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.224+0000 7fd765280700 1 -- 172.21.15.23:0/3067844200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd760101320 msgr2=0x7fd76010e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:26.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.224+0000 7fd765280700 1 --2- 172.21.15.23:0/3067844200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd760101320 0x7fd76010e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:26.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.224+0000 7fd765280700 1 -- 172.21.15.23:0/3067844200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd760101cd0 msgr2=0x7fd76010ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:26.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.224+0000 7fd765280700 1 --2- 172.21.15.23:0/3067844200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd760101cd0 0x7fd76010ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:26.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.224+0000 7fd765280700 1 -- 172.21.15.23:0/3067844200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7601a6ed0 con 0x7fd7601063b0 2024-03-28T05:20:26.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.224+0000 7fd75ffff700 1 --2- 172.21.15.23:0/3067844200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd760101cd0 0x7fd76010ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:20:26.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.224+0000 7fd765280700 1 --2- 172.21.15.23:0/3067844200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7601063b0 0x7fd76010f3c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd75800ada0 tx=0x7fd75800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:26.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.224+0000 7fd75dffb700 1 -- 172.21.15.23:0/3067844200 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd758017020 con 0x7fd7601063b0 2024-03-28T05:20:26.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.224+0000 7fd75dffb700 1 -- 172.21.15.23:0/3067844200 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd75800a0d0 con 0x7fd7601063b0 2024-03-28T05:20:26.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.224+0000 7fd766ce3700 1 -- 172.21.15.23:0/3067844200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7601a71c0 con 0x7fd7601063b0 2024-03-28T05:20:26.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.224+0000 7fd75dffb700 1 -- 172.21.15.23:0/3067844200 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd758005480 con 0x7fd7601063b0 2024-03-28T05:20:26.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.225+0000 7fd766ce3700 1 -- 172.21.15.23:0/3067844200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7601a7700 con 0x7fd7601063b0 2024-03-28T05:20:26.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.226+0000 7fd766ce3700 1 -- 172.21.15.23:0/3067844200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7601027b0 con 0x7fd7601063b0 2024-03-28T05:20:26.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.227+0000 7fd75dffb700 1 -- 172.21.15.23:0/3067844200 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd758005620 con 0x7fd7601063b0 2024-03-28T05:20:26.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.227+0000 7fd75dffb700 1 --2- 172.21.15.23:0/3067844200 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd7500a2890 0x7fd7500a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:26.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.227+0000 7fd764a7f700 1 --2- 172.21.15.23:0/3067844200 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd7500a2890 0x7fd7500a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:26.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.227+0000 7fd75dffb700 1 -- 172.21.15.23:0/3067844200 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd7580cdce0 con 0x7fd7601063b0 2024-03-28T05:20:26.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.228+0000 7fd764a7f700 1 --2- 172.21.15.23:0/3067844200 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd7500a2890 0x7fd7500a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd75400b5c0 tx=0x7fd754005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:26.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.232+0000 7fd75dffb700 1 -- 172.21.15.23:0/3067844200 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd758021020 con 0x7fd7601063b0 2024-03-28T05:20:26.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.437+0000 7fd766ce3700 1 -- 172.21.15.23:0/3067844200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd76002dbf0 con 0x7fd7601063b0 2024-03-28T05:20:26.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.438+0000 7fd75dffb700 1 -- 172.21.15.23:0/3067844200 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd75809b2b0 con 0x7fd7601063b0 2024-03-28T05:20:26.436 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-28T05:20:26.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.440+0000 7fd766ce3700 1 -- 172.21.15.23:0/3067844200 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd7500a2890 msgr2=0x7fd7500a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:26.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.440+0000 7fd766ce3700 1 --2- 172.21.15.23:0/3067844200 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd7500a2890 0x7fd7500a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd75400b5c0 tx=0x7fd754005960 comp rx=0 tx=0).stop 2024-03-28T05:20:26.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.440+0000 7fd766ce3700 1 -- 172.21.15.23:0/3067844200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7601063b0 msgr2=0x7fd76010f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:26.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.440+0000 7fd766ce3700 1 --2- 172.21.15.23:0/3067844200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7601063b0 0x7fd76010f3c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd75800ada0 tx=0x7fd75800c330 comp rx=0 tx=0).stop 2024-03-28T05:20:26.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.440+0000 7fd766ce3700 1 -- 172.21.15.23:0/3067844200 shutdown_connections 2024-03-28T05:20:26.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.440+0000 7fd766ce3700 1 --2- 172.21.15.23:0/3067844200 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd7500a2890 0x7fd7500a4d50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:26.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.440+0000 7fd766ce3700 1 --2- 172.21.15.23:0/3067844200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7601063b0 0x7fd76010f3c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:26.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.440+0000 7fd766ce3700 1 --2- 172.21.15.23:0/3067844200 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd760101320 0x7fd76010e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:26.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.440+0000 7fd766ce3700 1 --2- 172.21.15.23:0/3067844200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd760101cd0 0x7fd76010ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:26.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.440+0000 7fd766ce3700 1 -- 172.21.15.23:0/3067844200 >> 172.21.15.23:0/3067844200 conn(0x7fd7600770b0 msgr2=0x7fd760100850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:26.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.440+0000 7fd766ce3700 1 -- 172.21.15.23:0/3067844200 shutdown_connections 2024-03-28T05:20:26.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:26.440+0000 7fd766ce3700 1 -- 172.21.15.23:0/3067844200 wait complete. 2024-03-28T05:20:26.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:26 smithi023 ceph-mon[31954]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:26.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:26 smithi098 ceph-mon[36087]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:26.741 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:26 smithi113 ceph-mon[37164]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:27.125 INFO:tasks.cephadm:Adding mds.a on smithi023 2024-03-28T05:20:27.125 INFO:tasks.cephadm:Adding mds.d on smithi023 2024-03-28T05:20:27.125 INFO:tasks.cephadm:Adding mds.g on smithi023 2024-03-28T05:20:27.125 INFO:tasks.cephadm:Adding mds.j on smithi023 2024-03-28T05:20:27.125 INFO:tasks.cephadm:Adding mds.b on smithi098 2024-03-28T05:20:27.125 INFO:tasks.cephadm:Adding mds.e on smithi098 2024-03-28T05:20:27.125 INFO:tasks.cephadm:Adding mds.h on smithi098 2024-03-28T05:20:27.125 INFO:tasks.cephadm:Adding mds.k on smithi098 2024-03-28T05:20:27.126 INFO:tasks.cephadm:Adding mds.c on smithi113 2024-03-28T05:20:27.126 INFO:tasks.cephadm:Adding mds.f on smithi113 2024-03-28T05:20:27.126 INFO:tasks.cephadm:Adding mds.i on smithi113 2024-03-28T05:20:27.126 INFO:tasks.cephadm:Adding mds.l on smithi113 2024-03-28T05:20:27.126 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph orch apply mds all '12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi113=c;smithi113=f;smithi113=i;smithi113=l' 2024-03-28T05:20:27.379 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T05:20:27.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:27 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3067844200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:20:27.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:27 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3067844200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:20:27.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:27 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3067844200' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:20:28.515 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:28 smithi113 ceph-mon[37164]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:28.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:28 smithi023 ceph-mon[31954]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:28.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:28 smithi098 ceph-mon[36087]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:29.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.056+0000 7fd48cb2f700 1 -- 172.21.15.113:0/3637358244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd488102c50 msgr2=0x7fd488109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:29.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.056+0000 7fd48cb2f700 1 --2- 172.21.15.113:0/3637358244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd488102c50 0x7fd488109500 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd48000b0a0 tx=0x7fd48000b3b0 comp rx=0 tx=0).stop 2024-03-28T05:20:29.056 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.056+0000 7fd48cb2f700 1 -- 172.21.15.113:0/3637358244 shutdown_connections 2024-03-28T05:20:29.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.056+0000 7fd48cb2f700 1 --2- 172.21.15.113:0/3637358244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd488102c50 0x7fd488109500 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:29.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.056+0000 7fd48cb2f700 1 --2- 172.21.15.113:0/3637358244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4881010b0 0x7fd4881014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:29.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.056+0000 7fd48cb2f700 1 --2- 172.21.15.113:0/3637358244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4881022b0 0x7fd488102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:29.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.056+0000 7fd48cb2f700 1 -- 172.21.15.113:0/3637358244 >> 172.21.15.113:0/3637358244 conn(0x7fd4880fc670 msgr2=0x7fd4880fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:29.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.056+0000 7fd48cb2f700 1 -- 172.21.15.113:0/3637358244 shutdown_connections 2024-03-28T05:20:29.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.056+0000 7fd48cb2f700 1 -- 172.21.15.113:0/3637358244 wait complete. 2024-03-28T05:20:29.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.056+0000 7fd48cb2f700 1 Processor -- start 2024-03-28T05:20:29.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.056+0000 7fd48cb2f700 1 -- start start 2024-03-28T05:20:29.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.057+0000 7fd48cb2f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4881010b0 0x7fd48810ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:29.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.057+0000 7fd48cb2f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4881022b0 0x7fd48810a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:29.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.057+0000 7fd48cb2f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd488102c50 0x7fd48810a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:29.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.057+0000 7fd48cb2f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48810de10 con 0x7fd4881010b0 2024-03-28T05:20:29.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.057+0000 7fd48cb2f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48810dc90 con 0x7fd488102c50 2024-03-28T05:20:29.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.057+0000 7fd48cb2f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48810df90 con 0x7fd4881022b0 2024-03-28T05:20:29.057 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.057+0000 7fd485d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4881022b0 0x7fd48810a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:29.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.057+0000 7fd485d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4881022b0 0x7fd48810a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:50058/0 (socket says 172.21.15.113:50058) 2024-03-28T05:20:29.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.057+0000 7fd485d9b700 1 -- 172.21.15.113:0/3927997608 learned_addr learned my addr 172.21.15.113:0/3927997608 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-03-28T05:20:29.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.057+0000 7fd486d9d700 1 --2- 172.21.15.113:0/3927997608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd488102c50 0x7fd48810a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:29.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.057+0000 7fd485d9b700 1 -- 172.21.15.113:0/3927997608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd488102c50 msgr2=0x7fd48810a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:29.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.057+0000 7fd48659c700 1 --2- 172.21.15.113:0/3927997608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4881010b0 0x7fd48810ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:29.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.057+0000 7fd485d9b700 1 --2- 172.21.15.113:0/3927997608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd488102c50 0x7fd48810a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:29.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.057+0000 7fd485d9b700 1 -- 172.21.15.113:0/3927997608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4881010b0 msgr2=0x7fd48810ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:29.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.057+0000 7fd485d9b700 1 --2- 172.21.15.113:0/3927997608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4881010b0 0x7fd48810ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:29.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.057+0000 7fd485d9b700 1 -- 172.21.15.113:0/3927997608 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48810aec0 con 0x7fd4881022b0 2024-03-28T05:20:29.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.057+0000 7fd485d9b700 1 --2- 172.21.15.113:0/3927997608 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4881022b0 0x7fd48810a0d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd47c00c9b0 tx=0x7fd47c00cd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:29.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.057+0000 7fd4777fe700 1 -- 172.21.15.113:0/3927997608 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47c007b00 con 0x7fd4881022b0 2024-03-28T05:20:29.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.057+0000 7fd48cb2f700 1 -- 172.21.15.113:0/3927997608 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4881a2490 con 0x7fd4881022b0 2024-03-28T05:20:29.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.057+0000 7fd4777fe700 1 -- 172.21.15.113:0/3927997608 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd47c007ca0 con 0x7fd4881022b0 2024-03-28T05:20:29.058 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.057+0000 7fd4777fe700 1 -- 172.21.15.113:0/3927997608 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd47c018780 con 0x7fd4881022b0 2024-03-28T05:20:29.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.057+0000 7fd48cb2f700 1 -- 172.21.15.113:0/3927997608 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4881a2a50 con 0x7fd4881022b0 2024-03-28T05:20:29.059 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.059+0000 7fd48cb2f700 1 -- 172.21.15.113:0/3927997608 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd488068e50 con 0x7fd4881022b0 2024-03-28T05:20:29.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.059+0000 7fd4777fe700 1 -- 172.21.15.113:0/3927997608 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd47c00f450 con 0x7fd4881022b0 2024-03-28T05:20:29.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.060+0000 7fd4777fe700 1 --2- 172.21.15.113:0/3927997608 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd4700a2910 0x7fd4700a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:29.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.060+0000 7fd4777fe700 1 -- 172.21.15.113:0/3927997608 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd47c0ce5e0 con 0x7fd4881022b0 2024-03-28T05:20:29.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.060+0000 7fd48659c700 1 --2- 172.21.15.113:0/3927997608 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd4700a2910 0x7fd4700a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:29.060 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.061+0000 7fd48659c700 1 --2- 172.21.15.113:0/3927997608 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd4700a2910 0x7fd4700a4dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd478006fd0 tx=0x7fd478008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:29.063 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.064+0000 7fd4777fe700 1 -- 172.21.15.113:0/3927997608 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd47c014030 con 0x7fd4881022b0 2024-03-28T05:20:29.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.274+0000 7fd48cb2f700 1 -- 172.21.15.113:0/3927997608 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi113=c;smithi113=f;smithi113=i;smithi113=l", "target": ["mon-mgr", ""]}) v1 -- 0x7fd4880627d0 con 0x7fd4700a2910 2024-03-28T05:20:29.281 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.283+0000 7fd4777fe700 1 -- 172.21.15.113:0/3927997608 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fd4880627d0 con 0x7fd4700a2910 2024-03-28T05:20:29.282 INFO:teuthology.orchestra.run.smithi113.stdout:Scheduled mds.all update... 2024-03-28T05:20:29.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.285+0000 7fd48cb2f700 1 -- 172.21.15.113:0/3927997608 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd4700a2910 msgr2=0x7fd4700a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:29.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.285+0000 7fd48cb2f700 1 --2- 172.21.15.113:0/3927997608 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd4700a2910 0x7fd4700a4dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd478006fd0 tx=0x7fd478008040 comp rx=0 tx=0).stop 2024-03-28T05:20:29.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.285+0000 7fd48cb2f700 1 -- 172.21.15.113:0/3927997608 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4881022b0 msgr2=0x7fd48810a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:29.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.285+0000 7fd48cb2f700 1 --2- 172.21.15.113:0/3927997608 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4881022b0 0x7fd48810a0d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd47c00c9b0 tx=0x7fd47c00cd70 comp rx=0 tx=0).stop 2024-03-28T05:20:29.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.285+0000 7fd48cb2f700 1 -- 172.21.15.113:0/3927997608 shutdown_connections 2024-03-28T05:20:29.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.285+0000 7fd48cb2f700 1 --2- 172.21.15.113:0/3927997608 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd4700a2910 0x7fd4700a4dd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:29.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.285+0000 7fd48cb2f700 1 --2- 172.21.15.113:0/3927997608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd488102c50 0x7fd48810a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:29.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.285+0000 7fd48cb2f700 1 --2- 172.21.15.113:0/3927997608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4881010b0 0x7fd48810ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:29.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.285+0000 7fd48cb2f700 1 --2- 172.21.15.113:0/3927997608 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4881022b0 0x7fd48810a0d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:29.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.285+0000 7fd48cb2f700 1 -- 172.21.15.113:0/3927997608 >> 172.21.15.113:0/3927997608 conn(0x7fd4880fc670 msgr2=0x7fd4881054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:29.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.285+0000 7fd48cb2f700 1 -- 172.21.15.113:0/3927997608 shutdown_connections 2024-03-28T05:20:29.284 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-28T05:20:29.285+0000 7fd48cb2f700 1 -- 172.21.15.113:0/3927997608 wait complete. 2024-03-28T05:20:29.968 DEBUG:teuthology.orchestra.run.smithi023:mds.a> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.a.service 2024-03-28T05:20:29.971 DEBUG:teuthology.orchestra.run.smithi023:mds.d> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.d.service 2024-03-28T05:20:30.013 DEBUG:teuthology.orchestra.run.smithi023:mds.g> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.g.service 2024-03-28T05:20:30.056 DEBUG:teuthology.orchestra.run.smithi023:mds.j> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.j.service 2024-03-28T05:20:30.099 DEBUG:teuthology.orchestra.run.smithi098:mds.b> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.b.service 2024-03-28T05:20:30.102 DEBUG:teuthology.orchestra.run.smithi098:mds.e> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.e.service 2024-03-28T05:20:30.146 DEBUG:teuthology.orchestra.run.smithi098:mds.h> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.h.service 2024-03-28T05:20:30.189 DEBUG:teuthology.orchestra.run.smithi098:mds.k> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.k.service 2024-03-28T05:20:30.231 DEBUG:teuthology.orchestra.run.smithi113:mds.c> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.c.service 2024-03-28T05:20:30.233 DEBUG:teuthology.orchestra.run.smithi113:mds.f> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.f.service 2024-03-28T05:20:30.276 DEBUG:teuthology.orchestra.run.smithi113:mds.i> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.i.service 2024-03-28T05:20:30.319 DEBUG:teuthology.orchestra.run.smithi113:mds.l> sudo journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.l.service 2024-03-28T05:20:30.362 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-28T05:20:30.362 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T05:20:30.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.473+0000 7f1474ef1640 1 -- 172.21.15.23:0/393011685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14701021f0 msgr2=0x7f1470108a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:30.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.473+0000 7f1474ef1640 1 --2- 172.21.15.23:0/393011685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14701021f0 0x7f1470108a80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1468009ca0 tx=0x7f146802c0f0 comp rx=0 tx=0).stop 2024-03-28T05:20:30.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.473+0000 7f1474ef1640 1 -- 172.21.15.23:0/393011685 shutdown_connections 2024-03-28T05:20:30.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.473+0000 7f1474ef1640 1 --2- 172.21.15.23:0/393011685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14701021f0 0x7f1470108a80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:30.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.473+0000 7f1474ef1640 1 --2- 172.21.15.23:0/393011685 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1470101830 0x7f1470101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:30.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.473+0000 7f1474ef1640 1 --2- 172.21.15.23:0/393011685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1470100640 0x7f1470100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:30.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.473+0000 7f1474ef1640 1 -- 172.21.15.23:0/393011685 >> 172.21.15.23:0/393011685 conn(0x7f14700fbbd0 msgr2=0x7f14700fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:30.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.473+0000 7f1474ef1640 1 -- 172.21.15.23:0/393011685 shutdown_connections 2024-03-28T05:20:30.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.473+0000 7f1474ef1640 1 -- 172.21.15.23:0/393011685 wait complete. 2024-03-28T05:20:30.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.474+0000 7f1474ef1640 1 Processor -- start 2024-03-28T05:20:30.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.474+0000 7f1474ef1640 1 -- start start 2024-03-28T05:20:30.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.474+0000 7f1474ef1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1470100640 0x7f147006a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:30.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.474+0000 7f1474ef1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1470101830 0x7f147019c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:30.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.474+0000 7f1474ef1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14701021f0 0x7f147019ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:30.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.474+0000 7f1474ef1640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147010c250 con 0x7f14701021f0 2024-03-28T05:20:30.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.474+0000 7f1474ef1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147010c0d0 con 0x7f1470101830 2024-03-28T05:20:30.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.474+0000 7f1474ef1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147010c3d0 con 0x7f1470100640 2024-03-28T05:20:30.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.474+0000 7f146ed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14701021f0 0x7f147019ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:30.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.474+0000 7f146e59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1470100640 0x7f147006a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:30.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.474+0000 7f146e59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1470100640 0x7f147006a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:53558/0 (socket says 172.21.15.23:53558) 2024-03-28T05:20:30.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.474+0000 7f146ed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14701021f0 0x7f147019ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36528/0 (socket says 172.21.15.23:36528) 2024-03-28T05:20:30.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.474+0000 7f146e59c640 1 -- 172.21.15.23:0/2298645052 learned_addr learned my addr 172.21.15.23:0/2298645052 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:20:30.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.475+0000 7f1467fff640 1 --2- 172.21.15.23:0/2298645052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1470101830 0x7f147019c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:30.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.475+0000 7f146ed9d640 1 -- 172.21.15.23:0/2298645052 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1470100640 msgr2=0x7f147006a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:30.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.475+0000 7f146ed9d640 1 --2- 172.21.15.23:0/2298645052 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1470100640 0x7f147006a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:30.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.475+0000 7f146ed9d640 1 -- 172.21.15.23:0/2298645052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1470101830 msgr2=0x7f147019c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:30.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.475+0000 7f146ed9d640 1 --2- 172.21.15.23:0/2298645052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1470101830 0x7f147019c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:30.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.475+0000 7f146ed9d640 1 -- 172.21.15.23:0/2298645052 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147019f590 con 0x7f14701021f0 2024-03-28T05:20:30.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.475+0000 7f146e59c640 1 --2- 172.21.15.23:0/2298645052 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1470100640 0x7f147006a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:20:30.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.475+0000 7f146ed9d640 1 --2- 172.21.15.23:0/2298645052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14701021f0 0x7f147019ecd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f146802c630 tx=0x7f1468032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:30.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.475+0000 7f14677fe640 1 -- 172.21.15.23:0/2298645052 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468039040 con 0x7f14701021f0 2024-03-28T05:20:30.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.475+0000 7f1474ef1640 1 -- 172.21.15.23:0/2298645052 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147019f820 con 0x7f14701021f0 2024-03-28T05:20:30.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.475+0000 7f14677fe640 1 -- 172.21.15.23:0/2298645052 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1468044070 con 0x7f14701021f0 2024-03-28T05:20:30.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.475+0000 7f14677fe640 1 -- 172.21.15.23:0/2298645052 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14680042a0 con 0x7f14701021f0 2024-03-28T05:20:30.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.475+0000 7f1474ef1640 1 -- 172.21.15.23:0/2298645052 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147019fd30 con 0x7f14701021f0 2024-03-28T05:20:30.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.476+0000 7f1474ef1640 1 -- 172.21.15.23:0/2298645052 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1470107960 con 0x7f14701021f0 2024-03-28T05:20:30.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.477+0000 7f14677fe640 1 -- 172.21.15.23:0/2298645052 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1468032690 con 0x7f14701021f0 2024-03-28T05:20:30.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.478+0000 7f14677fe640 1 --2- 172.21.15.23:0/2298645052 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f145c0a2800 0x7f145c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:30.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.478+0000 7f146e59c640 1 --2- 172.21.15.23:0/2298645052 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f145c0a2800 0x7f145c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:30.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.478+0000 7f14677fe640 1 -- 172.21.15.23:0/2298645052 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f14680f0990 con 0x7f14701021f0 2024-03-28T05:20:30.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.478+0000 7f146e59c640 1 --2- 172.21.15.23:0/2298645052 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f145c0a2800 0x7f145c0a4cc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f14600104e0 tx=0x7f146000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:30.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.483+0000 7f14677fe640 1 -- 172.21.15.23:0/2298645052 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f14680bdf60 con 0x7f14701021f0 2024-03-28T05:20:30.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:30 smithi098 ceph-mon[36087]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:30.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:30 smithi098 ceph-mon[36087]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi113=c;smithi113=f;smithi113=i;smithi113=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:20:30.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:30 smithi098 ceph-mon[36087]: Saving service mds.all spec with placement smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi113=c;smithi113=f;smithi113=i;smithi113=l;count:12 2024-03-28T05:20:30.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:30 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:30.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:30 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:20:30.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:30 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:30.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:30 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:20:30.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:30 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:30.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:30 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:30.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:30 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:30.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:30 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:30.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.666+0000 7f1474ef1640 1 -- 172.21.15.23:0/2298645052 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f147002d200 con 0x7f14701021f0 2024-03-28T05:20:30.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.667+0000 7f14677fe640 1 -- 172.21.15.23:0/2298645052 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18410 (secure 0 0 0) 0x7f14680bd860 con 0x7f14701021f0 2024-03-28T05:20:30.666 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:20:30.666 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":81,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","created":"2024-03-28T05:13:13.453653+0000","modified":"2024-03-28T05:20:21.227349+0000","last_up_change":"2024-03-28T05:20:20.221527+0000","last_in_change":"2024-03-28T05:20:00.365478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T05:16:32.986974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18150942-59a4-4c3b-83d4-c1fc3ccf1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6803","nonce":1219956881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6805","nonce":1219956881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6809","nonce":1219956881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6807","nonce":1219956881}]},"public_addr":"172.21.15.23:6803/1219956881","cluster_addr":"172.21.15.23:6805/1219956881","heartbeat_back_addr":"172.21.15.23:6809/1219956881","heartbeat_front_addr":"172.21.15.23:6807/1219956881","state":["exists","up"]},{"osd":1,"uuid":"ba6d1a95-5135-4670-95f9-7878c0e7bd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6801","nonce":555491234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6803","nonce":555491234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6807","nonce":555491234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6805","nonce":555491234}]},"public_addr":"172.21.15.98:6801/555491234","cluster_addr":"172.21.15.98:6803/555491234","heartbeat_back_addr":"172.21.15.98:6807/555491234","heartbeat_front_addr":"172.21.15.98:6805/555491234","state":["exists","up"]},{"osd":2,"uuid":"873bdd60-2a47-403b-b1f4-ee9226a95067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6801","nonce":1535856914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6803","nonce":1535856914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6807","nonce":1535856914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6805","nonce":1535856914}]},"public_addr":"172.21.15.113:6801/1535856914","cluster_addr":"172.21.15.113:6803/1535856914","heartbeat_back_addr":"172.21.15.113:6807/1535856914","heartbeat_front_addr":"172.21.15.113:6805/1535856914","state":["exists","up"]},{"osd":3,"uuid":"83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6811","nonce":2949415020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6813","nonce":2949415020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6817","nonce":2949415020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6815","nonce":2949415020}]},"public_addr":"172.21.15.23:6811/2949415020","cluster_addr":"172.21.15.23:6813/2949415020","heartbeat_back_addr":"172.21.15.23:6817/2949415020","heartbeat_front_addr":"172.21.15.23:6815/2949415020","state":["exists","up"]},{"osd":4,"uuid":"76a90bac-b221-40a3-8238-40e8049cec36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6809","nonce":2615902819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6811","nonce":2615902819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6815","nonce":2615902819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6813","nonce":2615902819}]},"public_addr":"172.21.15.98:6809/2615902819","cluster_addr":"172.21.15.98:6811/2615902819","heartbeat_back_addr":"172.21.15.98:6815/2615902819","heartbeat_front_addr":"172.21.15.98:6813/2615902819","state":["exists","up"]},{"osd":5,"uuid":"b53fdb55-a8df-40a8-bd1b-8bcef634e26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6809","nonce":625271528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6811","nonce":625271528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6815","nonce":625271528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6813","nonce":625271528}]},"public_addr":"172.21.15.113:6809/625271528","cluster_addr":"172.21.15.113:6811/625271528","heartbeat_back_addr":"172.21.15.113:6815/625271528","heartbeat_front_addr":"172.21.15.113:6813/625271528","state":["exists","up"]},{"osd":6,"uuid":"f00283d2-8d67-4e97-a575-9a9d3b55d5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6819","nonce":4188464176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6821","nonce":4188464176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6825","nonce":4188464176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6823","nonce":4188464176}]},"public_addr":"172.21.15.23:6819/4188464176","cluster_addr":"172.21.15.23:6821/4188464176","heartbeat_back_addr":"172.21.15.23:6825/4188464176","heartbeat_front_addr":"172.21.15.23:6823/4188464176","state":["exists","up"]},{"osd":7,"uuid":"edc008a7-b07b-40a8-88b7-1614ba38a775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6817","nonce":1985546028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6819","nonce":1985546028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6823","nonce":1985546028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6821","nonce":1985546028}]},"public_addr":"172.21.15.98:6817/1985546028","cluster_addr":"172.21.15.98:6819/1985546028","heartbeat_back_addr":"172.21.15.98:6823/1985546028","heartbeat_front_addr":"172.21.15.98:6821/1985546028","state":["exists","up"]},{"osd":8,"uuid":"56e7b2d1-81f0-47b2-8638-f29832bed8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6817","nonce":3534876081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6819","nonce":3534876081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6823","nonce":3534876081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6821","nonce":3534876081}]},"public_addr":"172.21.15.113:6817/3534876081","cluster_addr":"172.21.15.113:6819/3534876081","heartbeat_back_addr":"172.21.15.113:6823/3534876081","heartbeat_front_addr":"172.21.15.113:6821/3534876081","state":["exists","up"]},{"osd":9,"uuid":"a2e90caa-6e9c-42bf-9f18-81f7f47880cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6827","nonce":672264207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6829","nonce":672264207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6833","nonce":672264207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6831","nonce":672264207}]},"public_addr":"172.21.15.23:6827/672264207","cluster_addr":"172.21.15.23:6829/672264207","heartbeat_back_addr":"172.21.15.23:6833/672264207","heartbeat_front_addr":"172.21.15.23:6831/672264207","state":["exists","up"]},{"osd":10,"uuid":"0eda917f-020c-437f-95ae-b26203f6c68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6825","nonce":2699920595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6827","nonce":2699920595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6831","nonce":2699920595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6829","nonce":2699920595}]},"public_addr":"172.21.15.98:6825/2699920595","cluster_addr":"172.21.15.98:6827/2699920595","heartbeat_back_addr":"172.21.15.98:6831/2699920595","heartbeat_front_addr":"172.21.15.98:6829/2699920595","state":["exists","up"]},{"osd":11,"uuid":"aae78501-78a8-469a-be20-3006f5ec5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6825","nonce":2113644020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6827","nonce":2113644020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6831","nonce":2113644020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6829","nonce":2113644020}]},"public_addr":"172.21.15.113:6825/2113644020","cluster_addr":"172.21.15.113:6827/2113644020","heartbeat_back_addr":"172.21.15.113:6831/2113644020","heartbeat_front_addr":"172.21.15.113:6829/2113644020","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:15:43.527475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:07.006538+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:30.594176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:55.810999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:20.595811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:45.474138+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:10.500409+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:35.787638+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:01.001188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:26.761115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:52.952752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:20:18.559832+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:6800/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1876451381":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2253559862":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2653562762":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/3337463830":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6800/1735221490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/2784771635":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2389902490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6801/1735221490":"2024-03-29T05:13:42.579607+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T05:20:30.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.669+0000 7f1474ef1640 1 -- 172.21.15.23:0/2298645052 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f145c0a2800 msgr2=0x7f145c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:30.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.669+0000 7f1474ef1640 1 --2- 172.21.15.23:0/2298645052 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f145c0a2800 0x7f145c0a4cc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f14600104e0 tx=0x7f146000b410 comp rx=0 tx=0).stop 2024-03-28T05:20:30.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.669+0000 7f1474ef1640 1 -- 172.21.15.23:0/2298645052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14701021f0 msgr2=0x7f147019ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:30.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.669+0000 7f1474ef1640 1 --2- 172.21.15.23:0/2298645052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14701021f0 0x7f147019ecd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f146802c630 tx=0x7f1468032f20 comp rx=0 tx=0).stop 2024-03-28T05:20:30.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.669+0000 7f1474ef1640 1 -- 172.21.15.23:0/2298645052 shutdown_connections 2024-03-28T05:20:30.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.669+0000 7f1474ef1640 1 --2- 172.21.15.23:0/2298645052 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f145c0a2800 0x7f145c0a4cc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:30.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.669+0000 7f1474ef1640 1 --2- 172.21.15.23:0/2298645052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14701021f0 0x7f147019ecd0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:30.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.669+0000 7f1474ef1640 1 --2- 172.21.15.23:0/2298645052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1470101830 0x7f147019c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:30.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.669+0000 7f1474ef1640 1 --2- 172.21.15.23:0/2298645052 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1470100640 0x7f147006a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:30.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.669+0000 7f1474ef1640 1 -- 172.21.15.23:0/2298645052 >> 172.21.15.23:0/2298645052 conn(0x7f14700fbbd0 msgr2=0x7f1470104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:30.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.669+0000 7f1474ef1640 1 -- 172.21.15.23:0/2298645052 shutdown_connections 2024-03-28T05:20:30.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.669+0000 7f1474ef1640 1 -- 172.21.15.23:0/2298645052 wait complete. 2024-03-28T05:20:30.677 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:30 smithi023 ceph-mon[31954]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:30.677 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:30 smithi023 ceph-mon[31954]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi113=c;smithi113=f;smithi113=i;smithi113=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:20:30.677 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:30 smithi023 ceph-mon[31954]: Saving service mds.all spec with placement smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi113=c;smithi113=f;smithi113=i;smithi113=l;count:12 2024-03-28T05:20:30.678 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:30 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:30.678 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:30 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:20:30.678 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:30 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:30.678 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:30 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:20:30.678 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:30 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:30.678 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:30 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:30.678 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:30 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:30.678 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:30 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:30.680 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T05:16:32.986974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T05:20:30.680 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T05:20:30.738 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:30 smithi113 ceph-mon[37164]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:30.738 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:30 smithi113 ceph-mon[37164]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi113=c;smithi113=f;smithi113=i;smithi113=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:20:30.738 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:30 smithi113 ceph-mon[37164]: Saving service mds.all spec with placement smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi113=c;smithi113=f;smithi113=i;smithi113=l;count:12 2024-03-28T05:20:30.738 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:30 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:30.738 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:30 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:20:30.738 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:30 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:30.738 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:30 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:20:30.738 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:30 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:30.738 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:30 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:30.738 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:30 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:30.738 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:30 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:30.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.788+0000 7f231ff13640 1 -- 172.21.15.23:0/1820935621 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2318101790 msgr2=0x7f2318101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:30.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.788+0000 7f231ff13640 1 --2- 172.21.15.23:0/1820935621 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2318101790 0x7f2318101c10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2308009900 tx=0x7f230802bc30 comp rx=0 tx=0).stop 2024-03-28T05:20:30.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.788+0000 7f231ff13640 1 -- 172.21.15.23:0/1820935621 shutdown_connections 2024-03-28T05:20:30.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.788+0000 7f231ff13640 1 --2- 172.21.15.23:0/1820935621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2318102150 0x7f23181089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:30.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.788+0000 7f231ff13640 1 --2- 172.21.15.23:0/1820935621 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2318101790 0x7f2318101c10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:30.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.788+0000 7f231ff13640 1 --2- 172.21.15.23:0/1820935621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23181005a0 0x7f23181009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:30.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.788+0000 7f231ff13640 1 -- 172.21.15.23:0/1820935621 >> 172.21.15.23:0/1820935621 conn(0x7f23180fbb30 msgr2=0x7f23180fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:30.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.788+0000 7f231ff13640 1 -- 172.21.15.23:0/1820935621 shutdown_connections 2024-03-28T05:20:30.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.788+0000 7f231ff13640 1 -- 172.21.15.23:0/1820935621 wait complete. 2024-03-28T05:20:30.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.789+0000 7f231ff13640 1 Processor -- start 2024-03-28T05:20:30.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.789+0000 7f231ff13640 1 -- start start 2024-03-28T05:20:30.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.789+0000 7f231ff13640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23181005a0 0x7f2318069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:30.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.789+0000 7f231ff13640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2318101790 0x7f231819c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:30.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.789+0000 7f231ff13640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2318102150 0x7f231819ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:30.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.789+0000 7f231ff13640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231810c230 con 0x7f2318102150 2024-03-28T05:20:30.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.789+0000 7f231ff13640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231810c0b0 con 0x7f23181005a0 2024-03-28T05:20:30.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.789+0000 7f231ff13640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231810c3b0 con 0x7f2318101790 2024-03-28T05:20:30.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.789+0000 7f231e4b0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2318102150 0x7f231819ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:30.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.789+0000 7f231e4b0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2318102150 0x7f231819ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36546/0 (socket says 172.21.15.23:36546) 2024-03-28T05:20:30.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.789+0000 7f231e4b0640 1 -- 172.21.15.23:0/720055456 learned_addr learned my addr 172.21.15.23:0/720055456 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:20:30.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.789+0000 7f231d4ae640 1 --2- 172.21.15.23:0/720055456 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2318101790 0x7f231819c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:30.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.789+0000 7f231dcaf640 1 --2- 172.21.15.23:0/720055456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23181005a0 0x7f2318069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:30.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.789+0000 7f231e4b0640 1 -- 172.21.15.23:0/720055456 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2318101790 msgr2=0x7f231819c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:30.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.789+0000 7f231e4b0640 1 --2- 172.21.15.23:0/720055456 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2318101790 0x7f231819c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:30.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.789+0000 7f231e4b0640 1 -- 172.21.15.23:0/720055456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23181005a0 msgr2=0x7f2318069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:30.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.789+0000 7f231e4b0640 1 --2- 172.21.15.23:0/720055456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23181005a0 0x7f2318069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:30.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.789+0000 7f231e4b0640 1 -- 172.21.15.23:0/720055456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231819f640 con 0x7f2318102150 2024-03-28T05:20:30.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.789+0000 7f231d4ae640 1 --2- 172.21.15.23:0/720055456 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2318101790 0x7f231819c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:20:30.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.789+0000 7f231e4b0640 1 --2- 172.21.15.23:0/720055456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2318102150 0x7f231819ed30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f231000eb10 tx=0x7f231000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:30.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.790+0000 7f230effd640 1 -- 172.21.15.23:0/720055456 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2310004b20 con 0x7f2318102150 2024-03-28T05:20:30.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.790+0000 7f230effd640 1 -- 172.21.15.23:0/720055456 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2310004cc0 con 0x7f2318102150 2024-03-28T05:20:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.790+0000 7f231ff13640 1 -- 172.21.15.23:0/720055456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f231819f8d0 con 0x7f2318102150 2024-03-28T05:20:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.790+0000 7f230effd640 1 -- 172.21.15.23:0/720055456 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2310010760 con 0x7f2318102150 2024-03-28T05:20:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.790+0000 7f231ff13640 1 -- 172.21.15.23:0/720055456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f231819fdf0 con 0x7f2318102150 2024-03-28T05:20:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.791+0000 7f231ff13640 1 -- 172.21.15.23:0/720055456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2318107940 con 0x7f2318102150 2024-03-28T05:20:30.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.792+0000 7f230effd640 1 -- 172.21.15.23:0/720055456 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2310005020 con 0x7f2318102150 2024-03-28T05:20:30.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.792+0000 7f230effd640 1 --2- 172.21.15.23:0/720055456 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f23040a2800 0x7f23040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:30.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.792+0000 7f231dcaf640 1 --2- 172.21.15.23:0/720055456 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f23040a2800 0x7f23040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:30.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.792+0000 7f230effd640 1 -- 172.21.15.23:0/720055456 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f23100ceca0 con 0x7f2318102150 2024-03-28T05:20:30.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.793+0000 7f231dcaf640 1 --2- 172.21.15.23:0/720055456 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f23040a2800 0x7f23040a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f23181015f0 tx=0x7f231400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:30.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.797+0000 7f230effd640 1 -- 172.21.15.23:0/720055456 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f231009c1e0 con 0x7f2318102150 2024-03-28T05:20:30.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.981+0000 7f231ff13640 1 -- 172.21.15.23:0/720055456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f231802d2e0 con 0x7f2318102150 2024-03-28T05:20:30.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.981+0000 7f230effd640 1 -- 172.21.15.23:0/720055456 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f231009bae0 con 0x7f2318102150 2024-03-28T05:20:30.979 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-28T05:20:30.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.983+0000 7f231ff13640 1 -- 172.21.15.23:0/720055456 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f23040a2800 msgr2=0x7f23040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:30.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.983+0000 7f231ff13640 1 --2- 172.21.15.23:0/720055456 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f23040a2800 0x7f23040a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f23181015f0 tx=0x7f231400a2d0 comp rx=0 tx=0).stop 2024-03-28T05:20:30.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.983+0000 7f231ff13640 1 -- 172.21.15.23:0/720055456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2318102150 msgr2=0x7f231819ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:30.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.983+0000 7f231ff13640 1 --2- 172.21.15.23:0/720055456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2318102150 0x7f231819ed30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f231000eb10 tx=0x7f231000c2d0 comp rx=0 tx=0).stop 2024-03-28T05:20:30.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.983+0000 7f231ff13640 1 -- 172.21.15.23:0/720055456 shutdown_connections 2024-03-28T05:20:30.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.983+0000 7f231ff13640 1 --2- 172.21.15.23:0/720055456 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f23040a2800 0x7f23040a4cc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:30.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.983+0000 7f231ff13640 1 --2- 172.21.15.23:0/720055456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2318102150 0x7f231819ed30 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:30.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.983+0000 7f231ff13640 1 --2- 172.21.15.23:0/720055456 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2318101790 0x7f231819c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:30.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.983+0000 7f231ff13640 1 --2- 172.21.15.23:0/720055456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23181005a0 0x7f2318069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:30.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.983+0000 7f231ff13640 1 -- 172.21.15.23:0/720055456 >> 172.21.15.23:0/720055456 conn(0x7f23180fbb30 msgr2=0x7f2318104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:30.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.983+0000 7f231ff13640 1 -- 172.21.15.23:0/720055456 shutdown_connections 2024-03-28T05:20:30.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:30.983+0000 7f231ff13640 1 -- 172.21.15.23:0/720055456 wait complete. 2024-03-28T05:20:30.992 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:20:31.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.097+0000 7f2aa31d4640 1 -- 172.21.15.23:0/1092139815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a9c103280 msgr2=0x7f2a9c109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:31.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.097+0000 7f2aa31d4640 1 --2- 172.21.15.23:0/1092139815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a9c103280 0x7f2a9c109b10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c009900 tx=0x7f2a8c02bc30 comp rx=0 tx=0).stop 2024-03-28T05:20:31.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.097+0000 7f2aa31d4640 1 -- 172.21.15.23:0/1092139815 shutdown_connections 2024-03-28T05:20:31.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.097+0000 7f2aa31d4640 1 --2- 172.21.15.23:0/1092139815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a9c103280 0x7f2a9c109b10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:31.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.097+0000 7f2aa31d4640 1 --2- 172.21.15.23:0/1092139815 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a9c1028c0 0x7f2a9c102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:31.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.097+0000 7f2aa31d4640 1 --2- 172.21.15.23:0/1092139815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a9c1016d0 0x7f2a9c101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:31.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.097+0000 7f2aa31d4640 1 -- 172.21.15.23:0/1092139815 >> 172.21.15.23:0/1092139815 conn(0x7f2a9c0fcc80 msgr2=0x7f2a9c0ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:31.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.097+0000 7f2aa31d4640 1 -- 172.21.15.23:0/1092139815 shutdown_connections 2024-03-28T05:20:31.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.097+0000 7f2aa31d4640 1 -- 172.21.15.23:0/1092139815 wait complete. 2024-03-28T05:20:31.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.097+0000 7f2aa31d4640 1 Processor -- start 2024-03-28T05:20:31.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.097+0000 7f2aa31d4640 1 -- start start 2024-03-28T05:20:31.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.098+0000 7f2aa31d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a9c1016d0 0x7f2a9c197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:31.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.098+0000 7f2aa31d4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a9c1028c0 0x7f2a9c197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:31.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.098+0000 7f2aa31d4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a9c103280 0x7f2a9c19eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:31.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.098+0000 7f2aa31d4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c10d350 con 0x7f2a9c1028c0 2024-03-28T05:20:31.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.098+0000 7f2aa31d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c10d1d0 con 0x7f2a9c1016d0 2024-03-28T05:20:31.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.098+0000 7f2aa31d4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9c10d4d0 con 0x7f2a9c103280 2024-03-28T05:20:31.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.098+0000 7f2a9bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a9c1028c0 0x7f2a9c197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:31.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.098+0000 7f2a9bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a9c1028c0 0x7f2a9c197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36576/0 (socket says 172.21.15.23:36576) 2024-03-28T05:20:31.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.098+0000 7f2a9bfff640 1 -- 172.21.15.23:0/2708979872 learned_addr learned my addr 172.21.15.23:0/2708979872 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:20:31.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.098+0000 7f2aa1771640 1 --2- 172.21.15.23:0/2708979872 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a9c103280 0x7f2a9c19eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:31.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.098+0000 7f2aa0f70640 1 --2- 172.21.15.23:0/2708979872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a9c1016d0 0x7f2a9c197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:31.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.098+0000 7f2a9bfff640 1 -- 172.21.15.23:0/2708979872 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a9c103280 msgr2=0x7f2a9c19eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:31.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.098+0000 7f2a9bfff640 1 --2- 172.21.15.23:0/2708979872 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a9c103280 0x7f2a9c19eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:31.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.098+0000 7f2a9bfff640 1 -- 172.21.15.23:0/2708979872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a9c1016d0 msgr2=0x7f2a9c197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:31.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.098+0000 7f2a9bfff640 1 --2- 172.21.15.23:0/2708979872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a9c1016d0 0x7f2a9c197930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:31.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.098+0000 7f2a9bfff640 1 -- 172.21.15.23:0/2708979872 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9c19f550 con 0x7f2a9c1028c0 2024-03-28T05:20:31.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.098+0000 7f2aa1771640 1 --2- 172.21.15.23:0/2708979872 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a9c103280 0x7f2a9c19eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:20:31.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.098+0000 7f2a9bfff640 1 --2- 172.21.15.23:0/2708979872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a9c1028c0 0x7f2a9c197e70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2a8400c970 tx=0x7f2a8400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:31.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.098+0000 7f2a99ffb640 1 -- 172.21.15.23:0/2708979872 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a84007bb0 con 0x7f2a9c1028c0 2024-03-28T05:20:31.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.098+0000 7f2a99ffb640 1 -- 172.21.15.23:0/2708979872 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a84007d50 con 0x7f2a9c1028c0 2024-03-28T05:20:31.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.098+0000 7f2aa31d4640 1 -- 172.21.15.23:0/2708979872 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a9c19f840 con 0x7f2a9c1028c0 2024-03-28T05:20:31.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.098+0000 7f2a99ffb640 1 -- 172.21.15.23:0/2708979872 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8400f450 con 0x7f2a9c1028c0 2024-03-28T05:20:31.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.099+0000 7f2aa31d4640 1 -- 172.21.15.23:0/2708979872 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a9c19fd40 con 0x7f2a9c1028c0 2024-03-28T05:20:31.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.100+0000 7f2aa31d4640 1 -- 172.21.15.23:0/2708979872 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a9c108920 con 0x7f2a9c1028c0 2024-03-28T05:20:31.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.100+0000 7f2a99ffb640 1 -- 172.21.15.23:0/2708979872 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2a84005020 con 0x7f2a9c1028c0 2024-03-28T05:20:31.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.101+0000 7f2a99ffb640 1 --2- 172.21.15.23:0/2708979872 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2a880a2800 0x7f2a880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:31.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.101+0000 7f2aa0f70640 1 --2- 172.21.15.23:0/2708979872 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2a880a2800 0x7f2a880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:31.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.101+0000 7f2a99ffb640 1 -- 172.21.15.23:0/2708979872 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2a840ce390 con 0x7f2a9c1028c0 2024-03-28T05:20:31.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.102+0000 7f2aa0f70640 1 --2- 172.21.15.23:0/2708979872 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2a880a2800 0x7f2a880a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2a94005c40 tx=0x7f2a94005bd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:31.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.106+0000 7f2a99ffb640 1 -- 172.21.15.23:0/2708979872 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2a84014030 con 0x7f2a9c1028c0 2024-03-28T05:20:31.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.329+0000 7f2aa31d4640 1 -- 172.21.15.23:0/2708979872 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a9c02d070 con 0x7f2a9c1028c0 2024-03-28T05:20:31.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.329+0000 7f2a99ffb640 1 -- 172.21.15.23:0/2708979872 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7f2a8409b8d0 con 0x7f2a9c1028c0 2024-03-28T05:20:31.328 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:20:31.328 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-28T05:20:31.329 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 1 2024-03-28T05:20:31.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.331+0000 7f2aa31d4640 1 -- 172.21.15.23:0/2708979872 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2a880a2800 msgr2=0x7f2a880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:31.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.331+0000 7f2aa31d4640 1 --2- 172.21.15.23:0/2708979872 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2a880a2800 0x7f2a880a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2a94005c40 tx=0x7f2a94005bd0 comp rx=0 tx=0).stop 2024-03-28T05:20:31.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.331+0000 7f2aa31d4640 1 -- 172.21.15.23:0/2708979872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a9c1028c0 msgr2=0x7f2a9c197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:31.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.331+0000 7f2aa31d4640 1 --2- 172.21.15.23:0/2708979872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a9c1028c0 0x7f2a9c197e70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2a8400c970 tx=0x7f2a8400ce30 comp rx=0 tx=0).stop 2024-03-28T05:20:31.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.331+0000 7f2aa31d4640 1 -- 172.21.15.23:0/2708979872 shutdown_connections 2024-03-28T05:20:31.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.331+0000 7f2aa31d4640 1 --2- 172.21.15.23:0/2708979872 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2a880a2800 0x7f2a880a4cc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:31.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.331+0000 7f2aa31d4640 1 --2- 172.21.15.23:0/2708979872 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2a9c103280 0x7f2a9c19eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:31.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.331+0000 7f2aa31d4640 1 --2- 172.21.15.23:0/2708979872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a9c1028c0 0x7f2a9c197e70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:31.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.331+0000 7f2aa31d4640 1 --2- 172.21.15.23:0/2708979872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a9c1016d0 0x7f2a9c197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:31.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.331+0000 7f2aa31d4640 1 -- 172.21.15.23:0/2708979872 >> 172.21.15.23:0/2708979872 conn(0x7f2a9c0fcc80 msgr2=0x7f2a9c105af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:31.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.331+0000 7f2aa31d4640 1 -- 172.21.15.23:0/2708979872 shutdown_connections 2024-03-28T05:20:31.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:31.331+0000 7f2aa31d4640 1 -- 172.21.15.23:0/2708979872 wait complete. 2024-03-28T05:20:31.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:31 smithi098 ceph-mon[36087]: Deploying daemon mds.l on smithi113 2024-03-28T05:20:31.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:31 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2298645052' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:20:31.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:31 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/720055456' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:20:31.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:31 smithi113 ceph-mon[37164]: Deploying daemon mds.l on smithi113 2024-03-28T05:20:31.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:31 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2298645052' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:20:31.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:31 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/720055456' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:20:31.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:31 smithi023 ceph-mon[31954]: Deploying daemon mds.l on smithi113 2024-03-28T05:20:31.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:31 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2298645052' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:20:31.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:31 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/720055456' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:20:32.447 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:32 smithi113 ceph-mon[37164]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:32.447 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:32 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2708979872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:32.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:32 smithi023 ceph-mon[31954]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:32.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:32 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2708979872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:32.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:32 smithi098 ceph-mon[36087]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:32.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:32 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2708979872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:32.992 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 28 05:20:32 smithi113 systemd[1]: Started Ceph mds.l for bf37ebc4-ecc1-11ee-95d0-87774f69a715. 2024-03-28T05:20:33.341 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:20:33.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.458+0000 7fb0cb630640 1 -- 172.21.15.23:0/3040510854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0c4100640 msgr2=0x7fb0c4100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.458+0000 7fb0cb630640 1 --2- 172.21.15.23:0/3040510854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0c4100640 0x7fb0c4100a60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc009900 tx=0x7fb0bc02bc30 comp rx=0 tx=0).stop 2024-03-28T05:20:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.458+0000 7fb0cb630640 1 -- 172.21.15.23:0/3040510854 shutdown_connections 2024-03-28T05:20:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.458+0000 7fb0cb630640 1 --2- 172.21.15.23:0/3040510854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c41021f0 0x7fb0c4108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.458+0000 7fb0cb630640 1 --2- 172.21.15.23:0/3040510854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0c4101830 0x7fb0c4101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.458+0000 7fb0cb630640 1 --2- 172.21.15.23:0/3040510854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0c4100640 0x7fb0c4100a60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.458+0000 7fb0cb630640 1 -- 172.21.15.23:0/3040510854 >> 172.21.15.23:0/3040510854 conn(0x7fb0c40fbbd0 msgr2=0x7fb0c40fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.458+0000 7fb0cb630640 1 -- 172.21.15.23:0/3040510854 shutdown_connections 2024-03-28T05:20:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.458+0000 7fb0cb630640 1 -- 172.21.15.23:0/3040510854 wait complete. 2024-03-28T05:20:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.459+0000 7fb0cb630640 1 Processor -- start 2024-03-28T05:20:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.459+0000 7fb0cb630640 1 -- start start 2024-03-28T05:20:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.459+0000 7fb0cb630640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c4100640 0x7fb0c41969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.459+0000 7fb0cb630640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0c4101830 0x7fb0c4196f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.459+0000 7fb0cb630640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0c41021f0 0x7fb0c419df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.459+0000 7fb0cb630640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c410c2c0 con 0x7fb0c41021f0 2024-03-28T05:20:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.459+0000 7fb0cb630640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c410c140 con 0x7fb0c4100640 2024-03-28T05:20:33.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.459+0000 7fb0cb630640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c410c440 con 0x7fb0c4101830 2024-03-28T05:20:33.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.459+0000 7fb0c9bcd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0c41021f0 0x7fb0c419df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:33.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.459+0000 7fb0c9bcd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0c41021f0 0x7fb0c419df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40066/0 (socket says 172.21.15.23:40066) 2024-03-28T05:20:33.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.459+0000 7fb0c9bcd640 1 -- 172.21.15.23:0/970113596 learned_addr learned my addr 172.21.15.23:0/970113596 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:20:33.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.459+0000 7fb0c8bcb640 1 --2- 172.21.15.23:0/970113596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0c4101830 0x7fb0c4196f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:33.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.459+0000 7fb0c93cc640 1 --2- 172.21.15.23:0/970113596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c4100640 0x7fb0c41969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:33.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.459+0000 7fb0c9bcd640 1 -- 172.21.15.23:0/970113596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0c4101830 msgr2=0x7fb0c4196f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:33.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.459+0000 7fb0c9bcd640 1 --2- 172.21.15.23:0/970113596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0c4101830 0x7fb0c4196f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:33.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.459+0000 7fb0c9bcd640 1 -- 172.21.15.23:0/970113596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c4100640 msgr2=0x7fb0c41969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:33.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.459+0000 7fb0c9bcd640 1 --2- 172.21.15.23:0/970113596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c4100640 0x7fb0c41969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:33.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.459+0000 7fb0c9bcd640 1 -- 172.21.15.23:0/970113596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c4069cd0 con 0x7fb0c41021f0 2024-03-28T05:20:33.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.460+0000 7fb0c9bcd640 1 --2- 172.21.15.23:0/970113596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0c41021f0 0x7fb0c419df00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb0c000b970 tx=0x7fb0c000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:33.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.460+0000 7fb0ba7fc640 1 -- 172.21.15.23:0/970113596 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0019070 con 0x7fb0c41021f0 2024-03-28T05:20:33.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.460+0000 7fb0ba7fc640 1 -- 172.21.15.23:0/970113596 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb0c000cb40 con 0x7fb0c41021f0 2024-03-28T05:20:33.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.460+0000 7fb0cb630640 1 -- 172.21.15.23:0/970113596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c4069fc0 con 0x7fb0c41021f0 2024-03-28T05:20:33.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.460+0000 7fb0ba7fc640 1 -- 172.21.15.23:0/970113596 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0007650 con 0x7fb0c41021f0 2024-03-28T05:20:33.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.460+0000 7fb0cb630640 1 -- 172.21.15.23:0/970113596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c406a510 con 0x7fb0c41021f0 2024-03-28T05:20:33.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.461+0000 7fb0cb630640 1 -- 172.21.15.23:0/970113596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c4107960 con 0x7fb0c41021f0 2024-03-28T05:20:33.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.462+0000 7fb0ba7fc640 1 -- 172.21.15.23:0/970113596 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb0c001c070 con 0x7fb0c41021f0 2024-03-28T05:20:33.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.462+0000 7fb0ba7fc640 1 --2- 172.21.15.23:0/970113596 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb0b00a2880 0x7fb0b00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:33.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.463+0000 7fb0c93cc640 1 --2- 172.21.15.23:0/970113596 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb0b00a2880 0x7fb0b00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:33.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.463+0000 7fb0ba7fc640 1 -- 172.21.15.23:0/970113596 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb0c00cdfa0 con 0x7fb0c41021f0 2024-03-28T05:20:33.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.463+0000 7fb0c93cc640 1 --2- 172.21.15.23:0/970113596 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb0b00a2880 0x7fb0b00a4d40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc000c00 tx=0x7fb0bc005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:33.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.466+0000 7fb0ba7fc640 1 -- 172.21.15.23:0/970113596 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb0c009ae90 con 0x7fb0c41021f0 2024-03-28T05:20:33.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.691+0000 7fb0cb630640 1 -- 172.21.15.23:0/970113596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0c402d070 con 0x7fb0c41021f0 2024-03-28T05:20:33.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.692+0000 7fb0ba7fc640 1 -- 172.21.15.23:0/970113596 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7fb0c009ef30 con 0x7fb0c41021f0 2024-03-28T05:20:33.690 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:20:33.690 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T05:20:33.690 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 2 2024-03-28T05:20:33.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.694+0000 7fb0cb630640 1 -- 172.21.15.23:0/970113596 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb0b00a2880 msgr2=0x7fb0b00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:33.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.694+0000 7fb0cb630640 1 --2- 172.21.15.23:0/970113596 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb0b00a2880 0x7fb0b00a4d40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc000c00 tx=0x7fb0bc005fd0 comp rx=0 tx=0).stop 2024-03-28T05:20:33.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.694+0000 7fb0cb630640 1 -- 172.21.15.23:0/970113596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0c41021f0 msgr2=0x7fb0c419df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:33.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.694+0000 7fb0cb630640 1 --2- 172.21.15.23:0/970113596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0c41021f0 0x7fb0c419df00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb0c000b970 tx=0x7fb0c000be30 comp rx=0 tx=0).stop 2024-03-28T05:20:33.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.694+0000 7fb0cb630640 1 -- 172.21.15.23:0/970113596 shutdown_connections 2024-03-28T05:20:33.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.694+0000 7fb0cb630640 1 --2- 172.21.15.23:0/970113596 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb0b00a2880 0x7fb0b00a4d40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:33.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.694+0000 7fb0cb630640 1 --2- 172.21.15.23:0/970113596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0c41021f0 0x7fb0c419df00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:33.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.694+0000 7fb0cb630640 1 --2- 172.21.15.23:0/970113596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0c4101830 0x7fb0c4196f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:33.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.694+0000 7fb0cb630640 1 --2- 172.21.15.23:0/970113596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c4100640 0x7fb0c41969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:33.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.694+0000 7fb0cb630640 1 -- 172.21.15.23:0/970113596 >> 172.21.15.23:0/970113596 conn(0x7fb0c40fbbd0 msgr2=0x7fb0c4104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:33.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.694+0000 7fb0cb630640 1 -- 172.21.15.23:0/970113596 shutdown_connections 2024-03-28T05:20:33.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:33.694+0000 7fb0cb630640 1 -- 172.21.15.23:0/970113596 wait complete. 2024-03-28T05:20:33.841 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:33 smithi098 ceph-mon[36087]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:33.841 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:33 smithi098 ceph-mon[36087]: mds.? [v2:172.21.15.113:6832/3436936808,v1:172.21.15.113:6833/3436936808] up:boot 2024-03-28T05:20:33.841 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:33 smithi098 ceph-mon[36087]: fsmap 1 up:standby 2024-03-28T05:20:33.841 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:33 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T05:20:33.841 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:33 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:33.842 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:33 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:33.842 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:33 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:33.842 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:33 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:33.842 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:33 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:33.842 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:33 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:33.842 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:33 smithi098 ceph-mon[36087]: Deploying daemon mds.k on smithi098 2024-03-28T05:20:33.842 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:33 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/970113596' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:34.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:33 smithi023 ceph-mon[31954]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:34.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:33 smithi023 ceph-mon[31954]: mds.? [v2:172.21.15.113:6832/3436936808,v1:172.21.15.113:6833/3436936808] up:boot 2024-03-28T05:20:34.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:33 smithi023 ceph-mon[31954]: fsmap 1 up:standby 2024-03-28T05:20:34.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:33 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T05:20:34.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:33 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:34.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:33 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:34.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:33 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:34.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:33 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:34.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:33 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:34.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:33 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:34.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:33 smithi023 ceph-mon[31954]: Deploying daemon mds.k on smithi098 2024-03-28T05:20:34.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:33 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/970113596' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:34.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:33 smithi113 ceph-mon[37164]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:34.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:33 smithi113 ceph-mon[37164]: mds.? [v2:172.21.15.113:6832/3436936808,v1:172.21.15.113:6833/3436936808] up:boot 2024-03-28T05:20:34.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:33 smithi113 ceph-mon[37164]: fsmap 1 up:standby 2024-03-28T05:20:34.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:33 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T05:20:34.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:33 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:34.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:33 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:34.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:33 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:34.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:33 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:34.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:33 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:34.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:33 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:34.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:33 smithi113 ceph-mon[37164]: Deploying daemon mds.k on smithi098 2024-03-28T05:20:34.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:33 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/970113596' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:35.365 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 28 05:20:35 smithi098 systemd[1]: Starting Ceph mds.k for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:20:35.694 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 28 05:20:35 smithi098 podman[64863]: 2024-03-28 05:20:35.336236033 +0000 UTC m=+0.017674806 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:20:35.694 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 28 05:20:35 smithi098 podman[64863]: 2024-03-28 05:20:35.443337144 +0000 UTC m=+0.124775894 container create 925761f3a98891341cbdd00d41be8fd849eb80b7e0b2dac70b5095f9e10b1464 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-k, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T05:20:35.694 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 28 05:20:35 smithi098 podman[64863]: 2024-03-28 05:20:35.670923405 +0000 UTC m=+0.352362169 container init 925761f3a98891341cbdd00d41be8fd849eb80b7e0b2dac70b5095f9e10b1464 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-k, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T05:20:35.694 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 28 05:20:35 smithi098 podman[64863]: 2024-03-28 05:20:35.673432281 +0000 UTC m=+0.354871055 container start 925761f3a98891341cbdd00d41be8fd849eb80b7e0b2dac70b5095f9e10b1464 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-k, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T05:20:35.703 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:20:35.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.822+0000 7f12c8e4b640 1 -- 172.21.15.23:0/943659549 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12c41005c0 msgr2=0x7f12c41009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:35.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.822+0000 7f12c8e4b640 1 --2- 172.21.15.23:0/943659549 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12c41005c0 0x7f12c41009e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009900 tx=0x7f12b402bc30 comp rx=0 tx=0).stop 2024-03-28T05:20:35.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.822+0000 7f12c8e4b640 1 -- 172.21.15.23:0/943659549 shutdown_connections 2024-03-28T05:20:35.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.822+0000 7f12c8e4b640 1 --2- 172.21.15.23:0/943659549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12c4102170 0x7f12c4108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:35.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.822+0000 7f12c8e4b640 1 --2- 172.21.15.23:0/943659549 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12c41017b0 0x7f12c4101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:35.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.822+0000 7f12c8e4b640 1 --2- 172.21.15.23:0/943659549 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12c41005c0 0x7f12c41009e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:35.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.822+0000 7f12c8e4b640 1 -- 172.21.15.23:0/943659549 >> 172.21.15.23:0/943659549 conn(0x7f12c40fbb30 msgr2=0x7f12c40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:35.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.822+0000 7f12c8e4b640 1 -- 172.21.15.23:0/943659549 shutdown_connections 2024-03-28T05:20:35.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.822+0000 7f12c8e4b640 1 -- 172.21.15.23:0/943659549 wait complete. 2024-03-28T05:20:35.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.822+0000 7f12c8e4b640 1 Processor -- start 2024-03-28T05:20:35.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.823+0000 7f12c8e4b640 1 -- start start 2024-03-28T05:20:35.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.823+0000 7f12c8e4b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12c41005c0 0x7f12c4196900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:35.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.823+0000 7f12c8e4b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12c41017b0 0x7f12c4196e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:35.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.823+0000 7f12c8e4b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12c4102170 0x7f12c419deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:35.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.823+0000 7f12c8e4b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c410c290 con 0x7f12c41005c0 2024-03-28T05:20:35.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.823+0000 7f12c8e4b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c410c110 con 0x7f12c41017b0 2024-03-28T05:20:35.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.823+0000 7f12c8e4b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c410c410 con 0x7f12c4102170 2024-03-28T05:20:35.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.823+0000 7f12c259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12c41005c0 0x7f12c4196900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:35.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.823+0000 7f12c259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12c41005c0 0x7f12c4196900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40084/0 (socket says 172.21.15.23:40084) 2024-03-28T05:20:35.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.823+0000 7f12c259c640 1 -- 172.21.15.23:0/66427714 learned_addr learned my addr 172.21.15.23:0/66427714 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:20:35.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.823+0000 7f12c1d9b640 1 --2- 172.21.15.23:0/66427714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12c41017b0 0x7f12c4196e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:35.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.823+0000 7f12c2d9d640 1 --2- 172.21.15.23:0/66427714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12c4102170 0x7f12c419deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:35.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.823+0000 7f12c259c640 1 -- 172.21.15.23:0/66427714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12c4102170 msgr2=0x7f12c419deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:35.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.823+0000 7f12c259c640 1 --2- 172.21.15.23:0/66427714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12c4102170 0x7f12c419deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:35.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.823+0000 7f12c259c640 1 -- 172.21.15.23:0/66427714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12c41017b0 msgr2=0x7f12c4196e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:35.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.823+0000 7f12c259c640 1 --2- 172.21.15.23:0/66427714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12c41017b0 0x7f12c4196e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:35.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.823+0000 7f12c259c640 1 -- 172.21.15.23:0/66427714 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c419e520 con 0x7f12c41005c0 2024-03-28T05:20:35.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.823+0000 7f12c2d9d640 1 --2- 172.21.15.23:0/66427714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12c4102170 0x7f12c419deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:20:35.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.823+0000 7f12c1d9b640 1 --2- 172.21.15.23:0/66427714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12c41017b0 0x7f12c4196e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:20:35.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.823+0000 7f12c259c640 1 --2- 172.21.15.23:0/66427714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12c41005c0 0x7f12c4196900 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f12b4000c00 tx=0x7f12b402c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:35.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.823+0000 7f12b37fe640 1 -- 172.21.15.23:0/66427714 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b4043020 con 0x7f12c41005c0 2024-03-28T05:20:35.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.824+0000 7f12b37fe640 1 -- 172.21.15.23:0/66427714 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12b40054e0 con 0x7f12c41005c0 2024-03-28T05:20:35.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.824+0000 7f12b37fe640 1 -- 172.21.15.23:0/66427714 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b4003ce0 con 0x7f12c41005c0 2024-03-28T05:20:35.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.824+0000 7f12c8e4b640 1 -- 172.21.15.23:0/66427714 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c419e7b0 con 0x7f12c41005c0 2024-03-28T05:20:35.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.824+0000 7f12c8e4b640 1 -- 172.21.15.23:0/66427714 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c419ec80 con 0x7f12c41005c0 2024-03-28T05:20:35.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.825+0000 7f12c8e4b640 1 -- 172.21.15.23:0/66427714 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c41078e0 con 0x7f12c41005c0 2024-03-28T05:20:35.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.826+0000 7f12b37fe640 1 -- 172.21.15.23:0/66427714 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f12b4046020 con 0x7f12c41005c0 2024-03-28T05:20:35.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.826+0000 7f12b37fe640 1 --2- 172.21.15.23:0/66427714 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f12ac0a2880 0x7f12ac0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:35.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.826+0000 7f12c1d9b640 1 --2- 172.21.15.23:0/66427714 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f12ac0a2880 0x7f12ac0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:35.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.826+0000 7f12b37fe640 1 -- 172.21.15.23:0/66427714 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f12b40f0a70 con 0x7f12c41005c0 2024-03-28T05:20:35.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.827+0000 7f12c1d9b640 1 --2- 172.21.15.23:0/66427714 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f12ac0a2880 0x7f12ac0a4d40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f12b8009fd0 tx=0x7f12b8009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:35.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:35.830+0000 7f12b37fe640 1 -- 172.21.15.23:0/66427714 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f12b404d020 con 0x7f12c41005c0 2024-03-28T05:20:36.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:35 smithi023 ceph-mon[31954]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:36.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:36.063+0000 7f12c8e4b640 1 -- 172.21.15.23:0/66427714 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12c402d070 con 0x7f12c41005c0 2024-03-28T05:20:36.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:36.063+0000 7f12b37fe640 1 -- 172.21.15.23:0/66427714 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2119 (secure 0 0 0) 0x7f12b40bdfb0 con 0x7f12c41005c0 2024-03-28T05:20:36.061 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:20:36.062 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T05:20:36.062 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 3 2024-03-28T05:20:36.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:36.065+0000 7f12c8e4b640 1 -- 172.21.15.23:0/66427714 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f12ac0a2880 msgr2=0x7f12ac0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:36.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:36.065+0000 7f12c8e4b640 1 --2- 172.21.15.23:0/66427714 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f12ac0a2880 0x7f12ac0a4d40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f12b8009fd0 tx=0x7f12b8009680 comp rx=0 tx=0).stop 2024-03-28T05:20:36.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:36.065+0000 7f12c8e4b640 1 -- 172.21.15.23:0/66427714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12c41005c0 msgr2=0x7f12c4196900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:36.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:36.065+0000 7f12c8e4b640 1 --2- 172.21.15.23:0/66427714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12c41005c0 0x7f12c4196900 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f12b4000c00 tx=0x7f12b402c6c0 comp rx=0 tx=0).stop 2024-03-28T05:20:36.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:36.065+0000 7f12c8e4b640 1 -- 172.21.15.23:0/66427714 shutdown_connections 2024-03-28T05:20:36.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:36.065+0000 7f12c8e4b640 1 --2- 172.21.15.23:0/66427714 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f12ac0a2880 0x7f12ac0a4d40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:36.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:36.065+0000 7f12c8e4b640 1 --2- 172.21.15.23:0/66427714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12c4102170 0x7f12c419deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:36.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:36.065+0000 7f12c8e4b640 1 --2- 172.21.15.23:0/66427714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12c41017b0 0x7f12c4196e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:36.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:36.065+0000 7f12c8e4b640 1 --2- 172.21.15.23:0/66427714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12c41005c0 0x7f12c4196900 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:36.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:36.065+0000 7f12c8e4b640 1 -- 172.21.15.23:0/66427714 >> 172.21.15.23:0/66427714 conn(0x7f12c40fbb30 msgr2=0x7f12c4104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:36.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:36.065+0000 7f12c8e4b640 1 -- 172.21.15.23:0/66427714 shutdown_connections 2024-03-28T05:20:36.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:36.065+0000 7f12c8e4b640 1 -- 172.21.15.23:0/66427714 wait complete. 2024-03-28T05:20:36.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:35 smithi098 ceph-mon[36087]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:36.115 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 28 05:20:35 smithi098 ceph-mds[64882]: -- 172.21.15.98:0/2580626630 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a52af7fa40 con 0x55a52af53000 2024-03-28T05:20:36.115 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 28 05:20:35 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-k[64878]: starting mds.k at 2024-03-28T05:20:36.115 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 28 05:20:35 smithi098 bash[64863]: 925761f3a98891341cbdd00d41be8fd849eb80b7e0b2dac70b5095f9e10b1464 2024-03-28T05:20:36.115 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 28 05:20:35 smithi098 systemd[1]: Started Ceph mds.k for bf37ebc4-ecc1-11ee-95d0-87774f69a715. 2024-03-28T05:20:36.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:35 smithi113 ceph-mon[37164]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:37.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:36 smithi023 ceph-mon[31954]: mds.? [v2:172.21.15.98:6832/1563379945,v1:172.21.15.98:6833/1563379945] up:boot 2024-03-28T05:20:37.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:36 smithi023 ceph-mon[31954]: fsmap 2 up:standby 2024-03-28T05:20:37.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:36 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T05:20:37.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:36 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:37.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:36 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:37.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:36 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:37.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:36 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:37.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:36 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:37.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:36 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:37.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:36 smithi023 ceph-mon[31954]: Deploying daemon mds.j on smithi023 2024-03-28T05:20:37.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:36 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/66427714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:37.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:36 smithi098 ceph-mon[36087]: mds.? [v2:172.21.15.98:6832/1563379945,v1:172.21.15.98:6833/1563379945] up:boot 2024-03-28T05:20:37.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:36 smithi098 ceph-mon[36087]: fsmap 2 up:standby 2024-03-28T05:20:37.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:36 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T05:20:37.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:36 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:37.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:36 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:37.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:36 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:37.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:36 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:37.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:36 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:37.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:36 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:37.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:36 smithi098 ceph-mon[36087]: Deploying daemon mds.j on smithi023 2024-03-28T05:20:37.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:36 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/66427714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:37.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:36 smithi113 ceph-mon[37164]: mds.? [v2:172.21.15.98:6832/1563379945,v1:172.21.15.98:6833/1563379945] up:boot 2024-03-28T05:20:37.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:36 smithi113 ceph-mon[37164]: fsmap 2 up:standby 2024-03-28T05:20:37.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:36 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T05:20:37.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:36 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:37.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:36 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:37.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:36 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:37.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:36 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:37.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:36 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:37.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:36 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:37.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:36 smithi113 ceph-mon[37164]: Deploying daemon mds.j on smithi023 2024-03-28T05:20:37.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:36 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/66427714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:38.075 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:20:38.102 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:37 smithi023 ceph-mon[31954]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:38.103 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 05:20:37 smithi023 systemd[1]: Starting Ceph mds.j for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:20:38.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.185+0000 7ff213937640 1 -- 172.21.15.23:0/3547253010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff20c104ad0 msgr2=0x7ff20c106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:38.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.185+0000 7ff213937640 1 --2- 172.21.15.23:0/3547253010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff20c104ad0 0x7ff20c106ec0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff200009900 tx=0x7ff20002bc30 comp rx=0 tx=0).stop 2024-03-28T05:20:38.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.185+0000 7ff213937640 1 -- 172.21.15.23:0/3547253010 shutdown_connections 2024-03-28T05:20:38.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.185+0000 7ff213937640 1 --2- 172.21.15.23:0/3547253010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff20c107400 0x7ff20c1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:38.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.185+0000 7ff213937640 1 --2- 172.21.15.23:0/3547253010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff20c104ad0 0x7ff20c106ec0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:38.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.185+0000 7ff213937640 1 --2- 172.21.15.23:0/3547253010 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff20c1021a0 0x7ff20c104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:38.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.185+0000 7ff213937640 1 -- 172.21.15.23:0/3547253010 >> 172.21.15.23:0/3547253010 conn(0x7ff20c0fbb30 msgr2=0x7ff20c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:38.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.185+0000 7ff213937640 1 -- 172.21.15.23:0/3547253010 shutdown_connections 2024-03-28T05:20:38.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.185+0000 7ff213937640 1 -- 172.21.15.23:0/3547253010 wait complete. 2024-03-28T05:20:38.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.186+0000 7ff213937640 1 Processor -- start 2024-03-28T05:20:38.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.186+0000 7ff213937640 1 -- start start 2024-03-28T05:20:38.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.186+0000 7ff213937640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff20c1021a0 0x7ff20c1979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:38.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.186+0000 7ff213937640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff20c104ad0 0x7ff20c197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:38.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.186+0000 7ff213937640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff20c107400 0x7ff20c19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:38.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.186+0000 7ff213937640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c10d330 con 0x7ff20c1021a0 2024-03-28T05:20:38.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.186+0000 7ff213937640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c10d1b0 con 0x7ff20c107400 2024-03-28T05:20:38.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.186+0000 7ff213937640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c10d4b0 con 0x7ff20c104ad0 2024-03-28T05:20:38.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.186+0000 7ff2116d3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff20c1021a0 0x7ff20c1979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:38.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.186+0000 7ff2116d3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff20c1021a0 0x7ff20c1979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40102/0 (socket says 172.21.15.23:40102) 2024-03-28T05:20:38.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.186+0000 7ff2116d3640 1 -- 172.21.15.23:0/36809542 learned_addr learned my addr 172.21.15.23:0/36809542 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:20:38.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.186+0000 7ff210ed2640 1 --2- 172.21.15.23:0/36809542 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff20c104ad0 0x7ff20c197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:38.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.186+0000 7ff211ed4640 1 --2- 172.21.15.23:0/36809542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff20c107400 0x7ff20c19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:38.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.187+0000 7ff2116d3640 1 -- 172.21.15.23:0/36809542 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff20c104ad0 msgr2=0x7ff20c197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:38.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.187+0000 7ff2116d3640 1 --2- 172.21.15.23:0/36809542 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff20c104ad0 0x7ff20c197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:38.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.187+0000 7ff2116d3640 1 -- 172.21.15.23:0/36809542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff20c107400 msgr2=0x7ff20c19ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:38.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.187+0000 7ff2116d3640 1 --2- 172.21.15.23:0/36809542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff20c107400 0x7ff20c19ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:38.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.187+0000 7ff2116d3640 1 -- 172.21.15.23:0/36809542 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20c19f5f0 con 0x7ff20c1021a0 2024-03-28T05:20:38.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.187+0000 7ff2116d3640 1 --2- 172.21.15.23:0/36809542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff20c1021a0 0x7ff20c1979d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff20800c970 tx=0x7ff20800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:38.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.187+0000 7ff1fe7fc640 1 -- 172.21.15.23:0/36809542 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff208007bb0 con 0x7ff20c1021a0 2024-03-28T05:20:38.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.187+0000 7ff213937640 1 -- 172.21.15.23:0/36809542 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff20c19f8e0 con 0x7ff20c1021a0 2024-03-28T05:20:38.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.187+0000 7ff1fe7fc640 1 -- 172.21.15.23:0/36809542 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff208007d50 con 0x7ff20c1021a0 2024-03-28T05:20:38.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.187+0000 7ff1fe7fc640 1 -- 172.21.15.23:0/36809542 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20800f450 con 0x7ff20c1021a0 2024-03-28T05:20:38.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.187+0000 7ff213937640 1 -- 172.21.15.23:0/36809542 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20c19fe00 con 0x7ff20c1021a0 2024-03-28T05:20:38.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.188+0000 7ff213937640 1 -- 172.21.15.23:0/36809542 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff20c0fd620 con 0x7ff20c1021a0 2024-03-28T05:20:38.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.189+0000 7ff1fe7fc640 1 -- 172.21.15.23:0/36809542 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff208005020 con 0x7ff20c1021a0 2024-03-28T05:20:38.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.190+0000 7ff1fe7fc640 1 --2- 172.21.15.23:0/36809542 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff1f80a2800 0x7ff1f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:38.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.190+0000 7ff210ed2640 1 --2- 172.21.15.23:0/36809542 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff1f80a2800 0x7ff1f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:38.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.190+0000 7ff1fe7fc640 1 -- 172.21.15.23:0/36809542 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff2080ce440 con 0x7ff20c1021a0 2024-03-28T05:20:38.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.190+0000 7ff210ed2640 1 --2- 172.21.15.23:0/36809542 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff1f80a2800 0x7ff1f80a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff20002c7b0 tx=0x7ff200000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:38.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.195+0000 7ff1fe7fc640 1 -- 172.21.15.23:0/36809542 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff208014030 con 0x7ff20c1021a0 2024-03-28T05:20:38.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:37 smithi113 ceph-mon[37164]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:38.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:37 smithi098 ceph-mon[36087]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:38.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.417+0000 7ff213937640 1 -- 172.21.15.23:0/36809542 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff20c02d270 con 0x7ff20c1021a0 2024-03-28T05:20:38.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.417+0000 7ff1fe7fc640 1 -- 172.21.15.23:0/36809542 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2119 (secure 0 0 0) 0x7ff20809b980 con 0x7ff20c1021a0 2024-03-28T05:20:38.416 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:20:38.416 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T05:20:38.416 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 3 2024-03-28T05:20:38.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.419+0000 7ff213937640 1 -- 172.21.15.23:0/36809542 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff1f80a2800 msgr2=0x7ff1f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:38.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.419+0000 7ff213937640 1 --2- 172.21.15.23:0/36809542 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff1f80a2800 0x7ff1f80a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff20002c7b0 tx=0x7ff200000bc0 comp rx=0 tx=0).stop 2024-03-28T05:20:38.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.419+0000 7ff213937640 1 -- 172.21.15.23:0/36809542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff20c1021a0 msgr2=0x7ff20c1979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:38.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.419+0000 7ff213937640 1 --2- 172.21.15.23:0/36809542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff20c1021a0 0x7ff20c1979d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff20800c970 tx=0x7ff20800ce30 comp rx=0 tx=0).stop 2024-03-28T05:20:38.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.419+0000 7ff213937640 1 -- 172.21.15.23:0/36809542 shutdown_connections 2024-03-28T05:20:38.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.419+0000 7ff213937640 1 --2- 172.21.15.23:0/36809542 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff1f80a2800 0x7ff1f80a4cc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:38.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.419+0000 7ff213937640 1 --2- 172.21.15.23:0/36809542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff20c107400 0x7ff20c19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:38.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.419+0000 7ff213937640 1 --2- 172.21.15.23:0/36809542 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff20c104ad0 0x7ff20c197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:38.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.419+0000 7ff213937640 1 --2- 172.21.15.23:0/36809542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff20c1021a0 0x7ff20c1979d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:38.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.419+0000 7ff213937640 1 -- 172.21.15.23:0/36809542 >> 172.21.15.23:0/36809542 conn(0x7ff20c0fbb30 msgr2=0x7ff20c102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:38.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.420+0000 7ff213937640 1 -- 172.21.15.23:0/36809542 shutdown_connections 2024-03-28T05:20:38.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:38.420+0000 7ff213937640 1 -- 172.21.15.23:0/36809542 wait complete. 2024-03-28T05:20:38.427 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 05:20:38 smithi023 podman[72511]: 2024-03-28 05:20:38.08636948 +0000 UTC m=+0.018326155 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:20:38.427 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 05:20:38 smithi023 podman[72511]: 2024-03-28 05:20:38.185065696 +0000 UTC m=+0.117022373 container create 13a59a33c52c1832caed2960329f25071c9aae086672c1f1192e58189f84cf5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-j, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T05:20:38.427 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 05:20:38 smithi023 podman[72511]: 2024-03-28 05:20:38.429783192 +0000 UTC m=+0.361739878 container init 13a59a33c52c1832caed2960329f25071c9aae086672c1f1192e58189f84cf5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-j, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-28T05:20:38.780 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 05:20:38 smithi023 podman[72511]: 2024-03-28 05:20:38.43286698 +0000 UTC m=+0.364823662 container start 13a59a33c52c1832caed2960329f25071c9aae086672c1f1192e58189f84cf5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-j, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T05:20:38.780 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 05:20:38 smithi023 ceph-mds[72571]: -- 172.21.15.23:0/2480889327 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f135eb5a40 con 0x55f135e88800 2024-03-28T05:20:38.781 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 05:20:38 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-j[72565]: starting mds.j at 2024-03-28T05:20:38.781 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 05:20:38 smithi023 bash[72511]: 13a59a33c52c1832caed2960329f25071c9aae086672c1f1192e58189f84cf5b 2024-03-28T05:20:38.781 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 05:20:38 smithi023 systemd[1]: Started Ceph mds.j for bf37ebc4-ecc1-11ee-95d0-87774f69a715. 2024-03-28T05:20:39.237 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:38 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/36809542' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:39.237 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:38 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:39.237 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:38 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:39.238 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:38 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:39.238 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:38 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:39.238 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:38 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:39.238 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:38 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:39.238 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:38 smithi023 ceph-mon[31954]: Deploying daemon mds.d on smithi023 2024-03-28T05:20:39.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:38 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/36809542' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:39.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:38 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:39.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:38 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:39.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:38 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:39.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:38 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:39.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:38 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:39.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:38 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:39.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:38 smithi113 ceph-mon[37164]: Deploying daemon mds.d on smithi023 2024-03-28T05:20:39.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:38 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/36809542' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:39.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:38 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:39.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:38 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:39.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:38 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:39.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:38 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:39.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:38 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:39.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:38 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:39.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:38 smithi098 ceph-mon[36087]: Deploying daemon mds.d on smithi023 2024-03-28T05:20:40.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:39 smithi113 ceph-mon[37164]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:40.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:39 smithi113 ceph-mon[37164]: mds.? [v2:172.21.15.23:6834/733913400,v1:172.21.15.23:6835/733913400] up:boot 2024-03-28T05:20:40.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:39 smithi113 ceph-mon[37164]: fsmap 3 up:standby 2024-03-28T05:20:40.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:39 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T05:20:40.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:39 smithi023 ceph-mon[31954]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:40.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:39 smithi023 ceph-mon[31954]: mds.? [v2:172.21.15.23:6834/733913400,v1:172.21.15.23:6835/733913400] up:boot 2024-03-28T05:20:40.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:39 smithi023 ceph-mon[31954]: fsmap 3 up:standby 2024-03-28T05:20:40.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:39 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T05:20:40.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:39 smithi098 ceph-mon[36087]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:40.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:39 smithi098 ceph-mon[36087]: mds.? [v2:172.21.15.23:6834/733913400,v1:172.21.15.23:6835/733913400] up:boot 2024-03-28T05:20:40.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:39 smithi098 ceph-mon[36087]: fsmap 3 up:standby 2024-03-28T05:20:40.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:39 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T05:20:40.429 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:20:40.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.668+0000 7f4677827640 1 -- 172.21.15.23:0/3887956640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4670102170 msgr2=0x7f4670108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:40.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.668+0000 7f4677827640 1 --2- 172.21.15.23:0/3887956640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4670102170 0x7f4670108a00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4668009ca0 tx=0x7f466802c0f0 comp rx=0 tx=0).stop 2024-03-28T05:20:40.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.668+0000 7f4677827640 1 -- 172.21.15.23:0/3887956640 shutdown_connections 2024-03-28T05:20:40.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.668+0000 7f4677827640 1 --2- 172.21.15.23:0/3887956640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4670102170 0x7f4670108a00 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:40.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.668+0000 7f4677827640 1 --2- 172.21.15.23:0/3887956640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46701017b0 0x7f4670101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:40.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.668+0000 7f4677827640 1 --2- 172.21.15.23:0/3887956640 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46701005c0 0x7f46701009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:40.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.668+0000 7f4677827640 1 -- 172.21.15.23:0/3887956640 >> 172.21.15.23:0/3887956640 conn(0x7f46700fbb30 msgr2=0x7f46700fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:40.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.668+0000 7f4677827640 1 -- 172.21.15.23:0/3887956640 shutdown_connections 2024-03-28T05:20:40.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.668+0000 7f4677827640 1 -- 172.21.15.23:0/3887956640 wait complete. 2024-03-28T05:20:40.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.669+0000 7f4677827640 1 Processor -- start 2024-03-28T05:20:40.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.669+0000 7f4677827640 1 -- start start 2024-03-28T05:20:40.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.669+0000 7f4677827640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46701005c0 0x7f4670196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:40.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.669+0000 7f4677827640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46701017b0 0x7f4670196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:40.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.670+0000 7f4677827640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4670102170 0x7f467019de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:40.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.670+0000 7f4677827640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467010c220 con 0x7f46701005c0 2024-03-28T05:20:40.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.670+0000 7f4677827640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467010c0a0 con 0x7f4670102170 2024-03-28T05:20:40.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.670+0000 7f4677827640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467010c3a0 con 0x7f46701017b0 2024-03-28T05:20:40.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.670+0000 7f46755c3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46701005c0 0x7f4670196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:40.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.670+0000 7f46755c3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46701005c0 0x7f4670196860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40128/0 (socket says 172.21.15.23:40128) 2024-03-28T05:20:40.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.670+0000 7f46755c3640 1 -- 172.21.15.23:0/673105089 learned_addr learned my addr 172.21.15.23:0/673105089 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:20:40.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.670+0000 7f4674dc2640 1 --2- 172.21.15.23:0/673105089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46701017b0 0x7f4670196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:40.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.670+0000 7f46755c3640 1 -- 172.21.15.23:0/673105089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46701017b0 msgr2=0x7f4670196da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:40.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.670+0000 7f4675dc4640 1 --2- 172.21.15.23:0/673105089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4670102170 0x7f467019de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:40.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.670+0000 7f46755c3640 1 --2- 172.21.15.23:0/673105089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46701017b0 0x7f4670196da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:40.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.670+0000 7f46755c3640 1 -- 172.21.15.23:0/673105089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4670102170 msgr2=0x7f467019de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:40.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.670+0000 7f46755c3640 1 --2- 172.21.15.23:0/673105089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4670102170 0x7f467019de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:40.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.670+0000 7f46755c3640 1 -- 172.21.15.23:0/673105089 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467019e480 con 0x7f46701005c0 2024-03-28T05:20:40.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.670+0000 7f4674dc2640 1 --2- 172.21.15.23:0/673105089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46701017b0 0x7f4670196da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:20:40.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.670+0000 7f4675dc4640 1 --2- 172.21.15.23:0/673105089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4670102170 0x7f467019de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:20:40.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.670+0000 7f46755c3640 1 --2- 172.21.15.23:0/673105089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46701005c0 0x7f4670196860 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f466000d950 tx=0x7f466000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:40.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.670+0000 7f46667fc640 1 -- 172.21.15.23:0/673105089 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4660017020 con 0x7f46701005c0 2024-03-28T05:20:40.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.670+0000 7f46667fc640 1 -- 172.21.15.23:0/673105089 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4660009c30 con 0x7f46701005c0 2024-03-28T05:20:40.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.670+0000 7f4677827640 1 -- 172.21.15.23:0/673105089 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f467019e770 con 0x7f46701005c0 2024-03-28T05:20:40.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.670+0000 7f46667fc640 1 -- 172.21.15.23:0/673105089 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466000f720 con 0x7f46701005c0 2024-03-28T05:20:40.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.670+0000 7f4677827640 1 -- 172.21.15.23:0/673105089 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f467019ecc0 con 0x7f46701005c0 2024-03-28T05:20:40.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.671+0000 7f4677827640 1 -- 172.21.15.23:0/673105089 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46701078e0 con 0x7f46701005c0 2024-03-28T05:20:40.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.673+0000 7f46667fc640 1 -- 172.21.15.23:0/673105089 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f466001a020 con 0x7f46701005c0 2024-03-28T05:20:40.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.673+0000 7f46667fc640 1 --2- 172.21.15.23:0/673105089 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f465c0a2800 0x7f465c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:40.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.673+0000 7f4674dc2640 1 --2- 172.21.15.23:0/673105089 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f465c0a2800 0x7f465c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:40.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.673+0000 7f46667fc640 1 -- 172.21.15.23:0/673105089 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f46600ce9a0 con 0x7f46701005c0 2024-03-28T05:20:40.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.674+0000 7f4674dc2640 1 --2- 172.21.15.23:0/673105089 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f465c0a2800 0x7f465c0a4cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f466c005fd0 tx=0x7f466c0094d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:40.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.677+0000 7f46667fc640 1 -- 172.21.15.23:0/673105089 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4660021020 con 0x7f46701005c0 2024-03-28T05:20:40.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.912+0000 7f4677827640 1 -- 172.21.15.23:0/673105089 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f467002d2e0 con 0x7f46701005c0 2024-03-28T05:20:40.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.913+0000 7f46667fc640 1 -- 172.21.15.23:0/673105089 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2907 (secure 0 0 0) 0x7f4660009de0 con 0x7f46701005c0 2024-03-28T05:20:40.911 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:20:40.912 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/733913400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":733913400},{"type":"v1","addr":"172.21.15.23:6835","nonce":733913400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T05:20:40.912 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 4 2024-03-28T05:20:40.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.915+0000 7f4677827640 1 -- 172.21.15.23:0/673105089 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f465c0a2800 msgr2=0x7f465c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:40.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.915+0000 7f4677827640 1 --2- 172.21.15.23:0/673105089 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f465c0a2800 0x7f465c0a4cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f466c005fd0 tx=0x7f466c0094d0 comp rx=0 tx=0).stop 2024-03-28T05:20:40.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.915+0000 7f4677827640 1 -- 172.21.15.23:0/673105089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46701005c0 msgr2=0x7f4670196860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:40.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.915+0000 7f4677827640 1 --2- 172.21.15.23:0/673105089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46701005c0 0x7f4670196860 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f466000d950 tx=0x7f466000de10 comp rx=0 tx=0).stop 2024-03-28T05:20:40.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.915+0000 7f4677827640 1 -- 172.21.15.23:0/673105089 shutdown_connections 2024-03-28T05:20:40.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.916+0000 7f4677827640 1 --2- 172.21.15.23:0/673105089 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f465c0a2800 0x7f465c0a4cc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:40.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.916+0000 7f4677827640 1 --2- 172.21.15.23:0/673105089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4670102170 0x7f467019de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:40.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.916+0000 7f4677827640 1 --2- 172.21.15.23:0/673105089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46701017b0 0x7f4670196da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:40.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.916+0000 7f4677827640 1 --2- 172.21.15.23:0/673105089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46701005c0 0x7f4670196860 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:40.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.916+0000 7f4677827640 1 -- 172.21.15.23:0/673105089 >> 172.21.15.23:0/673105089 conn(0x7f46700fbb30 msgr2=0x7f4670104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:40.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.916+0000 7f4677827640 1 -- 172.21.15.23:0/673105089 shutdown_connections 2024-03-28T05:20:40.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:40.916+0000 7f4677827640 1 -- 172.21.15.23:0/673105089 wait complete. 2024-03-28T05:20:40.925 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 05:20:40 smithi023 systemd[1]: Starting Ceph mds.d for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:20:41.259 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 05:20:41 smithi023 podman[72906]: 2024-03-28 05:20:40.924443291 +0000 UTC m=+0.019244424 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:20:41.259 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 05:20:41 smithi023 podman[72906]: 2024-03-28 05:20:41.033465183 +0000 UTC m=+0.128266357 container create c2781f8fc1e55dbd0626d42fa6b8fe6712e0928d8976c859d898d9f43a89dcc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-28T05:20:41.259 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 05:20:41 smithi023 podman[72906]: 2024-03-28 05:20:41.261063447 +0000 UTC m=+0.355864580 container init c2781f8fc1e55dbd0626d42fa6b8fe6712e0928d8976c859d898d9f43a89dcc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-d, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-28T05:20:41.530 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 05:20:41 smithi023 podman[72906]: 2024-03-28 05:20:41.264459837 +0000 UTC m=+0.359260970 container start c2781f8fc1e55dbd0626d42fa6b8fe6712e0928d8976c859d898d9f43a89dcc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-28T05:20:41.530 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 05:20:41 smithi023 ceph-mds[72927]: -- 172.21.15.23:0/526111189 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5556350e9a40 con 0x5556350bcc00 2024-03-28T05:20:41.531 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 05:20:41 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-d[72923]: starting mds.d at 2024-03-28T05:20:41.531 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 05:20:41 smithi023 bash[72906]: c2781f8fc1e55dbd0626d42fa6b8fe6712e0928d8976c859d898d9f43a89dcc6 2024-03-28T05:20:41.531 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 05:20:41 smithi023 systemd[1]: Started Ceph mds.d for bf37ebc4-ecc1-11ee-95d0-87774f69a715. 2024-03-28T05:20:42.170 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:41 smithi098 ceph-mon[36087]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:42.170 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:41 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/673105089' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:42.170 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:41 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:42.170 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:41 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:42.170 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:41 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:42.170 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:41 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:42.170 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:41 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:42.171 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:41 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:42.171 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:41 smithi098 ceph-mon[36087]: Deploying daemon mds.e on smithi098 2024-03-28T05:20:42.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:41 smithi113 ceph-mon[37164]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:42.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:41 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/673105089' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:42.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:41 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:42.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:41 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:42.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:41 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:42.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:41 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:42.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:41 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:42.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:41 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:42.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:41 smithi113 ceph-mon[37164]: Deploying daemon mds.e on smithi098 2024-03-28T05:20:42.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:41 smithi023 ceph-mon[31954]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:42.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:41 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/673105089' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:42.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:41 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:42.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:41 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:42.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:41 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:42.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:41 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:42.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:41 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:42.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:41 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:42.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:41 smithi023 ceph-mon[31954]: Deploying daemon mds.e on smithi098 2024-03-28T05:20:42.928 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:20:43.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.047+0000 7f89155f4640 1 -- 172.21.15.23:0/2364659636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910100640 msgr2=0x7f8910100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:43.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.047+0000 7f89155f4640 1 --2- 172.21.15.23:0/2364659636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910100640 0x7f8910100a60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8904009ca0 tx=0x7f890402c0f0 comp rx=0 tx=0).stop 2024-03-28T05:20:43.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.047+0000 7f89155f4640 1 -- 172.21.15.23:0/2364659636 shutdown_connections 2024-03-28T05:20:43.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.047+0000 7f89155f4640 1 --2- 172.21.15.23:0/2364659636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89101021f0 0x7f8910108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:43.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.047+0000 7f89155f4640 1 --2- 172.21.15.23:0/2364659636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8910101830 0x7f8910101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:43.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.047+0000 7f89155f4640 1 --2- 172.21.15.23:0/2364659636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910100640 0x7f8910100a60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:43.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.047+0000 7f89155f4640 1 -- 172.21.15.23:0/2364659636 >> 172.21.15.23:0/2364659636 conn(0x7f89100fbbd0 msgr2=0x7f89100fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:43.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.047+0000 7f89155f4640 1 -- 172.21.15.23:0/2364659636 shutdown_connections 2024-03-28T05:20:43.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.047+0000 7f89155f4640 1 -- 172.21.15.23:0/2364659636 wait complete. 2024-03-28T05:20:43.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.047+0000 7f89155f4640 1 Processor -- start 2024-03-28T05:20:43.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.047+0000 7f89155f4640 1 -- start start 2024-03-28T05:20:43.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.048+0000 7f89155f4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8910100640 0x7f89101045d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:43.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.048+0000 7f89155f4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8910101830 0x7f8910102c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:43.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.048+0000 7f89155f4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89101021f0 0x7f8910103150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:43.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.048+0000 7f89155f4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891010c320 con 0x7f89101021f0 2024-03-28T05:20:43.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.048+0000 7f89155f4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891010c1a0 con 0x7f8910101830 2024-03-28T05:20:43.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.048+0000 7f89155f4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891010c4a0 con 0x7f8910100640 2024-03-28T05:20:43.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.048+0000 7f890f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89101021f0 0x7f8910103150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:43.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.048+0000 7f890f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89101021f0 0x7f8910103150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40436/0 (socket says 172.21.15.23:40436) 2024-03-28T05:20:43.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.048+0000 7f890f7fe640 1 -- 172.21.15.23:0/2243102506 learned_addr learned my addr 172.21.15.23:0/2243102506 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:20:43.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.048+0000 7f890e7fc640 1 --2- 172.21.15.23:0/2243102506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8910101830 0x7f8910102c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:43.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.048+0000 7f890effd640 1 --2- 172.21.15.23:0/2243102506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8910100640 0x7f89101045d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:43.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.048+0000 7f890f7fe640 1 -- 172.21.15.23:0/2243102506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8910100640 msgr2=0x7f89101045d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:43.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.048+0000 7f890f7fe640 1 --2- 172.21.15.23:0/2243102506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8910100640 0x7f89101045d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:43.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.048+0000 7f890f7fe640 1 -- 172.21.15.23:0/2243102506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8910101830 msgr2=0x7f8910102c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:43.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.048+0000 7f890f7fe640 1 --2- 172.21.15.23:0/2243102506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8910101830 0x7f8910102c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:43.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.048+0000 7f890f7fe640 1 -- 172.21.15.23:0/2243102506 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89101039e0 con 0x7f89101021f0 2024-03-28T05:20:43.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.048+0000 7f890f7fe640 1 --2- 172.21.15.23:0/2243102506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89101021f0 0x7f8910103150 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8908009fd0 tx=0x7f890800b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:43.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.048+0000 7f88f7fff640 1 -- 172.21.15.23:0/2243102506 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908015020 con 0x7f89101021f0 2024-03-28T05:20:43.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.048+0000 7f88f7fff640 1 -- 172.21.15.23:0/2243102506 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8908011d10 con 0x7f89101021f0 2024-03-28T05:20:43.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.048+0000 7f89155f4640 1 -- 172.21.15.23:0/2243102506 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89101a1c10 con 0x7f89101021f0 2024-03-28T05:20:43.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.049+0000 7f88f7fff640 1 -- 172.21.15.23:0/2243102506 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890800f2e0 con 0x7f89101021f0 2024-03-28T05:20:43.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.049+0000 7f89155f4640 1 -- 172.21.15.23:0/2243102506 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89101a2110 con 0x7f89101021f0 2024-03-28T05:20:43.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.050+0000 7f89155f4640 1 -- 172.21.15.23:0/2243102506 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89101a2340 con 0x7f89101021f0 2024-03-28T05:20:43.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.051+0000 7f88f7fff640 1 -- 172.21.15.23:0/2243102506 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8908018020 con 0x7f89101021f0 2024-03-28T05:20:43.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.051+0000 7f88f7fff640 1 --2- 172.21.15.23:0/2243102506 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f88f80a2800 0x7f88f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:43.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.051+0000 7f890effd640 1 --2- 172.21.15.23:0/2243102506 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f88f80a2800 0x7f88f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:43.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.051+0000 7f88f7fff640 1 -- 172.21.15.23:0/2243102506 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f89080cde70 con 0x7f89101021f0 2024-03-28T05:20:43.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.052+0000 7f890effd640 1 --2- 172.21.15.23:0/2243102506 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f88f80a2800 0x7f88f80a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f890402c630 tx=0x7f8904005f50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:43.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.056+0000 7f88f7fff640 1 -- 172.21.15.23:0/2243102506 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f890801f020 con 0x7f89101021f0 2024-03-28T05:20:43.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:42 smithi098 ceph-mon[36087]: mds.? [v2:172.21.15.23:6836/1972901387,v1:172.21.15.23:6837/1972901387] up:boot 2024-03-28T05:20:43.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:42 smithi098 ceph-mon[36087]: fsmap 4 up:standby 2024-03-28T05:20:43.173 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:42 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T05:20:43.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:42 smithi113 ceph-mon[37164]: mds.? [v2:172.21.15.23:6836/1972901387,v1:172.21.15.23:6837/1972901387] up:boot 2024-03-28T05:20:43.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:42 smithi113 ceph-mon[37164]: fsmap 4 up:standby 2024-03-28T05:20:43.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:42 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T05:20:43.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.278+0000 7f89155f4640 1 -- 172.21.15.23:0/2243102506 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89101a2630 con 0x7f89101021f0 2024-03-28T05:20:43.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.279+0000 7f88f7fff640 1 -- 172.21.15.23:0/2243102506 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3698 (secure 0 0 0) 0x7f890809b3b0 con 0x7f89101021f0 2024-03-28T05:20:43.277 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:20:43.277 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/733913400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":733913400},{"type":"v1","addr":"172.21.15.23:6835","nonce":733913400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1972901387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1972901387},{"type":"v1","addr":"172.21.15.23:6837","nonce":1972901387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T05:20:43.277 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 5 2024-03-28T05:20:43.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.281+0000 7f89155f4640 1 -- 172.21.15.23:0/2243102506 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f88f80a2800 msgr2=0x7f88f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:43.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.281+0000 7f89155f4640 1 --2- 172.21.15.23:0/2243102506 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f88f80a2800 0x7f88f80a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f890402c630 tx=0x7f8904005f50 comp rx=0 tx=0).stop 2024-03-28T05:20:43.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.281+0000 7f89155f4640 1 -- 172.21.15.23:0/2243102506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89101021f0 msgr2=0x7f8910103150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:43.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.281+0000 7f89155f4640 1 --2- 172.21.15.23:0/2243102506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89101021f0 0x7f8910103150 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8908009fd0 tx=0x7f890800b820 comp rx=0 tx=0).stop 2024-03-28T05:20:43.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.281+0000 7f89155f4640 1 -- 172.21.15.23:0/2243102506 shutdown_connections 2024-03-28T05:20:43.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.281+0000 7f89155f4640 1 --2- 172.21.15.23:0/2243102506 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f88f80a2800 0x7f88f80a4cc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:43.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.281+0000 7f89155f4640 1 --2- 172.21.15.23:0/2243102506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89101021f0 0x7f8910103150 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:43.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.281+0000 7f89155f4640 1 --2- 172.21.15.23:0/2243102506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8910101830 0x7f8910102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:43.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.281+0000 7f89155f4640 1 --2- 172.21.15.23:0/2243102506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8910100640 0x7f89101045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:43.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.281+0000 7f89155f4640 1 -- 172.21.15.23:0/2243102506 >> 172.21.15.23:0/2243102506 conn(0x7f89100fbbd0 msgr2=0x7f89100fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:43.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.281+0000 7f89155f4640 1 -- 172.21.15.23:0/2243102506 shutdown_connections 2024-03-28T05:20:43.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:43.281+0000 7f89155f4640 1 -- 172.21.15.23:0/2243102506 wait complete. 2024-03-28T05:20:43.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:42 smithi023 ceph-mon[31954]: mds.? [v2:172.21.15.23:6836/1972901387,v1:172.21.15.23:6837/1972901387] up:boot 2024-03-28T05:20:43.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:42 smithi023 ceph-mon[31954]: fsmap 4 up:standby 2024-03-28T05:20:43.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:42 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T05:20:43.831 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 28 05:20:43 smithi098 systemd[1]: Starting Ceph mds.e for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:20:44.083 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:43 smithi098 ceph-mon[36087]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:44.084 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:43 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2243102506' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:44.084 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 28 05:20:43 smithi098 podman[65169]: 2024-03-28 05:20:43.735417458 +0000 UTC m=+0.017557725 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:20:44.084 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 28 05:20:43 smithi098 podman[65169]: 2024-03-28 05:20:43.852674565 +0000 UTC m=+0.134814836 container create 5ec0c7c7e8b8b4a924685dc50245177a3805f0e63ee1380001505fd51e39cfea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-e, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-28T05:20:44.084 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 28 05:20:44 smithi098 podman[65169]: 2024-03-28 05:20:44.085863167 +0000 UTC m=+0.368003414 container init 5ec0c7c7e8b8b4a924685dc50245177a3805f0e63ee1380001505fd51e39cfea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T05:20:44.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:43 smithi113 ceph-mon[37164]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:44.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:43 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2243102506' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:44.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:43 smithi023 ceph-mon[31954]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:44.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:43 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2243102506' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:44.365 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 28 05:20:44 smithi098 podman[65169]: 2024-03-28 05:20:44.088411387 +0000 UTC m=+0.370551640 container start 5ec0c7c7e8b8b4a924685dc50245177a3805f0e63ee1380001505fd51e39cfea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-28T05:20:44.365 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 28 05:20:44 smithi098 ceph-mds[65188]: -- 172.21.15.98:0/3959381255 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d08329ba40 con 0x55d08326e800 2024-03-28T05:20:44.365 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 28 05:20:44 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-e[65184]: starting mds.e at 2024-03-28T05:20:44.366 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 28 05:20:44 smithi098 bash[65169]: 5ec0c7c7e8b8b4a924685dc50245177a3805f0e63ee1380001505fd51e39cfea 2024-03-28T05:20:44.366 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 28 05:20:44 smithi098 systemd[1]: Started Ceph mds.e for bf37ebc4-ecc1-11ee-95d0-87774f69a715. 2024-03-28T05:20:45.291 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:20:45.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.408+0000 7f58aad1b640 1 -- 172.21.15.23:0/3765966963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58a41017d0 msgr2=0x7f58a4101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:45.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.408+0000 7f58aad1b640 1 --2- 172.21.15.23:0/3765966963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58a41017d0 0x7f58a4101c50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5890009900 tx=0x7f589002bc30 comp rx=0 tx=0).stop 2024-03-28T05:20:45.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.408+0000 7f58aad1b640 1 -- 172.21.15.23:0/3765966963 shutdown_connections 2024-03-28T05:20:45.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.408+0000 7f58aad1b640 1 --2- 172.21.15.23:0/3765966963 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58a4102190 0x7f58a4108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:45.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.408+0000 7f58aad1b640 1 --2- 172.21.15.23:0/3765966963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58a41017d0 0x7f58a4101c50 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:45.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.408+0000 7f58aad1b640 1 --2- 172.21.15.23:0/3765966963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58a41005e0 0x7f58a4100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:45.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.408+0000 7f58aad1b640 1 -- 172.21.15.23:0/3765966963 >> 172.21.15.23:0/3765966963 conn(0x7f58a40fbb50 msgr2=0x7f58a40fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:45.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.408+0000 7f58aad1b640 1 -- 172.21.15.23:0/3765966963 shutdown_connections 2024-03-28T05:20:45.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.408+0000 7f58aad1b640 1 -- 172.21.15.23:0/3765966963 wait complete. 2024-03-28T05:20:45.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.409+0000 7f58aad1b640 1 Processor -- start 2024-03-28T05:20:45.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.409+0000 7f58aad1b640 1 -- start start 2024-03-28T05:20:45.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.409+0000 7f58aad1b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58a41005e0 0x7f58a41979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:45.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.409+0000 7f58aad1b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58a41017d0 0x7f58a4197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:45.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.409+0000 7f58aad1b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58a4102190 0x7f58a419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:45.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.409+0000 7f58aad1b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a410c260 con 0x7f58a41005e0 2024-03-28T05:20:45.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.409+0000 7f58aad1b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a410c0e0 con 0x7f58a4102190 2024-03-28T05:20:45.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.409+0000 7f58aad1b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a410c3e0 con 0x7f58a41017d0 2024-03-28T05:20:45.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.409+0000 7f58a8ab7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58a41005e0 0x7f58a41979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:45.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.409+0000 7f58a8ab7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58a41005e0 0x7f58a41979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40454/0 (socket says 172.21.15.23:40454) 2024-03-28T05:20:45.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.409+0000 7f58a8ab7640 1 -- 172.21.15.23:0/3073144638 learned_addr learned my addr 172.21.15.23:0/3073144638 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:20:45.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.409+0000 7f58a92b8640 1 --2- 172.21.15.23:0/3073144638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58a4102190 0x7f58a419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:45.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.409+0000 7f58a3fff640 1 --2- 172.21.15.23:0/3073144638 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58a41017d0 0x7f58a4197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:45.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.409+0000 7f58a8ab7640 1 -- 172.21.15.23:0/3073144638 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58a41017d0 msgr2=0x7f58a4197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:45.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.409+0000 7f58a8ab7640 1 --2- 172.21.15.23:0/3073144638 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58a41017d0 0x7f58a4197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:45.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.409+0000 7f58a8ab7640 1 -- 172.21.15.23:0/3073144638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58a4102190 msgr2=0x7f58a419efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:45.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.409+0000 7f58a8ab7640 1 --2- 172.21.15.23:0/3073144638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58a4102190 0x7f58a419efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:45.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.409+0000 7f58a8ab7640 1 -- 172.21.15.23:0/3073144638 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a419f610 con 0x7f58a41005e0 2024-03-28T05:20:45.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.409+0000 7f58a8ab7640 1 --2- 172.21.15.23:0/3073144638 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58a41005e0 0x7f58a41979f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f589800d770 tx=0x7f589800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:45.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.410+0000 7f58a1ffb640 1 -- 172.21.15.23:0/3073144638 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5898017070 con 0x7f58a41005e0 2024-03-28T05:20:45.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.410+0000 7f58a1ffb640 1 -- 172.21.15.23:0/3073144638 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5898009c30 con 0x7f58a41005e0 2024-03-28T05:20:45.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.410+0000 7f58aad1b640 1 -- 172.21.15.23:0/3073144638 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a419f900 con 0x7f58a41005e0 2024-03-28T05:20:45.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.410+0000 7f58a1ffb640 1 -- 172.21.15.23:0/3073144638 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5898003dd0 con 0x7f58a41005e0 2024-03-28T05:20:45.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.410+0000 7f58aad1b640 1 -- 172.21.15.23:0/3073144638 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a419fe20 con 0x7f58a41005e0 2024-03-28T05:20:45.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.411+0000 7f58aad1b640 1 -- 172.21.15.23:0/3073144638 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a41a0050 con 0x7f58a41005e0 2024-03-28T05:20:45.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.412+0000 7f58a1ffb640 1 -- 172.21.15.23:0/3073144638 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5898003fb0 con 0x7f58a41005e0 2024-03-28T05:20:45.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.412+0000 7f58a1ffb640 1 --2- 172.21.15.23:0/3073144638 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f58940a2800 0x7f58940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:45.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.413+0000 7f58a3fff640 1 --2- 172.21.15.23:0/3073144638 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f58940a2800 0x7f58940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:45.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.413+0000 7f58a1ffb640 1 -- 172.21.15.23:0/3073144638 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f58980cdf40 con 0x7f58a41005e0 2024-03-28T05:20:45.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.413+0000 7f58a3fff640 1 --2- 172.21.15.23:0/3073144638 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f58940a2800 0x7f58940a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5890006010 tx=0x7f5890000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:45.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.417+0000 7f58a1ffb640 1 -- 172.21.15.23:0/3073144638 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f589809b480 con 0x7f58a41005e0 2024-03-28T05:20:45.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:45 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:45.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:45 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:45.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:45 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:45.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:45 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:45.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:45 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:45.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:45 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:45.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:45 smithi113 ceph-mon[37164]: Deploying daemon mds.f on smithi113 2024-03-28T05:20:45.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:45 smithi113 ceph-mon[37164]: mds.? [v2:172.21.15.98:6834/3313932732,v1:172.21.15.98:6835/3313932732] up:boot 2024-03-28T05:20:45.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:45 smithi113 ceph-mon[37164]: fsmap 5 up:standby 2024-03-28T05:20:45.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:45 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T05:20:45.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:45 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:45.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:45 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:45.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:45 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:45.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:45 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:45.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:45 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:45.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:45 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:45.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:45 smithi023 ceph-mon[31954]: Deploying daemon mds.f on smithi113 2024-03-28T05:20:45.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:45 smithi023 ceph-mon[31954]: mds.? [v2:172.21.15.98:6834/3313932732,v1:172.21.15.98:6835/3313932732] up:boot 2024-03-28T05:20:45.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:45 smithi023 ceph-mon[31954]: fsmap 5 up:standby 2024-03-28T05:20:45.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:45 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T05:20:45.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:45 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:45.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:45 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:45.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:45 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:45.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:45 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:45.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:45 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:45.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:45 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:45.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:45 smithi098 ceph-mon[36087]: Deploying daemon mds.f on smithi113 2024-03-28T05:20:45.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:45 smithi098 ceph-mon[36087]: mds.? [v2:172.21.15.98:6834/3313932732,v1:172.21.15.98:6835/3313932732] up:boot 2024-03-28T05:20:45.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:45 smithi098 ceph-mon[36087]: fsmap 5 up:standby 2024-03-28T05:20:45.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:45 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T05:20:45.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.641+0000 7f58aad1b640 1 -- 172.21.15.23:0/3073144638 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58a41a0220 con 0x7f58a41005e0 2024-03-28T05:20:45.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.641+0000 7f58a1ffb640 1 -- 172.21.15.23:0/3073144638 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4489 (secure 0 0 0) 0x7f5898009de0 con 0x7f58a41005e0 2024-03-28T05:20:45.640 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:20:45.640 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/733913400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":733913400},{"type":"v1","addr":"172.21.15.23:6835","nonce":733913400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1972901387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1972901387},{"type":"v1","addr":"172.21.15.23:6837","nonce":1972901387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3313932732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3313932732},{"type":"v1","addr":"172.21.15.98:6835","nonce":3313932732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-28T05:20:45.640 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 6 2024-03-28T05:20:45.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.643+0000 7f58aad1b640 1 -- 172.21.15.23:0/3073144638 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f58940a2800 msgr2=0x7f58940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:45.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.643+0000 7f58aad1b640 1 --2- 172.21.15.23:0/3073144638 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f58940a2800 0x7f58940a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5890006010 tx=0x7f5890000bc0 comp rx=0 tx=0).stop 2024-03-28T05:20:45.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.643+0000 7f58aad1b640 1 -- 172.21.15.23:0/3073144638 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58a41005e0 msgr2=0x7f58a41979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:45.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.643+0000 7f58aad1b640 1 --2- 172.21.15.23:0/3073144638 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58a41005e0 0x7f58a41979f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f589800d770 tx=0x7f589800dc30 comp rx=0 tx=0).stop 2024-03-28T05:20:45.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.643+0000 7f58aad1b640 1 -- 172.21.15.23:0/3073144638 shutdown_connections 2024-03-28T05:20:45.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.643+0000 7f58aad1b640 1 --2- 172.21.15.23:0/3073144638 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f58940a2800 0x7f58940a4cc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:45.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.643+0000 7f58aad1b640 1 --2- 172.21.15.23:0/3073144638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58a4102190 0x7f58a419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:45.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.643+0000 7f58aad1b640 1 --2- 172.21.15.23:0/3073144638 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58a41017d0 0x7f58a4197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:45.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.643+0000 7f58aad1b640 1 --2- 172.21.15.23:0/3073144638 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58a41005e0 0x7f58a41979f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:45.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.643+0000 7f58aad1b640 1 -- 172.21.15.23:0/3073144638 >> 172.21.15.23:0/3073144638 conn(0x7f58a40fbb50 msgr2=0x7f58a40fd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:45.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.643+0000 7f58aad1b640 1 -- 172.21.15.23:0/3073144638 shutdown_connections 2024-03-28T05:20:45.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:45.643+0000 7f58aad1b640 1 -- 172.21.15.23:0/3073144638 wait complete. 2024-03-28T05:20:46.448 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:46 smithi113 ceph-mon[37164]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:46.449 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:46 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3073144638' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:46.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:46 smithi023 ceph-mon[31954]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:46.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:46 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3073144638' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:46.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:46 smithi098 ceph-mon[36087]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:46.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:46 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3073144638' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:46.742 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 28 05:20:46 smithi113 podman[67484]: 2024-03-28 05:20:46.683713149 +0000 UTC m=+0.131893015 container create 3e4fd97c12f5fd7b2f52f22845983de77ad9220c2032b5bcd9a76eba6cbe8b68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-f, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T05:20:47.242 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 28 05:20:46 smithi113 podman[67484]: 2024-03-28 05:20:46.944900586 +0000 UTC m=+0.393080458 container init 3e4fd97c12f5fd7b2f52f22845983de77ad9220c2032b5bcd9a76eba6cbe8b68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-f, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-28T05:20:47.242 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 28 05:20:46 smithi113 podman[67484]: 2024-03-28 05:20:46.947651004 +0000 UTC m=+0.395830871 container start 3e4fd97c12f5fd7b2f52f22845983de77ad9220c2032b5bcd9a76eba6cbe8b68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-f, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T05:20:47.242 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 28 05:20:46 smithi113 ceph-mds[67503]: -- 172.21.15.113:0/3485995436 <== mon.1 v2:172.21.15.113:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b810769a40 con 0x55b81073cc00 2024-03-28T05:20:47.243 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 28 05:20:46 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-f[67499]: starting mds.f at 2024-03-28T05:20:47.243 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 28 05:20:47 smithi113 bash[67484]: 3e4fd97c12f5fd7b2f52f22845983de77ad9220c2032b5bcd9a76eba6cbe8b68 2024-03-28T05:20:47.243 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 28 05:20:47 smithi113 systemd[1]: Started Ceph mds.f for bf37ebc4-ecc1-11ee-95d0-87774f69a715. 2024-03-28T05:20:47.654 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:20:47.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.772+0000 7f2c8e2b2640 1 -- 172.21.15.23:0/1752179581 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c881005c0 msgr2=0x7f2c881009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:47.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.772+0000 7f2c8e2b2640 1 --2- 172.21.15.23:0/1752179581 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c881005c0 0x7f2c881009e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2c78009900 tx=0x7f2c7802bc30 comp rx=0 tx=0).stop 2024-03-28T05:20:47.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.772+0000 7f2c8e2b2640 1 -- 172.21.15.23:0/1752179581 shutdown_connections 2024-03-28T05:20:47.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.772+0000 7f2c8e2b2640 1 --2- 172.21.15.23:0/1752179581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c88102170 0x7f2c88108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:47.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.772+0000 7f2c8e2b2640 1 --2- 172.21.15.23:0/1752179581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c881017b0 0x7f2c88101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:47.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.772+0000 7f2c8e2b2640 1 --2- 172.21.15.23:0/1752179581 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c881005c0 0x7f2c881009e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:47.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.772+0000 7f2c8e2b2640 1 -- 172.21.15.23:0/1752179581 >> 172.21.15.23:0/1752179581 conn(0x7f2c880fbb30 msgr2=0x7f2c880fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:47.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.772+0000 7f2c8e2b2640 1 -- 172.21.15.23:0/1752179581 shutdown_connections 2024-03-28T05:20:47.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.772+0000 7f2c8e2b2640 1 -- 172.21.15.23:0/1752179581 wait complete. 2024-03-28T05:20:47.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.772+0000 7f2c8e2b2640 1 Processor -- start 2024-03-28T05:20:47.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.772+0000 7f2c8e2b2640 1 -- start start 2024-03-28T05:20:47.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.773+0000 7f2c8e2b2640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c881005c0 0x7f2c88197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:47.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.773+0000 7f2c8e2b2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c881017b0 0x7f2c88197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:47.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.773+0000 7f2c8e2b2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c88102170 0x7f2c8819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:47.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.773+0000 7f2c8e2b2640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8810c230 con 0x7f2c881017b0 2024-03-28T05:20:47.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.773+0000 7f2c8e2b2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8810c0b0 con 0x7f2c88102170 2024-03-28T05:20:47.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.773+0000 7f2c8e2b2640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8810c3b0 con 0x7f2c881005c0 2024-03-28T05:20:47.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.773+0000 7f2c877fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c881017b0 0x7f2c88197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:47.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.773+0000 7f2c877fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c881017b0 0x7f2c88197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40478/0 (socket says 172.21.15.23:40478) 2024-03-28T05:20:47.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.773+0000 7f2c877fe640 1 -- 172.21.15.23:0/3244812344 learned_addr learned my addr 172.21.15.23:0/3244812344 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:20:47.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.773+0000 7f2c87fff640 1 --2- 172.21.15.23:0/3244812344 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c881005c0 0x7f2c88197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.773+0000 7f2c8c84f640 1 --2- 172.21.15.23:0/3244812344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c88102170 0x7f2c8819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.773+0000 7f2c877fe640 1 -- 172.21.15.23:0/3244812344 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c881005c0 msgr2=0x7f2c88197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.773+0000 7f2c877fe640 1 --2- 172.21.15.23:0/3244812344 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c881005c0 0x7f2c88197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.773+0000 7f2c877fe640 1 -- 172.21.15.23:0/3244812344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c88102170 msgr2=0x7f2c8819efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.773+0000 7f2c877fe640 1 --2- 172.21.15.23:0/3244812344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c88102170 0x7f2c8819efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.773+0000 7f2c877fe640 1 -- 172.21.15.23:0/3244812344 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8819f650 con 0x7f2c881017b0 2024-03-28T05:20:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.773+0000 7f2c877fe640 1 --2- 172.21.15.23:0/3244812344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c881017b0 0x7f2c88197f70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2c7000d770 tx=0x7f2c7000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.773+0000 7f2c857fa640 1 -- 172.21.15.23:0/3244812344 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c70017070 con 0x7f2c881017b0 2024-03-28T05:20:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.773+0000 7f2c8e2b2640 1 -- 172.21.15.23:0/3244812344 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c8819f940 con 0x7f2c881017b0 2024-03-28T05:20:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.773+0000 7f2c857fa640 1 -- 172.21.15.23:0/3244812344 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c70005500 con 0x7f2c881017b0 2024-03-28T05:20:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.773+0000 7f2c857fa640 1 -- 172.21.15.23:0/3244812344 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c70004e00 con 0x7f2c881017b0 2024-03-28T05:20:47.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.774+0000 7f2c8e2b2640 1 -- 172.21.15.23:0/3244812344 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c8819fe40 con 0x7f2c881017b0 2024-03-28T05:20:47.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.774+0000 7f2c8e2b2640 1 -- 172.21.15.23:0/3244812344 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c881a0070 con 0x7f2c881017b0 2024-03-28T05:20:47.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.776+0000 7f2c857fa640 1 -- 172.21.15.23:0/3244812344 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2c70004fa0 con 0x7f2c881017b0 2024-03-28T05:20:47.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.776+0000 7f2c857fa640 1 --2- 172.21.15.23:0/3244812344 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2c740a2800 0x7f2c740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:47.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.776+0000 7f2c87fff640 1 --2- 172.21.15.23:0/3244812344 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2c740a2800 0x7f2c740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:47.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.776+0000 7f2c857fa640 1 -- 172.21.15.23:0/3244812344 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2c700ce400 con 0x7f2c881017b0 2024-03-28T05:20:47.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.777+0000 7f2c87fff640 1 --2- 172.21.15.23:0/3244812344 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2c740a2800 0x7f2c740a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2c78000c00 tx=0x7f2c78005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:47.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:47.780+0000 7f2c857fa640 1 -- 172.21.15.23:0/3244812344 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2c7009b940 con 0x7f2c881017b0 2024-03-28T05:20:48.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:48.002+0000 7f2c8e2b2640 1 -- 172.21.15.23:0/3244812344 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c881a0240 con 0x7f2c881017b0 2024-03-28T05:20:48.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:48.002+0000 7f2c857fa640 1 -- 172.21.15.23:0/3244812344 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5283 (secure 0 0 0) 0x7f2c7009b240 con 0x7f2c881017b0 2024-03-28T05:20:48.000 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:20:48.001 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/733913400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":733913400},{"type":"v1","addr":"172.21.15.23:6835","nonce":733913400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1972901387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1972901387},{"type":"v1","addr":"172.21.15.23:6837","nonce":1972901387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3313932732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3313932732},{"type":"v1","addr":"172.21.15.98:6835","nonce":3313932732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2695340024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2695340024},{"type":"v1","addr":"172.21.15.113:6835","nonce":2695340024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-28T05:20:48.001 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 7 2024-03-28T05:20:48.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:48.004+0000 7f2c8e2b2640 1 -- 172.21.15.23:0/3244812344 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2c740a2800 msgr2=0x7f2c740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:48.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:48.004+0000 7f2c8e2b2640 1 --2- 172.21.15.23:0/3244812344 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2c740a2800 0x7f2c740a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2c78000c00 tx=0x7f2c78005fd0 comp rx=0 tx=0).stop 2024-03-28T05:20:48.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:48.004+0000 7f2c8e2b2640 1 -- 172.21.15.23:0/3244812344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c881017b0 msgr2=0x7f2c88197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:48.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:48.004+0000 7f2c8e2b2640 1 --2- 172.21.15.23:0/3244812344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c881017b0 0x7f2c88197f70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2c7000d770 tx=0x7f2c7000dc30 comp rx=0 tx=0).stop 2024-03-28T05:20:48.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:48.004+0000 7f2c8e2b2640 1 -- 172.21.15.23:0/3244812344 shutdown_connections 2024-03-28T05:20:48.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:48.004+0000 7f2c8e2b2640 1 --2- 172.21.15.23:0/3244812344 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2c740a2800 0x7f2c740a4cc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:48.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:48.004+0000 7f2c8e2b2640 1 --2- 172.21.15.23:0/3244812344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c88102170 0x7f2c8819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:48.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:48.004+0000 7f2c8e2b2640 1 --2- 172.21.15.23:0/3244812344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c881017b0 0x7f2c88197f70 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:48.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:48.004+0000 7f2c8e2b2640 1 --2- 172.21.15.23:0/3244812344 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c881005c0 0x7f2c88197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:48.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:48.004+0000 7f2c8e2b2640 1 -- 172.21.15.23:0/3244812344 >> 172.21.15.23:0/3244812344 conn(0x7f2c880fbb30 msgr2=0x7f2c88068830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:48.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:48.004+0000 7f2c8e2b2640 1 -- 172.21.15.23:0/3244812344 shutdown_connections 2024-03-28T05:20:48.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:48.004+0000 7f2c8e2b2640 1 -- 172.21.15.23:0/3244812344 wait complete. 2024-03-28T05:20:48.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:48 smithi023 ceph-mon[31954]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:48.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:48.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:48.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:48.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:48.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:48.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:48.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:48 smithi023 ceph-mon[31954]: Deploying daemon mds.c on smithi113 2024-03-28T05:20:48.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:48 smithi023 ceph-mon[31954]: mds.? [v2:172.21.15.113:6834/2695340024,v1:172.21.15.113:6835/2695340024] up:boot 2024-03-28T05:20:48.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:48 smithi023 ceph-mon[31954]: fsmap 6 up:standby 2024-03-28T05:20:48.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:48 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T05:20:48.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:48 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3244812344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:48.282 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:48 smithi113 ceph-mon[37164]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:48.282 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:48 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:48.282 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:48 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:48.282 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:48 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:48.282 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:48 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:48.282 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:48 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:48.282 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:48 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:48.283 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:48 smithi113 ceph-mon[37164]: Deploying daemon mds.c on smithi113 2024-03-28T05:20:48.283 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:48 smithi113 ceph-mon[37164]: mds.? [v2:172.21.15.113:6834/2695340024,v1:172.21.15.113:6835/2695340024] up:boot 2024-03-28T05:20:48.283 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:48 smithi113 ceph-mon[37164]: fsmap 6 up:standby 2024-03-28T05:20:48.283 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:48 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T05:20:48.283 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:48 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3244812344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:48.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:48 smithi098 ceph-mon[36087]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:48.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:48 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:48.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:48 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:48.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:48 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:48.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:48 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:48.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:48 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:48.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:48 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:48.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:48 smithi098 ceph-mon[36087]: Deploying daemon mds.c on smithi113 2024-03-28T05:20:48.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:48 smithi098 ceph-mon[36087]: mds.? [v2:172.21.15.113:6834/2695340024,v1:172.21.15.113:6835/2695340024] up:boot 2024-03-28T05:20:48.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:48 smithi098 ceph-mon[36087]: fsmap 6 up:standby 2024-03-28T05:20:48.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:48 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T05:20:48.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:48 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3244812344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:49.492 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 28 05:20:49 smithi113 systemd[1]: Starting Ceph mds.c for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:20:49.791 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 28 05:20:49 smithi113 podman[67811]: 2024-03-28 05:20:49.424902255 +0000 UTC m=+0.017365952 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:20:49.791 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 28 05:20:49 smithi113 podman[67811]: 2024-03-28 05:20:49.539189102 +0000 UTC m=+0.131652776 container create 9a95cbf359a8f82b456b0310d42b9c53ba465d314e574159014b86d7a967a0b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True) 2024-03-28T05:20:49.791 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 28 05:20:49 smithi113 podman[67811]: 2024-03-28 05:20:49.76714929 +0000 UTC m=+0.359612978 container init 9a95cbf359a8f82b456b0310d42b9c53ba465d314e574159014b86d7a967a0b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-c, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T05:20:49.791 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 28 05:20:49 smithi113 podman[67811]: 2024-03-28 05:20:49.769700228 +0000 UTC m=+0.362163902 container start 9a95cbf359a8f82b456b0310d42b9c53ba465d314e574159014b86d7a967a0b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-c, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-28T05:20:49.791 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 28 05:20:49 smithi113 ceph-mds[67830]: -- 172.21.15.113:0/2863147601 <== mon.1 v2:172.21.15.113:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56517ba1ba40 con 0x56517b9ee800 2024-03-28T05:20:50.014 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:20:50.106 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 28 05:20:49 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-c[67826]: starting mds.c at 2024-03-28T05:20:50.106 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 28 05:20:49 smithi113 bash[67811]: 9a95cbf359a8f82b456b0310d42b9c53ba465d314e574159014b86d7a967a0b7 2024-03-28T05:20:50.106 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 28 05:20:49 smithi113 systemd[1]: Started Ceph mds.c for bf37ebc4-ecc1-11ee-95d0-87774f69a715. 2024-03-28T05:20:50.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.131+0000 7fe823f45640 1 -- 172.21.15.23:0/934224968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe81c102190 msgr2=0x7fe81c108a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:50.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.131+0000 7fe823f45640 1 --2- 172.21.15.23:0/934224968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe81c102190 0x7fe81c108a20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe814009ca0 tx=0x7fe81402c0f0 comp rx=0 tx=0).stop 2024-03-28T05:20:50.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.131+0000 7fe823f45640 1 -- 172.21.15.23:0/934224968 shutdown_connections 2024-03-28T05:20:50.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.131+0000 7fe823f45640 1 --2- 172.21.15.23:0/934224968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe81c102190 0x7fe81c108a20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:50.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.131+0000 7fe823f45640 1 --2- 172.21.15.23:0/934224968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe81c1017d0 0x7fe81c101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:50.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.131+0000 7fe823f45640 1 --2- 172.21.15.23:0/934224968 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe81c1005e0 0x7fe81c100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:50.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.131+0000 7fe823f45640 1 -- 172.21.15.23:0/934224968 >> 172.21.15.23:0/934224968 conn(0x7fe81c0fbb50 msgr2=0x7fe81c0fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:50.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.131+0000 7fe823f45640 1 -- 172.21.15.23:0/934224968 shutdown_connections 2024-03-28T05:20:50.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.131+0000 7fe823f45640 1 -- 172.21.15.23:0/934224968 wait complete. 2024-03-28T05:20:50.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.131+0000 7fe823f45640 1 Processor -- start 2024-03-28T05:20:50.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.131+0000 7fe823f45640 1 -- start start 2024-03-28T05:20:50.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe823f45640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe81c1005e0 0x7fe81c1979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:50.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe823f45640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe81c1017d0 0x7fe81c197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:50.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe823f45640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe81c102190 0x7fe81c19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:50.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe823f45640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c10c260 con 0x7fe81c1017d0 2024-03-28T05:20:50.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe823f45640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c10c0e0 con 0x7fe81c102190 2024-03-28T05:20:50.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe823f45640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c10c3e0 con 0x7fe81c1005e0 2024-03-28T05:20:50.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe8214e0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe81c1017d0 0x7fe81c197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:50.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe8214e0640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe81c1017d0 0x7fe81c197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40492/0 (socket says 172.21.15.23:40492) 2024-03-28T05:20:50.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe8214e0640 1 -- 172.21.15.23:0/2513109919 learned_addr learned my addr 172.21.15.23:0/2513109919 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:20:50.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe821ce1640 1 --2- 172.21.15.23:0/2513109919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe81c1005e0 0x7fe81c1979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:50.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe8224e2640 1 --2- 172.21.15.23:0/2513109919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe81c102190 0x7fe81c19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:50.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe8214e0640 1 -- 172.21.15.23:0/2513109919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe81c1005e0 msgr2=0x7fe81c1979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:50.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe8214e0640 1 --2- 172.21.15.23:0/2513109919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe81c1005e0 0x7fe81c1979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:50.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe8214e0640 1 -- 172.21.15.23:0/2513109919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe81c102190 msgr2=0x7fe81c19efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:50.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe8214e0640 1 --2- 172.21.15.23:0/2513109919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe81c102190 0x7fe81c19efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:50.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe8214e0640 1 -- 172.21.15.23:0/2513109919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe81c19f610 con 0x7fe81c1017d0 2024-03-28T05:20:50.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe8224e2640 1 --2- 172.21.15.23:0/2513109919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe81c102190 0x7fe81c19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:20:50.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe8214e0640 1 --2- 172.21.15.23:0/2513109919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe81c1017d0 0x7fe81c197f30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe81800e9f0 tx=0x7fe81800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:50.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe812ffd640 1 -- 172.21.15.23:0/2513109919 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81800cd80 con 0x7fe81c1017d0 2024-03-28T05:20:50.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe812ffd640 1 -- 172.21.15.23:0/2513109919 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe818010910 con 0x7fe81c1017d0 2024-03-28T05:20:50.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe823f45640 1 -- 172.21.15.23:0/2513109919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe81c19f900 con 0x7fe81c1017d0 2024-03-28T05:20:50.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe812ffd640 1 -- 172.21.15.23:0/2513109919 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe818018990 con 0x7fe81c1017d0 2024-03-28T05:20:50.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.132+0000 7fe823f45640 1 -- 172.21.15.23:0/2513109919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe81c19fe50 con 0x7fe81c1017d0 2024-03-28T05:20:50.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.133+0000 7fe823f45640 1 -- 172.21.15.23:0/2513109919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe81c1a0080 con 0x7fe81c1017d0 2024-03-28T05:20:50.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.134+0000 7fe812ffd640 1 -- 172.21.15.23:0/2513109919 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe818010ac0 con 0x7fe81c1017d0 2024-03-28T05:20:50.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.135+0000 7fe812ffd640 1 --2- 172.21.15.23:0/2513109919 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe8080a2800 0x7fe8080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:50.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.135+0000 7fe821ce1640 1 --2- 172.21.15.23:0/2513109919 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe8080a2800 0x7fe8080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:50.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.136+0000 7fe812ffd640 1 -- 172.21.15.23:0/2513109919 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe8180ce8f0 con 0x7fe81c1017d0 2024-03-28T05:20:50.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.136+0000 7fe821ce1640 1 --2- 172.21.15.23:0/2513109919 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe8080a2800 0x7fe8080a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe80c005fd0 tx=0x7fe80c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:50.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.140+0000 7fe812ffd640 1 -- 172.21.15.23:0/2513109919 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe81809b7e0 con 0x7fe81c1017d0 2024-03-28T05:20:50.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.362+0000 7fe823f45640 1 -- 172.21.15.23:0/2513109919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe81c1a0250 con 0x7fe81c1017d0 2024-03-28T05:20:50.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.362+0000 7fe812ffd640 1 -- 172.21.15.23:0/2513109919 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6077 (secure 0 0 0) 0x7fe81809f880 con 0x7fe81c1017d0 2024-03-28T05:20:50.361 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:20:50.361 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/733913400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":733913400},{"type":"v1","addr":"172.21.15.23:6835","nonce":733913400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1972901387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1972901387},{"type":"v1","addr":"172.21.15.23:6837","nonce":1972901387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3313932732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3313932732},{"type":"v1","addr":"172.21.15.98:6835","nonce":3313932732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2695340024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2695340024},{"type":"v1","addr":"172.21.15.113:6835","nonce":2695340024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/3222726274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":3222726274},{"type":"v1","addr":"172.21.15.113:6837","nonce":3222726274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-28T05:20:50.361 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 8 2024-03-28T05:20:50.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.364+0000 7fe823f45640 1 -- 172.21.15.23:0/2513109919 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe8080a2800 msgr2=0x7fe8080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:50.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.364+0000 7fe823f45640 1 --2- 172.21.15.23:0/2513109919 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe8080a2800 0x7fe8080a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe80c005fd0 tx=0x7fe80c00b390 comp rx=0 tx=0).stop 2024-03-28T05:20:50.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.365+0000 7fe823f45640 1 -- 172.21.15.23:0/2513109919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe81c1017d0 msgr2=0x7fe81c197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:50.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.365+0000 7fe823f45640 1 --2- 172.21.15.23:0/2513109919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe81c1017d0 0x7fe81c197f30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe81800e9f0 tx=0x7fe81800eeb0 comp rx=0 tx=0).stop 2024-03-28T05:20:50.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.365+0000 7fe823f45640 1 -- 172.21.15.23:0/2513109919 shutdown_connections 2024-03-28T05:20:50.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.365+0000 7fe823f45640 1 --2- 172.21.15.23:0/2513109919 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe8080a2800 0x7fe8080a4cc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:50.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.365+0000 7fe823f45640 1 --2- 172.21.15.23:0/2513109919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe81c102190 0x7fe81c19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:50.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.365+0000 7fe823f45640 1 --2- 172.21.15.23:0/2513109919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe81c1017d0 0x7fe81c197f30 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:50.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.365+0000 7fe823f45640 1 --2- 172.21.15.23:0/2513109919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe81c1005e0 0x7fe81c1979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:50.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.365+0000 7fe823f45640 1 -- 172.21.15.23:0/2513109919 >> 172.21.15.23:0/2513109919 conn(0x7fe81c0fbb50 msgr2=0x7fe81c0fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:50.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.365+0000 7fe823f45640 1 -- 172.21.15.23:0/2513109919 shutdown_connections 2024-03-28T05:20:50.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:50.365+0000 7fe823f45640 1 -- 172.21.15.23:0/2513109919 wait complete. 2024-03-28T05:20:50.364 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:50 smithi098 ceph-mon[36087]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:50.364 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:50 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:50.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:50 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:50.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:50 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:50.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:50 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:50.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:50 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:50.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:50 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:50.372 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:50 smithi023 ceph-mon[31954]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:50.373 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:50 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:50.373 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:50 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:50.373 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:50 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:50.373 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:50 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:50.373 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:50 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:50.373 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:50 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:50.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:50 smithi113 ceph-mon[37164]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:50.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:50 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:50.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:50 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:50.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:50 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:50.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:50 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:50.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:50 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:50.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:50 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:51.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:51 smithi098 ceph-mon[36087]: Deploying daemon mds.h on smithi098 2024-03-28T05:20:51.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:51 smithi098 ceph-mon[36087]: mds.? [v2:172.21.15.113:6836/3222726274,v1:172.21.15.113:6837/3222726274] up:boot 2024-03-28T05:20:51.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:51 smithi098 ceph-mon[36087]: fsmap 7 up:standby 2024-03-28T05:20:51.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:51 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T05:20:51.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:51 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2513109919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:51.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:51 smithi113 ceph-mon[37164]: Deploying daemon mds.h on smithi098 2024-03-28T05:20:51.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:51 smithi113 ceph-mon[37164]: mds.? [v2:172.21.15.113:6836/3222726274,v1:172.21.15.113:6837/3222726274] up:boot 2024-03-28T05:20:51.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:51 smithi113 ceph-mon[37164]: fsmap 7 up:standby 2024-03-28T05:20:51.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:51 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T05:20:51.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:51 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2513109919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:51.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:51 smithi023 ceph-mon[31954]: Deploying daemon mds.h on smithi098 2024-03-28T05:20:51.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:51 smithi023 ceph-mon[31954]: mds.? [v2:172.21.15.113:6836/3222726274,v1:172.21.15.113:6837/3222726274] up:boot 2024-03-28T05:20:51.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:51 smithi023 ceph-mon[31954]: fsmap 7 up:standby 2024-03-28T05:20:51.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:51 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T05:20:51.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:51 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2513109919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:52.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:52 smithi098 ceph-mon[36087]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:52.365 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 28 05:20:52 smithi098 systemd[1]: Starting Ceph mds.h for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:20:52.375 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:20:52.403 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:52 smithi023 ceph-mon[31954]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:52.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.492+0000 7fba8e80c640 1 -- 172.21.15.23:0/1017527192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba881005c0 msgr2=0x7fba881009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:52.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.492+0000 7fba8e80c640 1 --2- 172.21.15.23:0/1017527192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba881005c0 0x7fba881009e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fba70009900 tx=0x7fba7002bc30 comp rx=0 tx=0).stop 2024-03-28T05:20:52.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.492+0000 7fba8e80c640 1 -- 172.21.15.23:0/1017527192 shutdown_connections 2024-03-28T05:20:52.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.492+0000 7fba8e80c640 1 --2- 172.21.15.23:0/1017527192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba88102170 0x7fba88108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:52.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.492+0000 7fba8e80c640 1 --2- 172.21.15.23:0/1017527192 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba881017b0 0x7fba88101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:52.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.492+0000 7fba8e80c640 1 --2- 172.21.15.23:0/1017527192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba881005c0 0x7fba881009e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:52.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.492+0000 7fba8e80c640 1 -- 172.21.15.23:0/1017527192 >> 172.21.15.23:0/1017527192 conn(0x7fba880fbb30 msgr2=0x7fba880fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:52.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.492+0000 7fba8e80c640 1 -- 172.21.15.23:0/1017527192 shutdown_connections 2024-03-28T05:20:52.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.492+0000 7fba8e80c640 1 -- 172.21.15.23:0/1017527192 wait complete. 2024-03-28T05:20:52.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.492+0000 7fba8e80c640 1 Processor -- start 2024-03-28T05:20:52.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.492+0000 7fba8e80c640 1 -- start start 2024-03-28T05:20:52.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.493+0000 7fba8e80c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba881005c0 0x7fba88196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:52.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.493+0000 7fba8e80c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba881017b0 0x7fba88196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:52.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.493+0000 7fba8e80c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba88102170 0x7fba8819dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:52.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.493+0000 7fba8e80c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8810c230 con 0x7fba881017b0 2024-03-28T05:20:52.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.493+0000 7fba8e80c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8810c0b0 con 0x7fba881005c0 2024-03-28T05:20:52.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.493+0000 7fba8e80c640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8810c3b0 con 0x7fba88102170 2024-03-28T05:20:52.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.493+0000 7fba877fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba881017b0 0x7fba88196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:52.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.493+0000 7fba877fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba881017b0 0x7fba88196e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40502/0 (socket says 172.21.15.23:40502) 2024-03-28T05:20:52.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.493+0000 7fba877fe640 1 -- 172.21.15.23:0/375579699 learned_addr learned my addr 172.21.15.23:0/375579699 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:20:52.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.493+0000 7fba87fff640 1 --2- 172.21.15.23:0/375579699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba881005c0 0x7fba88196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:52.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.493+0000 7fba8cda9640 1 --2- 172.21.15.23:0/375579699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba88102170 0x7fba8819dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:52.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.493+0000 7fba877fe640 1 -- 172.21.15.23:0/375579699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba88102170 msgr2=0x7fba8819dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:52.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.493+0000 7fba877fe640 1 --2- 172.21.15.23:0/375579699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba88102170 0x7fba8819dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:52.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.493+0000 7fba877fe640 1 -- 172.21.15.23:0/375579699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba881005c0 msgr2=0x7fba88196910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:52.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.493+0000 7fba877fe640 1 --2- 172.21.15.23:0/375579699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba881005c0 0x7fba88196910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:52.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.493+0000 7fba877fe640 1 -- 172.21.15.23:0/375579699 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8819e530 con 0x7fba881017b0 2024-03-28T05:20:52.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.493+0000 7fba877fe640 1 --2- 172.21.15.23:0/375579699 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba881017b0 0x7fba88196e50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fba7800d770 tx=0x7fba7800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:52.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.494+0000 7fba857fa640 1 -- 172.21.15.23:0/375579699 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba78017070 con 0x7fba881017b0 2024-03-28T05:20:52.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.494+0000 7fba857fa640 1 -- 172.21.15.23:0/375579699 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba78005500 con 0x7fba881017b0 2024-03-28T05:20:52.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.494+0000 7fba8e80c640 1 -- 172.21.15.23:0/375579699 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba8819e820 con 0x7fba881017b0 2024-03-28T05:20:52.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.494+0000 7fba857fa640 1 -- 172.21.15.23:0/375579699 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba78004e00 con 0x7fba881017b0 2024-03-28T05:20:52.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.494+0000 7fba8e80c640 1 -- 172.21.15.23:0/375579699 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba8819ed20 con 0x7fba881017b0 2024-03-28T05:20:52.495 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:52 smithi113 ceph-mon[37164]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:52.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.495+0000 7fba8e80c640 1 -- 172.21.15.23:0/375579699 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba88107900 con 0x7fba881017b0 2024-03-28T05:20:52.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.496+0000 7fba857fa640 1 -- 172.21.15.23:0/375579699 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fba78004fa0 con 0x7fba881017b0 2024-03-28T05:20:52.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.496+0000 7fba857fa640 1 --2- 172.21.15.23:0/375579699 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fba740a2800 0x7fba740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:52.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.497+0000 7fba87fff640 1 --2- 172.21.15.23:0/375579699 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fba740a2800 0x7fba740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:52.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.497+0000 7fba857fa640 1 -- 172.21.15.23:0/375579699 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fba780ce000 con 0x7fba881017b0 2024-03-28T05:20:52.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.497+0000 7fba87fff640 1 --2- 172.21.15.23:0/375579699 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fba740a2800 0x7fba740a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fba70000c00 tx=0x7fba70005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:52.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.501+0000 7fba857fa640 1 -- 172.21.15.23:0/375579699 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fba7809b540 con 0x7fba881017b0 2024-03-28T05:20:52.656 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 28 05:20:52 smithi098 podman[65476]: 2024-03-28 05:20:52.288824419 +0000 UTC m=+0.017654136 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:20:52.657 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 28 05:20:52 smithi098 podman[65476]: 2024-03-28 05:20:52.406110557 +0000 UTC m=+0.134940278 container create 2747c54fcb7d4167f60cab80b397a75ae5b53a6ddc109554ad3f16be1b5608e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-h, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-28T05:20:52.657 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 28 05:20:52 smithi098 podman[65476]: 2024-03-28 05:20:52.659144102 +0000 UTC m=+0.387973807 container init 2747c54fcb7d4167f60cab80b397a75ae5b53a6ddc109554ad3f16be1b5608e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-h, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T05:20:52.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.724+0000 7fba8e80c640 1 -- 172.21.15.23:0/375579699 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba8802d2c0 con 0x7fba881017b0 2024-03-28T05:20:52.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.724+0000 7fba857fa640 1 -- 172.21.15.23:0/375579699 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6077 (secure 0 0 0) 0x7fba7809ae40 con 0x7fba881017b0 2024-03-28T05:20:52.722 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:20:52.722 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/733913400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":733913400},{"type":"v1","addr":"172.21.15.23:6835","nonce":733913400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1972901387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1972901387},{"type":"v1","addr":"172.21.15.23:6837","nonce":1972901387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3313932732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3313932732},{"type":"v1","addr":"172.21.15.98:6835","nonce":3313932732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2695340024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2695340024},{"type":"v1","addr":"172.21.15.113:6835","nonce":2695340024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/3222726274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":3222726274},{"type":"v1","addr":"172.21.15.113:6837","nonce":3222726274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-28T05:20:52.723 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 8 2024-03-28T05:20:52.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.726+0000 7fba8e80c640 1 -- 172.21.15.23:0/375579699 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fba740a2800 msgr2=0x7fba740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:52.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.726+0000 7fba8e80c640 1 --2- 172.21.15.23:0/375579699 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fba740a2800 0x7fba740a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fba70000c00 tx=0x7fba70005fd0 comp rx=0 tx=0).stop 2024-03-28T05:20:52.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.726+0000 7fba8e80c640 1 -- 172.21.15.23:0/375579699 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba881017b0 msgr2=0x7fba88196e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:52.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.726+0000 7fba8e80c640 1 --2- 172.21.15.23:0/375579699 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba881017b0 0x7fba88196e50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fba7800d770 tx=0x7fba7800dc30 comp rx=0 tx=0).stop 2024-03-28T05:20:52.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.726+0000 7fba8e80c640 1 -- 172.21.15.23:0/375579699 shutdown_connections 2024-03-28T05:20:52.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.726+0000 7fba8e80c640 1 --2- 172.21.15.23:0/375579699 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fba740a2800 0x7fba740a4cc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:52.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.726+0000 7fba8e80c640 1 --2- 172.21.15.23:0/375579699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fba88102170 0x7fba8819dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:52.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.726+0000 7fba8e80c640 1 --2- 172.21.15.23:0/375579699 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba881017b0 0x7fba88196e50 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:52.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.726+0000 7fba8e80c640 1 --2- 172.21.15.23:0/375579699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba881005c0 0x7fba88196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:52.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.726+0000 7fba8e80c640 1 -- 172.21.15.23:0/375579699 >> 172.21.15.23:0/375579699 conn(0x7fba880fbb30 msgr2=0x7fba88104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:52.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.726+0000 7fba8e80c640 1 -- 172.21.15.23:0/375579699 shutdown_connections 2024-03-28T05:20:52.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:52.726+0000 7fba8e80c640 1 -- 172.21.15.23:0/375579699 wait complete. 2024-03-28T05:20:53.115 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 28 05:20:52 smithi098 podman[65476]: 2024-03-28 05:20:52.661748905 +0000 UTC m=+0.390578607 container start 2747c54fcb7d4167f60cab80b397a75ae5b53a6ddc109554ad3f16be1b5608e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-h, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T05:20:53.115 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 28 05:20:52 smithi098 ceph-mds[65496]: -- 172.21.15.98:0/2271311197 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55565787da40 con 0x555657851000 2024-03-28T05:20:53.115 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 28 05:20:52 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-h[65492]: starting mds.h at 2024-03-28T05:20:53.116 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 28 05:20:52 smithi098 bash[65476]: 2747c54fcb7d4167f60cab80b397a75ae5b53a6ddc109554ad3f16be1b5608e4 2024-03-28T05:20:53.116 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 28 05:20:52 smithi098 systemd[1]: Started Ceph mds.h for bf37ebc4-ecc1-11ee-95d0-87774f69a715. 2024-03-28T05:20:53.481 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:53 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/375579699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:53.482 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:53 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:53.482 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:53 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:53.482 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:53 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:53.482 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:53 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:53.482 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:53 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:53.482 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:53 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:53.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:53 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/375579699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:53.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:53 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:53.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:53 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:53.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:53 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:53.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:53 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:53.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:53 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:53.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:53 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:53.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:53 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/375579699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:53.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:53 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:53.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:53 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:53.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:53 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:53.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:53 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:53.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:53 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:53.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:53 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:54.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:54 smithi113 ceph-mon[37164]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:54.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:54 smithi113 ceph-mon[37164]: Deploying daemon mds.i on smithi113 2024-03-28T05:20:54.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:54 smithi113 ceph-mon[37164]: mds.? [v2:172.21.15.98:6836/2750367469,v1:172.21.15.98:6837/2750367469] up:boot 2024-03-28T05:20:54.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:54 smithi113 ceph-mon[37164]: fsmap 8 up:standby 2024-03-28T05:20:54.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:54 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T05:20:54.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:54 smithi023 ceph-mon[31954]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:54.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:54 smithi023 ceph-mon[31954]: Deploying daemon mds.i on smithi113 2024-03-28T05:20:54.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:54 smithi023 ceph-mon[31954]: mds.? [v2:172.21.15.98:6836/2750367469,v1:172.21.15.98:6837/2750367469] up:boot 2024-03-28T05:20:54.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:54 smithi023 ceph-mon[31954]: fsmap 8 up:standby 2024-03-28T05:20:54.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:54 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T05:20:54.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:54 smithi098 ceph-mon[36087]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:54.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:54 smithi098 ceph-mon[36087]: Deploying daemon mds.i on smithi113 2024-03-28T05:20:54.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:54 smithi098 ceph-mon[36087]: mds.? [v2:172.21.15.98:6836/2750367469,v1:172.21.15.98:6837/2750367469] up:boot 2024-03-28T05:20:54.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:54 smithi098 ceph-mon[36087]: fsmap 8 up:standby 2024-03-28T05:20:54.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:54 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T05:20:54.736 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:20:54.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.853+0000 7f2ebb132640 1 -- 172.21.15.23:0/852502640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2eb41028a0 msgr2=0x7f2eb4102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:54.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.853+0000 7f2ebb132640 1 --2- 172.21.15.23:0/852502640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2eb41028a0 0x7f2eb4102d20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0009900 tx=0x7f2ea002bc30 comp rx=0 tx=0).stop 2024-03-28T05:20:54.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.853+0000 7f2ebb132640 1 -- 172.21.15.23:0/852502640 shutdown_connections 2024-03-28T05:20:54.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.853+0000 7f2ebb132640 1 --2- 172.21.15.23:0/852502640 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eb4103260 0x7f2eb4109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:54.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.853+0000 7f2ebb132640 1 --2- 172.21.15.23:0/852502640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2eb41028a0 0x7f2eb4102d20 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:54.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.853+0000 7f2ebb132640 1 --2- 172.21.15.23:0/852502640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eb41016b0 0x7f2eb4101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:54.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.853+0000 7f2ebb132640 1 -- 172.21.15.23:0/852502640 >> 172.21.15.23:0/852502640 conn(0x7f2eb40fcc80 msgr2=0x7f2eb40ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:54.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.853+0000 7f2ebb132640 1 -- 172.21.15.23:0/852502640 shutdown_connections 2024-03-28T05:20:54.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.853+0000 7f2ebb132640 1 -- 172.21.15.23:0/852502640 wait complete. 2024-03-28T05:20:54.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.854+0000 7f2ebb132640 1 Processor -- start 2024-03-28T05:20:54.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.854+0000 7f2ebb132640 1 -- start start 2024-03-28T05:20:54.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.854+0000 7f2ebb132640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eb41016b0 0x7f2eb4075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:54.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.854+0000 7f2ebb132640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2eb41028a0 0x7f2eb4073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:54.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.854+0000 7f2ebb132640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eb4103260 0x7f2eb4073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:54.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.854+0000 7f2ebb132640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb410d420 con 0x7f2eb41028a0 2024-03-28T05:20:54.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.854+0000 7f2ebb132640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb410d2a0 con 0x7f2eb41016b0 2024-03-28T05:20:54.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.854+0000 7f2ebb132640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb410d5a0 con 0x7f2eb4103260 2024-03-28T05:20:54.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.854+0000 7f2eb3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2eb41028a0 0x7f2eb4073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:54.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.854+0000 7f2eb3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2eb41028a0 0x7f2eb4073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60452/0 (socket says 172.21.15.23:60452) 2024-03-28T05:20:54.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.854+0000 7f2eb3fff640 1 -- 172.21.15.23:0/887475964 learned_addr learned my addr 172.21.15.23:0/887475964 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:20:54.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.854+0000 7f2eb96cf640 1 --2- 172.21.15.23:0/887475964 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eb4103260 0x7f2eb4073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:54.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.854+0000 7f2eb8ece640 1 --2- 172.21.15.23:0/887475964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eb41016b0 0x7f2eb4075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:54.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.854+0000 7f2eb3fff640 1 -- 172.21.15.23:0/887475964 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eb4103260 msgr2=0x7f2eb4073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:54.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.854+0000 7f2eb3fff640 1 --2- 172.21.15.23:0/887475964 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eb4103260 0x7f2eb4073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:54.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.854+0000 7f2eb3fff640 1 -- 172.21.15.23:0/887475964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eb41016b0 msgr2=0x7f2eb4075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:54.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.854+0000 7f2eb3fff640 1 --2- 172.21.15.23:0/887475964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eb41016b0 0x7f2eb4075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:54.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.854+0000 7f2eb3fff640 1 -- 172.21.15.23:0/887475964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb4074860 con 0x7f2eb41028a0 2024-03-28T05:20:54.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.854+0000 7f2eb8ece640 1 --2- 172.21.15.23:0/887475964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eb41016b0 0x7f2eb4075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:20:54.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.854+0000 7f2eb3fff640 1 --2- 172.21.15.23:0/887475964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2eb41028a0 0x7f2eb4073a90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0000c00 tx=0x7f2ea0005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:54.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.855+0000 7f2eb1ffb640 1 -- 172.21.15.23:0/887475964 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0043020 con 0x7f2eb41028a0 2024-03-28T05:20:54.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.855+0000 7f2eb1ffb640 1 -- 172.21.15.23:0/887475964 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ea00054e0 con 0x7f2eb41028a0 2024-03-28T05:20:54.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.855+0000 7f2ebb132640 1 -- 172.21.15.23:0/887475964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb41a1b80 con 0x7f2eb41028a0 2024-03-28T05:20:54.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.855+0000 7f2eb1ffb640 1 -- 172.21.15.23:0/887475964 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0003e60 con 0x7f2eb41028a0 2024-03-28T05:20:54.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.855+0000 7f2ebb132640 1 -- 172.21.15.23:0/887475964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb41a20d0 con 0x7f2eb41028a0 2024-03-28T05:20:54.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.856+0000 7f2ebb132640 1 -- 172.21.15.23:0/887475964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb4108970 con 0x7f2eb41028a0 2024-03-28T05:20:54.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.857+0000 7f2eb1ffb640 1 -- 172.21.15.23:0/887475964 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2ea0046020 con 0x7f2eb41028a0 2024-03-28T05:20:54.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.858+0000 7f2eb1ffb640 1 --2- 172.21.15.23:0/887475964 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2ea40a2800 0x7f2ea40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:54.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.858+0000 7f2eb1ffb640 1 -- 172.21.15.23:0/887475964 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2ea00f0c10 con 0x7f2eb41028a0 2024-03-28T05:20:54.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.858+0000 7f2eb8ece640 1 --2- 172.21.15.23:0/887475964 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2ea40a2800 0x7f2ea40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:54.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.858+0000 7f2eb8ece640 1 --2- 172.21.15.23:0/887475964 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2ea40a2800 0x7f2ea40a4cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8006fd0 tx=0x7f2ea8008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:54.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:54.861+0000 7f2eb1ffb640 1 -- 172.21.15.23:0/887475964 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2ea004d020 con 0x7f2eb41028a0 2024-03-28T05:20:55.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:55.083+0000 7f2ebb132640 1 -- 172.21.15.23:0/887475964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2eb402d2c0 con 0x7f2eb41028a0 2024-03-28T05:20:55.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:55.084+0000 7f2eb1ffb640 1 -- 172.21.15.23:0/887475964 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6868 (secure 0 0 0) 0x7f2ea00be150 con 0x7f2eb41028a0 2024-03-28T05:20:55.082 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:20:55.082 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/733913400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":733913400},{"type":"v1","addr":"172.21.15.23:6835","nonce":733913400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1972901387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1972901387},{"type":"v1","addr":"172.21.15.23:6837","nonce":1972901387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3313932732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3313932732},{"type":"v1","addr":"172.21.15.98:6835","nonce":3313932732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2695340024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2695340024},{"type":"v1","addr":"172.21.15.113:6835","nonce":2695340024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2750367469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2750367469},{"type":"v1","addr":"172.21.15.98:6837","nonce":2750367469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/3222726274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":3222726274},{"type":"v1","addr":"172.21.15.113:6837","nonce":3222726274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-28T05:20:55.083 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 9 2024-03-28T05:20:55.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:55.085+0000 7f2ebb132640 1 -- 172.21.15.23:0/887475964 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2ea40a2800 msgr2=0x7f2ea40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:55.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:55.085+0000 7f2ebb132640 1 --2- 172.21.15.23:0/887475964 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2ea40a2800 0x7f2ea40a4cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8006fd0 tx=0x7f2ea8008040 comp rx=0 tx=0).stop 2024-03-28T05:20:55.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:55.085+0000 7f2ebb132640 1 -- 172.21.15.23:0/887475964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2eb41028a0 msgr2=0x7f2eb4073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:55.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:55.085+0000 7f2ebb132640 1 --2- 172.21.15.23:0/887475964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2eb41028a0 0x7f2eb4073a90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0000c00 tx=0x7f2ea0005dd0 comp rx=0 tx=0).stop 2024-03-28T05:20:55.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:55.086+0000 7f2ebb132640 1 -- 172.21.15.23:0/887475964 shutdown_connections 2024-03-28T05:20:55.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:55.086+0000 7f2ebb132640 1 --2- 172.21.15.23:0/887475964 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2ea40a2800 0x7f2ea40a4cc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:55.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:55.086+0000 7f2ebb132640 1 --2- 172.21.15.23:0/887475964 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2eb4103260 0x7f2eb4073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:55.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:55.086+0000 7f2ebb132640 1 --2- 172.21.15.23:0/887475964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2eb41028a0 0x7f2eb4073a90 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:55.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:55.086+0000 7f2ebb132640 1 --2- 172.21.15.23:0/887475964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2eb41016b0 0x7f2eb4075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:55.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:55.086+0000 7f2ebb132640 1 -- 172.21.15.23:0/887475964 >> 172.21.15.23:0/887475964 conn(0x7f2eb40fcc80 msgr2=0x7f2eb4105b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:55.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:55.086+0000 7f2ebb132640 1 -- 172.21.15.23:0/887475964 shutdown_connections 2024-03-28T05:20:55.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:55.086+0000 7f2ebb132640 1 -- 172.21.15.23:0/887475964 wait complete. 2024-03-28T05:20:55.144 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 28 05:20:54 smithi113 systemd[1]: Starting Ceph mds.i for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:20:55.464 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:55 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/887475964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:55.465 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 28 05:20:55 smithi113 podman[68117]: 2024-03-28 05:20:55.101792139 +0000 UTC m=+0.017755820 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:20:55.465 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 28 05:20:55 smithi113 podman[68117]: 2024-03-28 05:20:55.213670021 +0000 UTC m=+0.129633692 container create a3c39012d51c34548fbffe086fe5ecb0b21dd14ca120417ea03a80bb93042649 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-i, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-28T05:20:55.465 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 28 05:20:55 smithi113 podman[68117]: 2024-03-28 05:20:55.466560715 +0000 UTC m=+0.382524386 container init a3c39012d51c34548fbffe086fe5ecb0b21dd14ca120417ea03a80bb93042649 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-28T05:20:55.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:55 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/887475964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:55.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:55 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/887475964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:55.742 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 28 05:20:55 smithi113 podman[68117]: 2024-03-28 05:20:55.469533656 +0000 UTC m=+0.385497328 container start a3c39012d51c34548fbffe086fe5ecb0b21dd14ca120417ea03a80bb93042649 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-i, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-28T05:20:55.742 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 28 05:20:55 smithi113 ceph-mds[68136]: -- 172.21.15.113:0/3218923602 <== mon.1 v2:172.21.15.113:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5643914d5a40 con 0x5643914a8800 2024-03-28T05:20:55.742 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 28 05:20:55 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-i[68132]: starting mds.i at 2024-03-28T05:20:55.742 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 28 05:20:55 smithi113 bash[68117]: a3c39012d51c34548fbffe086fe5ecb0b21dd14ca120417ea03a80bb93042649 2024-03-28T05:20:55.742 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 28 05:20:55 smithi113 systemd[1]: Started Ceph mds.i for bf37ebc4-ecc1-11ee-95d0-87774f69a715. 2024-03-28T05:20:56.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:56 smithi098 ceph-mon[36087]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:56.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:56.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:56.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:56.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:56.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:56.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:56.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:56 smithi098 ceph-mon[36087]: Deploying daemon mds.g on smithi023 2024-03-28T05:20:56.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:56 smithi098 ceph-mon[36087]: mds.? [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] up:boot 2024-03-28T05:20:56.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:56 smithi098 ceph-mon[36087]: fsmap 9 up:standby 2024-03-28T05:20:56.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:56 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T05:20:56.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:56 smithi113 ceph-mon[37164]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:56.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:56.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:56.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:56.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:56.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:56.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:56.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:56 smithi113 ceph-mon[37164]: Deploying daemon mds.g on smithi023 2024-03-28T05:20:56.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:56 smithi113 ceph-mon[37164]: mds.? [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] up:boot 2024-03-28T05:20:56.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:56 smithi113 ceph-mon[37164]: fsmap 9 up:standby 2024-03-28T05:20:56.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:56 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T05:20:57.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:56 smithi023 ceph-mon[31954]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:57.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:56 smithi023 ceph-mon[31954]: Deploying daemon mds.g on smithi023 2024-03-28T05:20:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:56 smithi023 ceph-mon[31954]: mds.? [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] up:boot 2024-03-28T05:20:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:56 smithi023 ceph-mon[31954]: fsmap 9 up:standby 2024-03-28T05:20:57.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:56 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T05:20:57.096 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:20:57.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.203+0000 7f60ae881640 1 -- 172.21.15.23:0/1342672755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60a8102160 msgr2=0x7f60a8104550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:57.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.203+0000 7f60ae881640 1 --2- 172.21.15.23:0/1342672755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60a8102160 0x7f60a8104550 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6098009900 tx=0x7f609802bc30 comp rx=0 tx=0).stop 2024-03-28T05:20:57.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.204+0000 7f60ae881640 1 -- 172.21.15.23:0/1342672755 shutdown_connections 2024-03-28T05:20:57.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.204+0000 7f60ae881640 1 --2- 172.21.15.23:0/1342672755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60a81073c0 0x7f60a8109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:57.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.204+0000 7f60ae881640 1 --2- 172.21.15.23:0/1342672755 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60a8104a90 0x7f60a8106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:57.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.204+0000 7f60ae881640 1 --2- 172.21.15.23:0/1342672755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60a8102160 0x7f60a8104550 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:57.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.204+0000 7f60ae881640 1 -- 172.21.15.23:0/1342672755 >> 172.21.15.23:0/1342672755 conn(0x7f60a80fbb50 msgr2=0x7f60a80fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:57.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.204+0000 7f60ae881640 1 -- 172.21.15.23:0/1342672755 shutdown_connections 2024-03-28T05:20:57.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.204+0000 7f60ae881640 1 -- 172.21.15.23:0/1342672755 wait complete. 2024-03-28T05:20:57.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.204+0000 7f60ae881640 1 Processor -- start 2024-03-28T05:20:57.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.204+0000 7f60ae881640 1 -- start start 2024-03-28T05:20:57.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.204+0000 7f60ae881640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60a8102160 0x7f60a8197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:57.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.204+0000 7f60ae881640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60a8104a90 0x7f60a8197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:57.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.204+0000 7f60ae881640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60a81073c0 0x7f60a819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:57.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.205+0000 7f60ae881640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a810d380 con 0x7f60a8102160 2024-03-28T05:20:57.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.205+0000 7f60ae881640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a810d200 con 0x7f60a81073c0 2024-03-28T05:20:57.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.205+0000 7f60ae881640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a810d500 con 0x7f60a8104a90 2024-03-28T05:20:57.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.205+0000 7f60a7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60a8102160 0x7f60a8197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:57.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.205+0000 7f60a7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60a8102160 0x7f60a8197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60474/0 (socket says 172.21.15.23:60474) 2024-03-28T05:20:57.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.205+0000 7f60a7fff640 1 -- 172.21.15.23:0/2909906189 learned_addr learned my addr 172.21.15.23:0/2909906189 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:20:57.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.205+0000 7f60ace1e640 1 --2- 172.21.15.23:0/2909906189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60a81073c0 0x7f60a819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:57.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.205+0000 7f60a77fe640 1 --2- 172.21.15.23:0/2909906189 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60a8104a90 0x7f60a8197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:57.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.205+0000 7f60a7fff640 1 -- 172.21.15.23:0/2909906189 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60a8104a90 msgr2=0x7f60a8197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:57.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.205+0000 7f60a7fff640 1 --2- 172.21.15.23:0/2909906189 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60a8104a90 0x7f60a8197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:57.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.205+0000 7f60a7fff640 1 -- 172.21.15.23:0/2909906189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60a81073c0 msgr2=0x7f60a819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:57.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.205+0000 7f60a7fff640 1 --2- 172.21.15.23:0/2909906189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60a81073c0 0x7f60a819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:57.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.205+0000 7f60a7fff640 1 -- 172.21.15.23:0/2909906189 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a819f620 con 0x7f60a8102160 2024-03-28T05:20:57.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.205+0000 7f60ace1e640 1 --2- 172.21.15.23:0/2909906189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60a81073c0 0x7f60a819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:20:57.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.205+0000 7f60a7fff640 1 --2- 172.21.15.23:0/2909906189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60a8102160 0x7f60a8197a00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6098009230 tx=0x7f6098005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:57.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.205+0000 7f60a57fa640 1 -- 172.21.15.23:0/2909906189 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6098041070 con 0x7f60a8102160 2024-03-28T05:20:57.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.205+0000 7f60a57fa640 1 -- 172.21.15.23:0/2909906189 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60980054e0 con 0x7f60a8102160 2024-03-28T05:20:57.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.205+0000 7f60ae881640 1 -- 172.21.15.23:0/2909906189 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60a819f8b0 con 0x7f60a8102160 2024-03-28T05:20:57.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.205+0000 7f60a57fa640 1 -- 172.21.15.23:0/2909906189 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6098034440 con 0x7f60a8102160 2024-03-28T05:20:57.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.205+0000 7f60ae881640 1 -- 172.21.15.23:0/2909906189 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a819fda0 con 0x7f60a8102160 2024-03-28T05:20:57.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.206+0000 7f60ae881640 1 -- 172.21.15.23:0/2909906189 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60a80fd640 con 0x7f60a8102160 2024-03-28T05:20:57.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.207+0000 7f60a57fa640 1 -- 172.21.15.23:0/2909906189 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6098044070 con 0x7f60a8102160 2024-03-28T05:20:57.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.208+0000 7f60a57fa640 1 --2- 172.21.15.23:0/2909906189 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f60900a2800 0x7f60900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:57.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.208+0000 7f60a77fe640 1 --2- 172.21.15.23:0/2909906189 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f60900a2800 0x7f60900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:57.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.208+0000 7f60a57fa640 1 -- 172.21.15.23:0/2909906189 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f60980f0380 con 0x7f60a8102160 2024-03-28T05:20:57.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.208+0000 7f60a77fe640 1 --2- 172.21.15.23:0/2909906189 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f60900a2800 0x7f60900a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f609c005fd0 tx=0x7f609c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:57.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.212+0000 7f60a57fa640 1 -- 172.21.15.23:0/2909906189 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f60980bd8c0 con 0x7f60a8102160 2024-03-28T05:20:57.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.445+0000 7f60ae881640 1 -- 172.21.15.23:0/2909906189 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60a802d070 con 0x7f60a8102160 2024-03-28T05:20:57.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.446+0000 7f60a57fa640 1 -- 172.21.15.23:0/2909906189 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7661 (secure 0 0 0) 0x7f60980bd1c0 con 0x7f60a8102160 2024-03-28T05:20:57.445 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:20:57.445 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/733913400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":733913400},{"type":"v1","addr":"172.21.15.23:6835","nonce":733913400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1972901387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1972901387},{"type":"v1","addr":"172.21.15.23:6837","nonce":1972901387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3313932732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3313932732},{"type":"v1","addr":"172.21.15.98:6835","nonce":3313932732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2695340024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2695340024},{"type":"v1","addr":"172.21.15.113:6835","nonce":2695340024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2750367469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2750367469},{"type":"v1","addr":"172.21.15.98:6837","nonce":2750367469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/3222726274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":3222726274},{"type":"v1","addr":"172.21.15.113:6837","nonce":3222726274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6839/212060489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":212060489},{"type":"v1","addr":"172.21.15.113:6839","nonce":212060489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T05:20:57.445 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 10 2024-03-28T05:20:57.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.449+0000 7f60ae881640 1 -- 172.21.15.23:0/2909906189 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f60900a2800 msgr2=0x7f60900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:57.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.449+0000 7f60ae881640 1 --2- 172.21.15.23:0/2909906189 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f60900a2800 0x7f60900a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f609c005fd0 tx=0x7f609c005850 comp rx=0 tx=0).stop 2024-03-28T05:20:57.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.449+0000 7f60ae881640 1 -- 172.21.15.23:0/2909906189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60a8102160 msgr2=0x7f60a8197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:57.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.449+0000 7f60ae881640 1 --2- 172.21.15.23:0/2909906189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60a8102160 0x7f60a8197a00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6098009230 tx=0x7f6098005dd0 comp rx=0 tx=0).stop 2024-03-28T05:20:57.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.449+0000 7f60ae881640 1 -- 172.21.15.23:0/2909906189 shutdown_connections 2024-03-28T05:20:57.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.449+0000 7f60ae881640 1 --2- 172.21.15.23:0/2909906189 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f60900a2800 0x7f60900a4cc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:57.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.449+0000 7f60ae881640 1 --2- 172.21.15.23:0/2909906189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60a81073c0 0x7f60a819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:57.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.449+0000 7f60ae881640 1 --2- 172.21.15.23:0/2909906189 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60a8104a90 0x7f60a8197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:57.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.449+0000 7f60ae881640 1 --2- 172.21.15.23:0/2909906189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60a8102160 0x7f60a8197a00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:57.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.449+0000 7f60ae881640 1 -- 172.21.15.23:0/2909906189 >> 172.21.15.23:0/2909906189 conn(0x7f60a80fbb50 msgr2=0x7f60a8102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:57.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.449+0000 7f60ae881640 1 -- 172.21.15.23:0/2909906189 shutdown_connections 2024-03-28T05:20:57.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:57.449+0000 7f60ae881640 1 -- 172.21.15.23:0/2909906189 wait complete. 2024-03-28T05:20:57.769 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:57 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2909906189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:57.770 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 05:20:57 smithi023 systemd[1]: Starting Ceph mds.g for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:20:57.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:57 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2909906189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:58.030 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 05:20:57 smithi023 podman[73506]: 2024-03-28 05:20:57.888786018 +0000 UTC m=+0.017539075 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:20:58.030 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 05:20:57 smithi023 podman[73506]: 2024-03-28 05:20:57.989132531 +0000 UTC m=+0.117885588 container create 155fb204768b62ac186e523df983b29d0016fc985240d9276636143b71ca8e5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-g, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-28T05:20:58.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:57 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2909906189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:20:58.530 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 05:20:58 smithi023 podman[73506]: 2024-03-28 05:20:58.250606582 +0000 UTC m=+0.379359640 container init 155fb204768b62ac186e523df983b29d0016fc985240d9276636143b71ca8e5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-g, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-28T05:20:58.531 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 05:20:58 smithi023 podman[73506]: 2024-03-28 05:20:58.253321625 +0000 UTC m=+0.382074683 container start 155fb204768b62ac186e523df983b29d0016fc985240d9276636143b71ca8e5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-g, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T05:20:58.531 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 05:20:58 smithi023 ceph-mds[73525]: -- 172.21.15.23:0/2754817358 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5565c9047a40 con 0x5565c901a800 2024-03-28T05:20:58.531 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 05:20:58 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-g[73521]: starting mds.g at 2024-03-28T05:20:58.531 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 05:20:58 smithi023 bash[73506]: 155fb204768b62ac186e523df983b29d0016fc985240d9276636143b71ca8e5a 2024-03-28T05:20:58.531 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 05:20:58 smithi023 systemd[1]: Started Ceph mds.g for bf37ebc4-ecc1-11ee-95d0-87774f69a715. 2024-03-28T05:20:58.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:58 smithi113 ceph-mon[37164]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:58.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:58 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:58.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:58 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:58.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:58 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:58.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:58 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:58.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:58 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:58.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:58 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:59.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:58 smithi023 ceph-mon[31954]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:59.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:58 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:59.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:58 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:59.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:58 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:59.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:58 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:59.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:58 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:59.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:58 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:59.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:58 smithi098 ceph-mon[36087]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:20:59.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:58 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:59.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:58 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:59.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:58 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:20:59.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:58 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:20:59.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:58 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:20:59.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:58 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:20:59.460 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:20:59.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.572+0000 7f88baa8c640 1 -- 172.21.15.23:0/3389646194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88b41073a0 msgr2=0x7f88b41097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:59.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.572+0000 7f88baa8c640 1 --2- 172.21.15.23:0/3389646194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88b41073a0 0x7f88b41097f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f88ac009ca0 tx=0x7f88ac02c0f0 comp rx=0 tx=0).stop 2024-03-28T05:20:59.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.572+0000 7f88baa8c640 1 -- 172.21.15.23:0/3389646194 shutdown_connections 2024-03-28T05:20:59.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.572+0000 7f88baa8c640 1 --2- 172.21.15.23:0/3389646194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88b41073a0 0x7f88b41097f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:59.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.572+0000 7f88baa8c640 1 --2- 172.21.15.23:0/3389646194 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88b4104a70 0x7f88b4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:59.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.572+0000 7f88baa8c640 1 --2- 172.21.15.23:0/3389646194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88b4102140 0x7f88b4104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:59.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.572+0000 7f88baa8c640 1 -- 172.21.15.23:0/3389646194 >> 172.21.15.23:0/3389646194 conn(0x7f88b40fbb30 msgr2=0x7f88b40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:59.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.572+0000 7f88baa8c640 1 -- 172.21.15.23:0/3389646194 shutdown_connections 2024-03-28T05:20:59.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.572+0000 7f88baa8c640 1 -- 172.21.15.23:0/3389646194 wait complete. 2024-03-28T05:20:59.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.573+0000 7f88baa8c640 1 Processor -- start 2024-03-28T05:20:59.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.573+0000 7f88baa8c640 1 -- start start 2024-03-28T05:20:59.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.573+0000 7f88baa8c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88b4102140 0x7f88b41979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:59.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.573+0000 7f88baa8c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88b4104a70 0x7f88b4197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:59.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.573+0000 7f88baa8c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88b41073a0 0x7f88b419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:59.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.573+0000 7f88baa8c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b410d360 con 0x7f88b4102140 2024-03-28T05:20:59.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.573+0000 7f88baa8c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b410d1e0 con 0x7f88b41073a0 2024-03-28T05:20:59.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.573+0000 7f88baa8c640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b410d4e0 con 0x7f88b4104a70 2024-03-28T05:20:59.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.573+0000 7f88b8828640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88b4102140 0x7f88b41979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:59.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.573+0000 7f88b8828640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88b4102140 0x7f88b41979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60522/0 (socket says 172.21.15.23:60522) 2024-03-28T05:20:59.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.573+0000 7f88b8828640 1 -- 172.21.15.23:0/2647588702 learned_addr learned my addr 172.21.15.23:0/2647588702 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:20:59.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.573+0000 7f88b9029640 1 --2- 172.21.15.23:0/2647588702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88b41073a0 0x7f88b419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:59.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.573+0000 7f88b3fff640 1 --2- 172.21.15.23:0/2647588702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88b4104a70 0x7f88b4197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:59.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.573+0000 7f88b8828640 1 -- 172.21.15.23:0/2647588702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88b4104a70 msgr2=0x7f88b4197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:59.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.573+0000 7f88b8828640 1 --2- 172.21.15.23:0/2647588702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88b4104a70 0x7f88b4197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:59.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.573+0000 7f88b8828640 1 -- 172.21.15.23:0/2647588702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88b41073a0 msgr2=0x7f88b419ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:59.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.573+0000 7f88b8828640 1 --2- 172.21.15.23:0/2647588702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88b41073a0 0x7f88b419ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:59.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.573+0000 7f88b8828640 1 -- 172.21.15.23:0/2647588702 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b419f5d0 con 0x7f88b4102140 2024-03-28T05:20:59.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.573+0000 7f88b3fff640 1 --2- 172.21.15.23:0/2647588702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88b4104a70 0x7f88b4197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:20:59.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.573+0000 7f88b9029640 1 --2- 172.21.15.23:0/2647588702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88b41073a0 0x7f88b419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:20:59.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.574+0000 7f88b8828640 1 --2- 172.21.15.23:0/2647588702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88b4102140 0x7f88b41979b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f88a800e9f0 tx=0x7f88a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:59.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.574+0000 7f88b1ffb640 1 -- 172.21.15.23:0/2647588702 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a800cd80 con 0x7f88b4102140 2024-03-28T05:20:59.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.574+0000 7f88baa8c640 1 -- 172.21.15.23:0/2647588702 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b419f8c0 con 0x7f88b4102140 2024-03-28T05:20:59.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.574+0000 7f88b1ffb640 1 -- 172.21.15.23:0/2647588702 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88a8005500 con 0x7f88b4102140 2024-03-28T05:20:59.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.574+0000 7f88b1ffb640 1 -- 172.21.15.23:0/2647588702 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a8010690 con 0x7f88b4102140 2024-03-28T05:20:59.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.574+0000 7f88baa8c640 1 -- 172.21.15.23:0/2647588702 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b419fe10 con 0x7f88b4102140 2024-03-28T05:20:59.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.575+0000 7f88baa8c640 1 -- 172.21.15.23:0/2647588702 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b40fd620 con 0x7f88b4102140 2024-03-28T05:20:59.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.576+0000 7f88b1ffb640 1 -- 172.21.15.23:0/2647588702 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f88a80046c0 con 0x7f88b4102140 2024-03-28T05:20:59.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.577+0000 7f88b1ffb640 1 --2- 172.21.15.23:0/2647588702 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f88a40a2800 0x7f88a40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:20:59.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.577+0000 7f88b1ffb640 1 -- 172.21.15.23:0/2647588702 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f88a80ced50 con 0x7f88b4102140 2024-03-28T05:20:59.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.577+0000 7f88b3fff640 1 --2- 172.21.15.23:0/2647588702 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f88a40a2800 0x7f88a40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:20:59.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.577+0000 7f88b3fff640 1 --2- 172.21.15.23:0/2647588702 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f88a40a2800 0x7f88a40a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f88a00058c0 tx=0x7f88a0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:20:59.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.581+0000 7f88b1ffb640 1 -- 172.21.15.23:0/2647588702 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f88a8053740 con 0x7f88b4102140 2024-03-28T05:20:59.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.811+0000 7f88baa8c640 1 -- 172.21.15.23:0/2647588702 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88b402d070 con 0x7f88b4102140 2024-03-28T05:20:59.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.811+0000 7f88b1ffb640 1 -- 172.21.15.23:0/2647588702 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8453 (secure 0 0 0) 0x7f88a801f340 con 0x7f88b4102140 2024-03-28T05:20:59.810 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:20:59.810 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/733913400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":733913400},{"type":"v1","addr":"172.21.15.23:6835","nonce":733913400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1972901387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1972901387},{"type":"v1","addr":"172.21.15.23:6837","nonce":1972901387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/2390673788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2390673788},{"type":"v1","addr":"172.21.15.23:6839","nonce":2390673788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3313932732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3313932732},{"type":"v1","addr":"172.21.15.98:6835","nonce":3313932732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2695340024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2695340024},{"type":"v1","addr":"172.21.15.113:6835","nonce":2695340024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2750367469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2750367469},{"type":"v1","addr":"172.21.15.98:6837","nonce":2750367469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/3222726274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":3222726274},{"type":"v1","addr":"172.21.15.113:6837","nonce":3222726274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6839/212060489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":212060489},{"type":"v1","addr":"172.21.15.113:6839","nonce":212060489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T05:20:59.810 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 11 2024-03-28T05:20:59.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.813+0000 7f88baa8c640 1 -- 172.21.15.23:0/2647588702 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f88a40a2800 msgr2=0x7f88a40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:59.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.813+0000 7f88baa8c640 1 --2- 172.21.15.23:0/2647588702 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f88a40a2800 0x7f88a40a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f88a00058c0 tx=0x7f88a0005850 comp rx=0 tx=0).stop 2024-03-28T05:20:59.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.813+0000 7f88baa8c640 1 -- 172.21.15.23:0/2647588702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88b4102140 msgr2=0x7f88b41979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:20:59.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.813+0000 7f88baa8c640 1 --2- 172.21.15.23:0/2647588702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88b4102140 0x7f88b41979b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f88a800e9f0 tx=0x7f88a800edf0 comp rx=0 tx=0).stop 2024-03-28T05:20:59.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.813+0000 7f88baa8c640 1 -- 172.21.15.23:0/2647588702 shutdown_connections 2024-03-28T05:20:59.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.813+0000 7f88baa8c640 1 --2- 172.21.15.23:0/2647588702 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f88a40a2800 0x7f88a40a4cc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:59.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.813+0000 7f88baa8c640 1 --2- 172.21.15.23:0/2647588702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88b41073a0 0x7f88b419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:59.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.813+0000 7f88baa8c640 1 --2- 172.21.15.23:0/2647588702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88b4104a70 0x7f88b4197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:59.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.813+0000 7f88baa8c640 1 --2- 172.21.15.23:0/2647588702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88b4102140 0x7f88b41979b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:20:59.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.813+0000 7f88baa8c640 1 -- 172.21.15.23:0/2647588702 >> 172.21.15.23:0/2647588702 conn(0x7f88b40fbb30 msgr2=0x7f88b4102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:20:59.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.814+0000 7f88baa8c640 1 -- 172.21.15.23:0/2647588702 shutdown_connections 2024-03-28T05:20:59.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:20:59.814+0000 7f88baa8c640 1 -- 172.21.15.23:0/2647588702 wait complete. 2024-03-28T05:20:59.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:59 smithi113 ceph-mon[37164]: Deploying daemon mds.a on smithi023 2024-03-28T05:20:59.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:59 smithi113 ceph-mon[37164]: mds.? [v2:172.21.15.23:6838/2390673788,v1:172.21.15.23:6839/2390673788] up:boot 2024-03-28T05:20:59.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:59 smithi113 ceph-mon[37164]: fsmap 10 up:standby 2024-03-28T05:20:59.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:20:59 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T05:21:00.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:59 smithi023 ceph-mon[31954]: Deploying daemon mds.a on smithi023 2024-03-28T05:21:00.018 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:59 smithi023 ceph-mon[31954]: mds.? [v2:172.21.15.23:6838/2390673788,v1:172.21.15.23:6839/2390673788] up:boot 2024-03-28T05:21:00.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:59 smithi023 ceph-mon[31954]: fsmap 10 up:standby 2024-03-28T05:21:00.019 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:20:59 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T05:21:00.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:59 smithi098 ceph-mon[36087]: Deploying daemon mds.a on smithi023 2024-03-28T05:21:00.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:59 smithi098 ceph-mon[36087]: mds.? [v2:172.21.15.23:6838/2390673788,v1:172.21.15.23:6839/2390673788] up:boot 2024-03-28T05:21:00.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:59 smithi098 ceph-mon[36087]: fsmap 10 up:standby 2024-03-28T05:21:00.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:20:59 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T05:21:00.634 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 05:21:00 smithi023 systemd[1]: Starting Ceph mds.a for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:21:00.991 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:00 smithi023 ceph-mon[31954]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:21:00.991 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:00 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2647588702' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:21:00.991 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 05:21:00 smithi023 podman[73871]: 2024-03-28 05:21:00.62435366 +0000 UTC m=+0.017727525 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:21:00.992 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 05:21:00 smithi023 podman[73871]: 2024-03-28 05:21:00.731520486 +0000 UTC m=+0.124894317 container create 85fb08824ad8e3f29dabb037e4bd1093220c7e2eebb46dbc3a98876c6792d713 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-a, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T05:21:00.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:00 smithi113 ceph-mon[37164]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:21:00.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:00 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2647588702' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:21:01.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:00 smithi098 ceph-mon[36087]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:21:01.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:00 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2647588702' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:21:01.280 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 05:21:00 smithi023 podman[73871]: 2024-03-28 05:21:00.993150682 +0000 UTC m=+0.386524514 container init 85fb08824ad8e3f29dabb037e4bd1093220c7e2eebb46dbc3a98876c6792d713 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-a, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-28T05:21:01.280 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 05:21:00 smithi023 podman[73871]: 2024-03-28 05:21:00.996074728 +0000 UTC m=+0.389448567 container start 85fb08824ad8e3f29dabb037e4bd1093220c7e2eebb46dbc3a98876c6792d713 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-a, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-03-28T05:21:01.281 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 05:21:01 smithi023 ceph-mds[73890]: -- 172.21.15.23:0/2508759611 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cca59e1a40 con 0x55cca59b5000 2024-03-28T05:21:01.281 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 05:21:01 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-a[73886]: starting mds.a at 2024-03-28T05:21:01.281 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 05:21:01 smithi023 bash[73871]: 85fb08824ad8e3f29dabb037e4bd1093220c7e2eebb46dbc3a98876c6792d713 2024-03-28T05:21:01.281 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 05:21:01 smithi023 systemd[1]: Started Ceph mds.a for bf37ebc4-ecc1-11ee-95d0-87774f69a715. 2024-03-28T05:21:01.826 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:21:01.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.941+0000 7fa456297640 1 -- 172.21.15.23:0/2172784765 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4501021c0 msgr2=0x7fa4501045b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:01.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.941+0000 7fa456297640 1 --2- 172.21.15.23:0/2172784765 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4501021c0 0x7fa4501045b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa438009900 tx=0x7fa43802bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:01.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.941+0000 7fa456297640 1 -- 172.21.15.23:0/2172784765 shutdown_connections 2024-03-28T05:21:01.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.941+0000 7fa456297640 1 --2- 172.21.15.23:0/2172784765 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa450107420 0x7fa450109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:01.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.941+0000 7fa456297640 1 --2- 172.21.15.23:0/2172784765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa450104af0 0x7fa450106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:01.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.941+0000 7fa456297640 1 --2- 172.21.15.23:0/2172784765 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4501021c0 0x7fa4501045b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:01.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.941+0000 7fa456297640 1 -- 172.21.15.23:0/2172784765 >> 172.21.15.23:0/2172784765 conn(0x7fa4500fbbd0 msgr2=0x7fa4500fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:01.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.941+0000 7fa456297640 1 -- 172.21.15.23:0/2172784765 shutdown_connections 2024-03-28T05:21:01.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.941+0000 7fa456297640 1 -- 172.21.15.23:0/2172784765 wait complete. 2024-03-28T05:21:01.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.942+0000 7fa456297640 1 Processor -- start 2024-03-28T05:21:01.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.942+0000 7fa456297640 1 -- start start 2024-03-28T05:21:01.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.942+0000 7fa456297640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4501021c0 0x7fa450101b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:01.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.942+0000 7fa456297640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa450104af0 0x7fa450100170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:01.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.942+0000 7fa456297640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa450107420 0x7fa4501006b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:01.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.942+0000 7fa456297640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45010d360 con 0x7fa450104af0 2024-03-28T05:21:01.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.942+0000 7fa456297640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45010d1e0 con 0x7fa450107420 2024-03-28T05:21:01.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.942+0000 7fa456297640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45010d4e0 con 0x7fa4501021c0 2024-03-28T05:21:01.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.942+0000 7fa44f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa450104af0 0x7fa450100170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:01.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.942+0000 7fa44f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa450104af0 0x7fa450100170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60562/0 (socket says 172.21.15.23:60562) 2024-03-28T05:21:01.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.942+0000 7fa44f7fe640 1 -- 172.21.15.23:0/2752124818 learned_addr learned my addr 172.21.15.23:0/2752124818 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:01.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.942+0000 7fa454834640 1 --2- 172.21.15.23:0/2752124818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa450107420 0x7fa4501006b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:01.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.942+0000 7fa44ffff640 1 --2- 172.21.15.23:0/2752124818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4501021c0 0x7fa450101b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:01.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.942+0000 7fa44f7fe640 1 -- 172.21.15.23:0/2752124818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4501021c0 msgr2=0x7fa450101b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:01.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.942+0000 7fa44f7fe640 1 --2- 172.21.15.23:0/2752124818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4501021c0 0x7fa450101b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:01.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.942+0000 7fa44f7fe640 1 -- 172.21.15.23:0/2752124818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa450107420 msgr2=0x7fa4501006b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:01.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.942+0000 7fa44f7fe640 1 --2- 172.21.15.23:0/2752124818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa450107420 0x7fa4501006b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:01.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.943+0000 7fa44f7fe640 1 -- 172.21.15.23:0/2752124818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa450100f40 con 0x7fa450104af0 2024-03-28T05:21:01.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.943+0000 7fa44ffff640 1 --2- 172.21.15.23:0/2752124818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4501021c0 0x7fa450101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:21:01.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.943+0000 7fa44f7fe640 1 --2- 172.21.15.23:0/2752124818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa450104af0 0x7fa450100170 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa44000c970 tx=0x7fa44000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:01.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.943+0000 7fa44d7fa640 1 -- 172.21.15.23:0/2752124818 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa440014070 con 0x7fa450104af0 2024-03-28T05:21:01.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.943+0000 7fa44d7fa640 1 -- 172.21.15.23:0/2752124818 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa440004440 con 0x7fa450104af0 2024-03-28T05:21:01.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.943+0000 7fa44d7fa640 1 -- 172.21.15.23:0/2752124818 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44000f450 con 0x7fa450104af0 2024-03-28T05:21:01.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.943+0000 7fa456297640 1 -- 172.21.15.23:0/2752124818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4501a1a10 con 0x7fa450104af0 2024-03-28T05:21:01.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.943+0000 7fa456297640 1 -- 172.21.15.23:0/2752124818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4501a1f10 con 0x7fa450104af0 2024-03-28T05:21:01.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.944+0000 7fa456297640 1 -- 172.21.15.23:0/2752124818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4500fd6c0 con 0x7fa450104af0 2024-03-28T05:21:01.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.945+0000 7fa44d7fa640 1 -- 172.21.15.23:0/2752124818 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa440017070 con 0x7fa450104af0 2024-03-28T05:21:01.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.945+0000 7fa44d7fa640 1 --2- 172.21.15.23:0/2752124818 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa43c0a2880 0x7fa43c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:01.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.946+0000 7fa44d7fa640 1 -- 172.21.15.23:0/2752124818 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa4400ce400 con 0x7fa450104af0 2024-03-28T05:21:01.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.946+0000 7fa44ffff640 1 --2- 172.21.15.23:0/2752124818 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa43c0a2880 0x7fa43c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:01.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.946+0000 7fa44ffff640 1 --2- 172.21.15.23:0/2752124818 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa43c0a2880 0x7fa43c0a4d40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa4380096c0 tx=0x7fa438000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:01.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:01.950+0000 7fa44d7fa640 1 -- 172.21.15.23:0/2752124818 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa44001e020 con 0x7fa450104af0 2024-03-28T05:21:02.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:02.177+0000 7fa456297640 1 -- 172.21.15.23:0/2752124818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa45002d070 con 0x7fa450104af0 2024-03-28T05:21:02.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:02.178+0000 7fa44d7fa640 1 -- 172.21.15.23:0/2752124818 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9242 (secure 0 0 0) 0x7fa44009b940 con 0x7fa450104af0 2024-03-28T05:21:02.176 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:21:02.176 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/733913400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":733913400},{"type":"v1","addr":"172.21.15.23:6835","nonce":733913400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1972901387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1972901387},{"type":"v1","addr":"172.21.15.23:6837","nonce":1972901387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/2390673788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2390673788},{"type":"v1","addr":"172.21.15.23:6839","nonce":2390673788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/289263156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":289263156},{"type":"v1","addr":"172.21.15.23:6841","nonce":289263156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3313932732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3313932732},{"type":"v1","addr":"172.21.15.98:6835","nonce":3313932732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2695340024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2695340024},{"type":"v1","addr":"172.21.15.113:6835","nonce":2695340024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2750367469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2750367469},{"type":"v1","addr":"172.21.15.98:6837","nonce":2750367469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/3222726274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":3222726274},{"type":"v1","addr":"172.21.15.113:6837","nonce":3222726274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6839/212060489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":212060489},{"type":"v1","addr":"172.21.15.113:6839","nonce":212060489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T05:21:02.177 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 12 2024-03-28T05:21:02.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:02.180+0000 7fa456297640 1 -- 172.21.15.23:0/2752124818 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa43c0a2880 msgr2=0x7fa43c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:02.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:02.180+0000 7fa456297640 1 --2- 172.21.15.23:0/2752124818 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa43c0a2880 0x7fa43c0a4d40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa4380096c0 tx=0x7fa438000bc0 comp rx=0 tx=0).stop 2024-03-28T05:21:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:02.180+0000 7fa456297640 1 -- 172.21.15.23:0/2752124818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa450104af0 msgr2=0x7fa450100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:02.180+0000 7fa456297640 1 --2- 172.21.15.23:0/2752124818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa450104af0 0x7fa450100170 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa44000c970 tx=0x7fa44000ce30 comp rx=0 tx=0).stop 2024-03-28T05:21:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:02.180+0000 7fa456297640 1 -- 172.21.15.23:0/2752124818 shutdown_connections 2024-03-28T05:21:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:02.180+0000 7fa456297640 1 --2- 172.21.15.23:0/2752124818 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa43c0a2880 0x7fa43c0a4d40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:02.180+0000 7fa456297640 1 --2- 172.21.15.23:0/2752124818 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa450107420 0x7fa4501006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:02.180+0000 7fa456297640 1 --2- 172.21.15.23:0/2752124818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa450104af0 0x7fa450100170 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:02.180+0000 7fa456297640 1 --2- 172.21.15.23:0/2752124818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4501021c0 0x7fa450101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:02.180+0000 7fa456297640 1 -- 172.21.15.23:0/2752124818 >> 172.21.15.23:0/2752124818 conn(0x7fa4500fbbd0 msgr2=0x7fa450102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:02.180+0000 7fa456297640 1 -- 172.21.15.23:0/2752124818 shutdown_connections 2024-03-28T05:21:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:02.180+0000 7fa456297640 1 -- 172.21.15.23:0/2752124818 wait complete. 2024-03-28T05:21:02.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:02 smithi098 ceph-mon[36087]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:21:02.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:02 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:02.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:02 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:02.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:02 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:02.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:02 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:21:02.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:02 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:21:02.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:02 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:21:02.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:02 smithi098 ceph-mon[36087]: Deploying daemon mds.b on smithi098 2024-03-28T05:21:02.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:02 smithi098 ceph-mon[36087]: mds.? [v2:172.21.15.23:6840/289263156,v1:172.21.15.23:6841/289263156] up:boot 2024-03-28T05:21:02.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:02 smithi098 ceph-mon[36087]: fsmap 11 up:standby 2024-03-28T05:21:02.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:02 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T05:21:02.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:02 smithi113 ceph-mon[37164]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:21:02.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:02 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:02.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:02 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:02.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:02 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:02.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:02 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:21:02.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:02 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:21:02.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:02 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:21:02.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:02 smithi113 ceph-mon[37164]: Deploying daemon mds.b on smithi098 2024-03-28T05:21:02.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:02 smithi113 ceph-mon[37164]: mds.? [v2:172.21.15.23:6840/289263156,v1:172.21.15.23:6841/289263156] up:boot 2024-03-28T05:21:02.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:02 smithi113 ceph-mon[37164]: fsmap 11 up:standby 2024-03-28T05:21:02.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:02 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T05:21:02.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:02 smithi023 ceph-mon[31954]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:21:02.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:02 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:02.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:02 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:02.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:02 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:02.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:02 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T05:21:02.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:02 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T05:21:02.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:02 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:21:02.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:02 smithi023 ceph-mon[31954]: Deploying daemon mds.b on smithi098 2024-03-28T05:21:02.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:02 smithi023 ceph-mon[31954]: mds.? [v2:172.21.15.23:6840/289263156,v1:172.21.15.23:6841/289263156] up:boot 2024-03-28T05:21:02.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:02 smithi023 ceph-mon[31954]: fsmap 11 up:standby 2024-03-28T05:21:02.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:02 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T05:21:03.308 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:03 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2752124818' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:21:03.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:03 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2752124818' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:21:03.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:03 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2752124818' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:21:03.615 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 28 05:21:03 smithi098 systemd[1]: Starting Ceph mds.b for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:21:04.005 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 28 05:21:03 smithi098 podman[65785]: 2024-03-28 05:21:03.611954607 +0000 UTC m=+0.018028735 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:21:04.006 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 28 05:21:03 smithi098 podman[65785]: 2024-03-28 05:21:03.737437702 +0000 UTC m=+0.143511827 container create 61e83ec3302d427d3a1c78950c5577f797524c0da590b6c4b861b75f8e72fb58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-28T05:21:04.006 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 28 05:21:04 smithi098 podman[65785]: 2024-03-28 05:21:04.007224006 +0000 UTC m=+0.413298135 container init 61e83ec3302d427d3a1c78950c5577f797524c0da590b6c4b861b75f8e72fb58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-28T05:21:04.191 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:21:04.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.318+0000 7f1d47a9a640 1 -- 172.21.15.23:0/727099979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d401028a0 msgr2=0x7f1d40102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:04.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.318+0000 7f1d47a9a640 1 --2- 172.21.15.23:0/727099979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d401028a0 0x7f1d40102d20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1d30009900 tx=0x7f1d3002bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:04.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.318+0000 7f1d47a9a640 1 -- 172.21.15.23:0/727099979 shutdown_connections 2024-03-28T05:21:04.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.318+0000 7f1d47a9a640 1 --2- 172.21.15.23:0/727099979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d40103260 0x7f1d40109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:04.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.318+0000 7f1d47a9a640 1 --2- 172.21.15.23:0/727099979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d401028a0 0x7f1d40102d20 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:04.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.318+0000 7f1d47a9a640 1 --2- 172.21.15.23:0/727099979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d401016b0 0x7f1d40101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:04.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.318+0000 7f1d47a9a640 1 -- 172.21.15.23:0/727099979 >> 172.21.15.23:0/727099979 conn(0x7f1d400fcc80 msgr2=0x7f1d400ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:04.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.318+0000 7f1d47a9a640 1 -- 172.21.15.23:0/727099979 shutdown_connections 2024-03-28T05:21:04.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.318+0000 7f1d47a9a640 1 -- 172.21.15.23:0/727099979 wait complete. 2024-03-28T05:21:04.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.319+0000 7f1d47a9a640 1 Processor -- start 2024-03-28T05:21:04.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.319+0000 7f1d47a9a640 1 -- start start 2024-03-28T05:21:04.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.319+0000 7f1d47a9a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d401016b0 0x7f1d40197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:04.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.319+0000 7f1d47a9a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d401028a0 0x7f1d40197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:04.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.319+0000 7f1d47a9a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d40103260 0x7f1d4019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:04.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.319+0000 7f1d47a9a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4010d330 con 0x7f1d40103260 2024-03-28T05:21:04.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.319+0000 7f1d47a9a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4010d1b0 con 0x7f1d401028a0 2024-03-28T05:21:04.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.319+0000 7f1d47a9a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4010d4b0 con 0x7f1d401016b0 2024-03-28T05:21:04.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.319+0000 7f1d46037640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d40103260 0x7f1d4019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:04.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.319+0000 7f1d46037640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d40103260 0x7f1d4019efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43404/0 (socket says 172.21.15.23:43404) 2024-03-28T05:21:04.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.319+0000 7f1d46037640 1 -- 172.21.15.23:0/310722785 learned_addr learned my addr 172.21.15.23:0/310722785 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:04.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.319+0000 7f1d45836640 1 --2- 172.21.15.23:0/310722785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d401016b0 0x7f1d40197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:04.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.319+0000 7f1d45035640 1 --2- 172.21.15.23:0/310722785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d401028a0 0x7f1d40197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:04.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.320+0000 7f1d46037640 1 -- 172.21.15.23:0/310722785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d401016b0 msgr2=0x7f1d40197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:04.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.320+0000 7f1d46037640 1 --2- 172.21.15.23:0/310722785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d401016b0 0x7f1d40197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:04.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.320+0000 7f1d46037640 1 -- 172.21.15.23:0/310722785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d401028a0 msgr2=0x7f1d40197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:04.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.320+0000 7f1d46037640 1 --2- 172.21.15.23:0/310722785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d401028a0 0x7f1d40197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:04.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.320+0000 7f1d46037640 1 -- 172.21.15.23:0/310722785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d4019f620 con 0x7f1d40103260 2024-03-28T05:21:04.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.320+0000 7f1d46037640 1 --2- 172.21.15.23:0/310722785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d40103260 0x7f1d4019efb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1d3800eb10 tx=0x7f1d3800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:04.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.320+0000 7f1d36ffd640 1 -- 172.21.15.23:0/310722785 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d38004b20 con 0x7f1d40103260 2024-03-28T05:21:04.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.320+0000 7f1d36ffd640 1 -- 172.21.15.23:0/310722785 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d38004cc0 con 0x7f1d40103260 2024-03-28T05:21:04.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.320+0000 7f1d36ffd640 1 -- 172.21.15.23:0/310722785 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d38010760 con 0x7f1d40103260 2024-03-28T05:21:04.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.320+0000 7f1d47a9a640 1 -- 172.21.15.23:0/310722785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d4019f910 con 0x7f1d40103260 2024-03-28T05:21:04.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.320+0000 7f1d47a9a640 1 -- 172.21.15.23:0/310722785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d4019fe30 con 0x7f1d40103260 2024-03-28T05:21:04.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.321+0000 7f1d47a9a640 1 -- 172.21.15.23:0/310722785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d401089d0 con 0x7f1d40103260 2024-03-28T05:21:04.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.322+0000 7f1d36ffd640 1 -- 172.21.15.23:0/310722785 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1d38010900 con 0x7f1d40103260 2024-03-28T05:21:04.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.323+0000 7f1d36ffd640 1 --2- 172.21.15.23:0/310722785 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1d2c0a2880 0x7f1d2c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:04.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.323+0000 7f1d45836640 1 --2- 172.21.15.23:0/310722785 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1d2c0a2880 0x7f1d2c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:04.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.323+0000 7f1d36ffd640 1 -- 172.21.15.23:0/310722785 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1d380cece0 con 0x7f1d40103260 2024-03-28T05:21:04.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.323+0000 7f1d45836640 1 --2- 172.21.15.23:0/310722785 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1d2c0a2880 0x7f1d2c0a4d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1d40102700 tx=0x7f1d3c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:04.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.326+0000 7f1d36ffd640 1 -- 172.21.15.23:0/310722785 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1d38053740 con 0x7f1d40103260 2024-03-28T05:21:04.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:04 smithi098 ceph-mon[36087]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:21:04.365 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 28 05:21:04 smithi098 podman[65785]: 2024-03-28 05:21:04.010017921 +0000 UTC m=+0.416092056 container start 61e83ec3302d427d3a1c78950c5577f797524c0da590b6c4b861b75f8e72fb58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-b, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-28T05:21:04.365 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 28 05:21:04 smithi098 ceph-mds[65804]: -- 172.21.15.98:0/375995751 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563907e9ba40 con 0x563907e6ec00 2024-03-28T05:21:04.365 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 28 05:21:04 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-b[65800]: starting mds.b at 2024-03-28T05:21:04.365 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 28 05:21:04 smithi098 bash[65785]: 61e83ec3302d427d3a1c78950c5577f797524c0da590b6c4b861b75f8e72fb58 2024-03-28T05:21:04.365 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 28 05:21:04 smithi098 systemd[1]: Started Ceph mds.b for bf37ebc4-ecc1-11ee-95d0-87774f69a715. 2024-03-28T05:21:04.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:04 smithi113 ceph-mon[37164]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:21:04.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:04 smithi023 ceph-mon[31954]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:21:04.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.551+0000 7f1d47a9a640 1 -- 172.21.15.23:0/310722785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d4002d070 con 0x7f1d40103260 2024-03-28T05:21:04.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.552+0000 7f1d36ffd640 1 -- 172.21.15.23:0/310722785 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10031 (secure 0 0 0) 0x7f1d3809c220 con 0x7f1d40103260 2024-03-28T05:21:04.551 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:21:04.551 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/733913400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":733913400},{"type":"v1","addr":"172.21.15.23:6835","nonce":733913400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1972901387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1972901387},{"type":"v1","addr":"172.21.15.23:6837","nonce":1972901387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/2390673788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2390673788},{"type":"v1","addr":"172.21.15.23:6839","nonce":2390673788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/289263156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":289263156},{"type":"v1","addr":"172.21.15.23:6841","nonce":289263156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3313932732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3313932732},{"type":"v1","addr":"172.21.15.98:6835","nonce":3313932732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2695340024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2695340024},{"type":"v1","addr":"172.21.15.113:6835","nonce":2695340024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2750367469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2750367469},{"type":"v1","addr":"172.21.15.98:6837","nonce":2750367469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/3222726274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":3222726274},{"type":"v1","addr":"172.21.15.113:6837","nonce":3222726274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24473,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/144113007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":144113007},{"type":"v1","addr":"172.21.15.98:6839","nonce":144113007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6839/212060489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":212060489},{"type":"v1","addr":"172.21.15.113:6839","nonce":212060489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T05:21:04.551 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 13 2024-03-28T05:21:04.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.554+0000 7f1d47a9a640 1 -- 172.21.15.23:0/310722785 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1d2c0a2880 msgr2=0x7f1d2c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:04.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.554+0000 7f1d47a9a640 1 --2- 172.21.15.23:0/310722785 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1d2c0a2880 0x7f1d2c0a4d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1d40102700 tx=0x7f1d3c005f90 comp rx=0 tx=0).stop 2024-03-28T05:21:04.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.554+0000 7f1d47a9a640 1 -- 172.21.15.23:0/310722785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d40103260 msgr2=0x7f1d4019efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:04.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.554+0000 7f1d47a9a640 1 --2- 172.21.15.23:0/310722785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d40103260 0x7f1d4019efb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1d3800eb10 tx=0x7f1d3800c2d0 comp rx=0 tx=0).stop 2024-03-28T05:21:04.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.554+0000 7f1d47a9a640 1 -- 172.21.15.23:0/310722785 shutdown_connections 2024-03-28T05:21:04.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.554+0000 7f1d47a9a640 1 --2- 172.21.15.23:0/310722785 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1d2c0a2880 0x7f1d2c0a4d40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:04.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.554+0000 7f1d47a9a640 1 --2- 172.21.15.23:0/310722785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d40103260 0x7f1d4019efb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:04.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.554+0000 7f1d47a9a640 1 --2- 172.21.15.23:0/310722785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d401028a0 0x7f1d40197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:04.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.554+0000 7f1d47a9a640 1 --2- 172.21.15.23:0/310722785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d401016b0 0x7f1d40197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:04.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.554+0000 7f1d47a9a640 1 -- 172.21.15.23:0/310722785 >> 172.21.15.23:0/310722785 conn(0x7f1d400fcc80 msgr2=0x7f1d40105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:04.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.554+0000 7f1d47a9a640 1 -- 172.21.15.23:0/310722785 shutdown_connections 2024-03-28T05:21:04.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.554+0000 7f1d47a9a640 1 -- 172.21.15.23:0/310722785 wait complete. 2024-03-28T05:21:04.565 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T05:21:04.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.671+0000 7ff3035a9640 1 -- 172.21.15.23:0/2690698353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2fc102170 msgr2=0x7ff2fc108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:04.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.671+0000 7ff3035a9640 1 --2- 172.21.15.23:0/2690698353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2fc102170 0x7ff2fc108a00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4009ca0 tx=0x7ff2f402c0f0 comp rx=0 tx=0).stop 2024-03-28T05:21:04.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.671+0000 7ff3035a9640 1 -- 172.21.15.23:0/2690698353 shutdown_connections 2024-03-28T05:21:04.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.671+0000 7ff3035a9640 1 --2- 172.21.15.23:0/2690698353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2fc102170 0x7ff2fc108a00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:04.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.671+0000 7ff3035a9640 1 --2- 172.21.15.23:0/2690698353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2fc1017b0 0x7ff2fc101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:04.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.671+0000 7ff3035a9640 1 --2- 172.21.15.23:0/2690698353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2fc1005c0 0x7ff2fc1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:04.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.671+0000 7ff3035a9640 1 -- 172.21.15.23:0/2690698353 >> 172.21.15.23:0/2690698353 conn(0x7ff2fc0fbb30 msgr2=0x7ff2fc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:04.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.671+0000 7ff3035a9640 1 -- 172.21.15.23:0/2690698353 shutdown_connections 2024-03-28T05:21:04.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.671+0000 7ff3035a9640 1 -- 172.21.15.23:0/2690698353 wait complete. 2024-03-28T05:21:04.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.672+0000 7ff3035a9640 1 Processor -- start 2024-03-28T05:21:04.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.672+0000 7ff3035a9640 1 -- start start 2024-03-28T05:21:04.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.672+0000 7ff3035a9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2fc1005c0 0x7ff2fc197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:04.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.672+0000 7ff3035a9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2fc1017b0 0x7ff2fc197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:04.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.672+0000 7ff3035a9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2fc102170 0x7ff2fc19eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:04.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.672+0000 7ff3035a9640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc10d360 con 0x7ff2fc1005c0 2024-03-28T05:21:04.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.672+0000 7ff3035a9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc10d1e0 con 0x7ff2fc1017b0 2024-03-28T05:21:04.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.672+0000 7ff3035a9640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc10d4e0 con 0x7ff2fc102170 2024-03-28T05:21:04.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.672+0000 7ff301345640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2fc1005c0 0x7ff2fc197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:04.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.672+0000 7ff301345640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2fc1005c0 0x7ff2fc197930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43422/0 (socket says 172.21.15.23:43422) 2024-03-28T05:21:04.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.672+0000 7ff301345640 1 -- 172.21.15.23:0/2989634456 learned_addr learned my addr 172.21.15.23:0/2989634456 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:04.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.672+0000 7ff300b44640 1 --2- 172.21.15.23:0/2989634456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2fc1017b0 0x7ff2fc197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:04.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.672+0000 7ff301b46640 1 --2- 172.21.15.23:0/2989634456 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2fc102170 0x7ff2fc19eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:04.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.672+0000 7ff301345640 1 -- 172.21.15.23:0/2989634456 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2fc102170 msgr2=0x7ff2fc19eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:04.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.672+0000 7ff301345640 1 --2- 172.21.15.23:0/2989634456 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2fc102170 0x7ff2fc19eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:04.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.672+0000 7ff301345640 1 -- 172.21.15.23:0/2989634456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2fc1017b0 msgr2=0x7ff2fc197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:04.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.672+0000 7ff301345640 1 --2- 172.21.15.23:0/2989634456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2fc1017b0 0x7ff2fc197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:04.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.672+0000 7ff301345640 1 -- 172.21.15.23:0/2989634456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2fc19f550 con 0x7ff2fc1005c0 2024-03-28T05:21:04.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.672+0000 7ff301345640 1 --2- 172.21.15.23:0/2989634456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2fc1005c0 0x7ff2fc197930 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff2f800e9c0 tx=0x7ff2f800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:04.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.673+0000 7ff2f27fc640 1 -- 172.21.15.23:0/2989634456 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f800cd80 con 0x7ff2fc1005c0 2024-03-28T05:21:04.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.673+0000 7ff2f27fc640 1 -- 172.21.15.23:0/2989634456 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff2f80035f0 con 0x7ff2fc1005c0 2024-03-28T05:21:04.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.673+0000 7ff3035a9640 1 -- 172.21.15.23:0/2989634456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2fc19f840 con 0x7ff2fc1005c0 2024-03-28T05:21:04.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.673+0000 7ff2f27fc640 1 -- 172.21.15.23:0/2989634456 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8010690 con 0x7ff2fc1005c0 2024-03-28T05:21:04.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.673+0000 7ff3035a9640 1 -- 172.21.15.23:0/2989634456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2fc19fde0 con 0x7ff2fc1005c0 2024-03-28T05:21:04.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.674+0000 7ff3035a9640 1 -- 172.21.15.23:0/2989634456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2fc1078e0 con 0x7ff2fc1005c0 2024-03-28T05:21:04.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.675+0000 7ff2f27fc640 1 -- 172.21.15.23:0/2989634456 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff2f80037a0 con 0x7ff2fc1005c0 2024-03-28T05:21:04.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.675+0000 7ff2f27fc640 1 --2- 172.21.15.23:0/2989634456 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff2e80a2880 0x7ff2e80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:04.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.675+0000 7ff300b44640 1 --2- 172.21.15.23:0/2989634456 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff2e80a2880 0x7ff2e80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:04.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.676+0000 7ff2f27fc640 1 -- 172.21.15.23:0/2989634456 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff2f80ce9c0 con 0x7ff2fc1005c0 2024-03-28T05:21:04.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.676+0000 7ff300b44640 1 --2- 172.21.15.23:0/2989634456 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff2e80a2880 0x7ff2e80a4d40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc198e40 tx=0x7ff2ec005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:04.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.679+0000 7ff2f27fc640 1 -- 172.21.15.23:0/2989634456 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff2f809bf90 con 0x7ff2fc1005c0 2024-03-28T05:21:04.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.867+0000 7ff3035a9640 1 -- 172.21.15.23:0/2989634456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2fc0622d0 con 0x7ff2fc1005c0 2024-03-28T05:21:04.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.868+0000 7ff2f27fc640 1 -- 172.21.15.23:0/2989634456 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18410 (secure 0 0 0) 0x7ff2f809b890 con 0x7ff2fc1005c0 2024-03-28T05:21:04.866 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:21:04.867 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":81,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","created":"2024-03-28T05:13:13.453653+0000","modified":"2024-03-28T05:20:21.227349+0000","last_up_change":"2024-03-28T05:20:20.221527+0000","last_in_change":"2024-03-28T05:20:00.365478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T05:16:32.986974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18150942-59a4-4c3b-83d4-c1fc3ccf1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6803","nonce":1219956881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6805","nonce":1219956881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6809","nonce":1219956881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6807","nonce":1219956881}]},"public_addr":"172.21.15.23:6803/1219956881","cluster_addr":"172.21.15.23:6805/1219956881","heartbeat_back_addr":"172.21.15.23:6809/1219956881","heartbeat_front_addr":"172.21.15.23:6807/1219956881","state":["exists","up"]},{"osd":1,"uuid":"ba6d1a95-5135-4670-95f9-7878c0e7bd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6801","nonce":555491234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6803","nonce":555491234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6807","nonce":555491234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6805","nonce":555491234}]},"public_addr":"172.21.15.98:6801/555491234","cluster_addr":"172.21.15.98:6803/555491234","heartbeat_back_addr":"172.21.15.98:6807/555491234","heartbeat_front_addr":"172.21.15.98:6805/555491234","state":["exists","up"]},{"osd":2,"uuid":"873bdd60-2a47-403b-b1f4-ee9226a95067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6801","nonce":1535856914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6803","nonce":1535856914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6807","nonce":1535856914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6805","nonce":1535856914}]},"public_addr":"172.21.15.113:6801/1535856914","cluster_addr":"172.21.15.113:6803/1535856914","heartbeat_back_addr":"172.21.15.113:6807/1535856914","heartbeat_front_addr":"172.21.15.113:6805/1535856914","state":["exists","up"]},{"osd":3,"uuid":"83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6811","nonce":2949415020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6813","nonce":2949415020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6817","nonce":2949415020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6815","nonce":2949415020}]},"public_addr":"172.21.15.23:6811/2949415020","cluster_addr":"172.21.15.23:6813/2949415020","heartbeat_back_addr":"172.21.15.23:6817/2949415020","heartbeat_front_addr":"172.21.15.23:6815/2949415020","state":["exists","up"]},{"osd":4,"uuid":"76a90bac-b221-40a3-8238-40e8049cec36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6809","nonce":2615902819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6811","nonce":2615902819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6815","nonce":2615902819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6813","nonce":2615902819}]},"public_addr":"172.21.15.98:6809/2615902819","cluster_addr":"172.21.15.98:6811/2615902819","heartbeat_back_addr":"172.21.15.98:6815/2615902819","heartbeat_front_addr":"172.21.15.98:6813/2615902819","state":["exists","up"]},{"osd":5,"uuid":"b53fdb55-a8df-40a8-bd1b-8bcef634e26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6809","nonce":625271528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6811","nonce":625271528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6815","nonce":625271528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6813","nonce":625271528}]},"public_addr":"172.21.15.113:6809/625271528","cluster_addr":"172.21.15.113:6811/625271528","heartbeat_back_addr":"172.21.15.113:6815/625271528","heartbeat_front_addr":"172.21.15.113:6813/625271528","state":["exists","up"]},{"osd":6,"uuid":"f00283d2-8d67-4e97-a575-9a9d3b55d5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6819","nonce":4188464176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6821","nonce":4188464176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6825","nonce":4188464176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6823","nonce":4188464176}]},"public_addr":"172.21.15.23:6819/4188464176","cluster_addr":"172.21.15.23:6821/4188464176","heartbeat_back_addr":"172.21.15.23:6825/4188464176","heartbeat_front_addr":"172.21.15.23:6823/4188464176","state":["exists","up"]},{"osd":7,"uuid":"edc008a7-b07b-40a8-88b7-1614ba38a775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6817","nonce":1985546028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6819","nonce":1985546028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6823","nonce":1985546028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6821","nonce":1985546028}]},"public_addr":"172.21.15.98:6817/1985546028","cluster_addr":"172.21.15.98:6819/1985546028","heartbeat_back_addr":"172.21.15.98:6823/1985546028","heartbeat_front_addr":"172.21.15.98:6821/1985546028","state":["exists","up"]},{"osd":8,"uuid":"56e7b2d1-81f0-47b2-8638-f29832bed8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6817","nonce":3534876081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6819","nonce":3534876081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6823","nonce":3534876081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6821","nonce":3534876081}]},"public_addr":"172.21.15.113:6817/3534876081","cluster_addr":"172.21.15.113:6819/3534876081","heartbeat_back_addr":"172.21.15.113:6823/3534876081","heartbeat_front_addr":"172.21.15.113:6821/3534876081","state":["exists","up"]},{"osd":9,"uuid":"a2e90caa-6e9c-42bf-9f18-81f7f47880cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6827","nonce":672264207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6829","nonce":672264207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6833","nonce":672264207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6831","nonce":672264207}]},"public_addr":"172.21.15.23:6827/672264207","cluster_addr":"172.21.15.23:6829/672264207","heartbeat_back_addr":"172.21.15.23:6833/672264207","heartbeat_front_addr":"172.21.15.23:6831/672264207","state":["exists","up"]},{"osd":10,"uuid":"0eda917f-020c-437f-95ae-b26203f6c68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6825","nonce":2699920595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6827","nonce":2699920595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6831","nonce":2699920595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6829","nonce":2699920595}]},"public_addr":"172.21.15.98:6825/2699920595","cluster_addr":"172.21.15.98:6827/2699920595","heartbeat_back_addr":"172.21.15.98:6831/2699920595","heartbeat_front_addr":"172.21.15.98:6829/2699920595","state":["exists","up"]},{"osd":11,"uuid":"aae78501-78a8-469a-be20-3006f5ec5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6825","nonce":2113644020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6827","nonce":2113644020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6831","nonce":2113644020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6829","nonce":2113644020}]},"public_addr":"172.21.15.113:6825/2113644020","cluster_addr":"172.21.15.113:6827/2113644020","heartbeat_back_addr":"172.21.15.113:6831/2113644020","heartbeat_front_addr":"172.21.15.113:6829/2113644020","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:15:43.527475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:07.006538+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:30.594176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:55.810999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:20.595811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:45.474138+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:10.500409+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:35.787638+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:01.001188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:26.761115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:52.952752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:20:18.559832+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:6800/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1876451381":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2253559862":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2653562762":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/3337463830":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6800/1735221490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/2784771635":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2389902490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6801/1735221490":"2024-03-29T05:13:42.579607+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T05:21:04.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.870+0000 7ff3035a9640 1 -- 172.21.15.23:0/2989634456 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff2e80a2880 msgr2=0x7ff2e80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:04.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.870+0000 7ff3035a9640 1 --2- 172.21.15.23:0/2989634456 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff2e80a2880 0x7ff2e80a4d40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc198e40 tx=0x7ff2ec005f90 comp rx=0 tx=0).stop 2024-03-28T05:21:04.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.870+0000 7ff3035a9640 1 -- 172.21.15.23:0/2989634456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2fc1005c0 msgr2=0x7ff2fc197930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:04.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.870+0000 7ff3035a9640 1 --2- 172.21.15.23:0/2989634456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2fc1005c0 0x7ff2fc197930 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff2f800e9c0 tx=0x7ff2f800ee80 comp rx=0 tx=0).stop 2024-03-28T05:21:04.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.870+0000 7ff3035a9640 1 -- 172.21.15.23:0/2989634456 shutdown_connections 2024-03-28T05:21:04.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.870+0000 7ff3035a9640 1 --2- 172.21.15.23:0/2989634456 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff2e80a2880 0x7ff2e80a4d40 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:04.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.870+0000 7ff3035a9640 1 --2- 172.21.15.23:0/2989634456 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2fc102170 0x7ff2fc19eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:04.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.870+0000 7ff3035a9640 1 --2- 172.21.15.23:0/2989634456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2fc1017b0 0x7ff2fc197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:04.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.870+0000 7ff3035a9640 1 --2- 172.21.15.23:0/2989634456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2fc1005c0 0x7ff2fc197930 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:04.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.870+0000 7ff3035a9640 1 -- 172.21.15.23:0/2989634456 >> 172.21.15.23:0/2989634456 conn(0x7ff2fc0fbb30 msgr2=0x7ff2fc104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:04.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.870+0000 7ff3035a9640 1 -- 172.21.15.23:0/2989634456 shutdown_connections 2024-03-28T05:21:04.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.870+0000 7ff3035a9640 1 -- 172.21.15.23:0/2989634456 wait complete. 2024-03-28T05:21:04.881 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T05:16:32.986974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T05:21:04.881 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T05:21:04.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.998+0000 7fd9f0910640 1 -- 172.21.15.23:0/2879605781 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9ec102950 msgr2=0x7fd9ec102dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:05.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.998+0000 7fd9f0910640 1 --2- 172.21.15.23:0/2879605781 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9ec102950 0x7fd9ec102dd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0009930 tx=0x7fd9e002bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:05.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.998+0000 7fd9f0910640 1 -- 172.21.15.23:0/2879605781 shutdown_connections 2024-03-28T05:21:05.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.998+0000 7fd9f0910640 1 --2- 172.21.15.23:0/2879605781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9ec103310 0x7fd9ec109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.998+0000 7fd9f0910640 1 --2- 172.21.15.23:0/2879605781 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9ec102950 0x7fd9ec102dd0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.998+0000 7fd9f0910640 1 --2- 172.21.15.23:0/2879605781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ec0ff980 0x7fd9ec0ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.998+0000 7fd9f0910640 1 -- 172.21.15.23:0/2879605781 >> 172.21.15.23:0/2879605781 conn(0x7fd9ec0fb290 msgr2=0x7fd9ec0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:05.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.998+0000 7fd9f0910640 1 -- 172.21.15.23:0/2879605781 shutdown_connections 2024-03-28T05:21:05.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.998+0000 7fd9f0910640 1 -- 172.21.15.23:0/2879605781 wait complete. 2024-03-28T05:21:05.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.998+0000 7fd9f0910640 1 Processor -- start 2024-03-28T05:21:05.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.999+0000 7fd9f0910640 1 -- start start 2024-03-28T05:21:05.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.999+0000 7fd9f0910640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9ec0ff980 0x7fd9ec1022c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:05.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.999+0000 7fd9f0910640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ec102950 0x7fd9ec100900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:05.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.999+0000 7fd9f0910640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9ec103310 0x7fd9ec100e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:05.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.999+0000 7fd9f0910640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec10d3b0 con 0x7fd9ec103310 2024-03-28T05:21:05.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.999+0000 7fd9f0910640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec10d230 con 0x7fd9ec102950 2024-03-28T05:21:05.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.999+0000 7fd9f0910640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec10d530 con 0x7fd9ec0ff980 2024-03-28T05:21:05.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.999+0000 7fd9ead9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9ec103310 0x7fd9ec100e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:05.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.999+0000 7fd9ead9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9ec103310 0x7fd9ec100e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43446/0 (socket says 172.21.15.23:43446) 2024-03-28T05:21:05.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.999+0000 7fd9ead9d640 1 -- 172.21.15.23:0/2939284032 learned_addr learned my addr 172.21.15.23:0/2939284032 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:05.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.999+0000 7fd9ea59c640 1 --2- 172.21.15.23:0/2939284032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9ec0ff980 0x7fd9ec1022c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:05.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.999+0000 7fd9e9d9b640 1 --2- 172.21.15.23:0/2939284032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ec102950 0x7fd9ec100900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:05.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.999+0000 7fd9ead9d640 1 -- 172.21.15.23:0/2939284032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9ec0ff980 msgr2=0x7fd9ec1022c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:05.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.999+0000 7fd9ead9d640 1 --2- 172.21.15.23:0/2939284032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9ec0ff980 0x7fd9ec1022c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.999+0000 7fd9ead9d640 1 -- 172.21.15.23:0/2939284032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ec102950 msgr2=0x7fd9ec100900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:05.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.999+0000 7fd9ead9d640 1 --2- 172.21.15.23:0/2939284032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ec102950 0x7fd9ec100900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.999+0000 7fd9ead9d640 1 -- 172.21.15.23:0/2939284032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ec1016d0 con 0x7fd9ec103310 2024-03-28T05:21:05.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:04.999+0000 7fd9ead9d640 1 --2- 172.21.15.23:0/2939284032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9ec103310 0x7fd9ec100e40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd9e400eb10 tx=0x7fd9e400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:05.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.000+0000 7fd9db7fe640 1 -- 172.21.15.23:0/2939284032 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4004b20 con 0x7fd9ec103310 2024-03-28T05:21:05.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.000+0000 7fd9db7fe640 1 -- 172.21.15.23:0/2939284032 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9e4004cc0 con 0x7fd9ec103310 2024-03-28T05:21:05.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.000+0000 7fd9f0910640 1 -- 172.21.15.23:0/2939284032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9ec1a1b20 con 0x7fd9ec103310 2024-03-28T05:21:05.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.000+0000 7fd9db7fe640 1 -- 172.21.15.23:0/2939284032 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4010760 con 0x7fd9ec103310 2024-03-28T05:21:05.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.000+0000 7fd9f0910640 1 -- 172.21.15.23:0/2939284032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ec1a2040 con 0x7fd9ec103310 2024-03-28T05:21:05.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.001+0000 7fd9f0910640 1 -- 172.21.15.23:0/2939284032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9ec0688e0 con 0x7fd9ec103310 2024-03-28T05:21:05.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.002+0000 7fd9db7fe640 1 -- 172.21.15.23:0/2939284032 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd9e4005020 con 0x7fd9ec103310 2024-03-28T05:21:05.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.003+0000 7fd9db7fe640 1 --2- 172.21.15.23:0/2939284032 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd9d40a2800 0x7fd9d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:05.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.003+0000 7fd9ea59c640 1 --2- 172.21.15.23:0/2939284032 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd9d40a2800 0x7fd9d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:05.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.003+0000 7fd9db7fe640 1 -- 172.21.15.23:0/2939284032 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd9e40cecc0 con 0x7fd9ec103310 2024-03-28T05:21:05.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.003+0000 7fd9ea59c640 1 --2- 172.21.15.23:0/2939284032 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd9d40a2800 0x7fd9d40a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec069a70 tx=0x7fd9dc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:05.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.008+0000 7fd9db7fe640 1 -- 172.21.15.23:0/2939284032 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd9e409c200 con 0x7fd9ec103310 2024-03-28T05:21:05.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.197+0000 7fd9f0910640 1 -- 172.21.15.23:0/2939284032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd9ec02d0a0 con 0x7fd9ec103310 2024-03-28T05:21:05.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.198+0000 7fd9db7fe640 1 -- 172.21.15.23:0/2939284032 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd9e409bb00 con 0x7fd9ec103310 2024-03-28T05:21:05.196 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-28T05:21:05.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.200+0000 7fd9f0910640 1 -- 172.21.15.23:0/2939284032 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd9d40a2800 msgr2=0x7fd9d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:05.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.200+0000 7fd9f0910640 1 --2- 172.21.15.23:0/2939284032 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd9d40a2800 0x7fd9d40a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec069a70 tx=0x7fd9dc005f90 comp rx=0 tx=0).stop 2024-03-28T05:21:05.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.200+0000 7fd9f0910640 1 -- 172.21.15.23:0/2939284032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9ec103310 msgr2=0x7fd9ec100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:05.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.200+0000 7fd9f0910640 1 --2- 172.21.15.23:0/2939284032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9ec103310 0x7fd9ec100e40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd9e400eb10 tx=0x7fd9e400c2d0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.200+0000 7fd9f0910640 1 -- 172.21.15.23:0/2939284032 shutdown_connections 2024-03-28T05:21:05.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.200+0000 7fd9f0910640 1 --2- 172.21.15.23:0/2939284032 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd9d40a2800 0x7fd9d40a4cc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.200+0000 7fd9f0910640 1 --2- 172.21.15.23:0/2939284032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9ec103310 0x7fd9ec100e40 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.200+0000 7fd9f0910640 1 --2- 172.21.15.23:0/2939284032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ec102950 0x7fd9ec100900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.200+0000 7fd9f0910640 1 --2- 172.21.15.23:0/2939284032 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9ec0ff980 0x7fd9ec1022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.200+0000 7fd9f0910640 1 -- 172.21.15.23:0/2939284032 >> 172.21.15.23:0/2939284032 conn(0x7fd9ec0fb290 msgr2=0x7fd9ec105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:05.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.200+0000 7fd9f0910640 1 -- 172.21.15.23:0/2939284032 shutdown_connections 2024-03-28T05:21:05.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.200+0000 7fd9f0910640 1 -- 172.21.15.23:0/2939284032 wait complete. 2024-03-28T05:21:05.209 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T05:21:05.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.318+0000 7f9ae3522640 1 -- 172.21.15.23:0/2961843190 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9adc104b10 msgr2=0x7f9adc106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:05.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.318+0000 7f9ae3522640 1 --2- 172.21.15.23:0/2961843190 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9adc104b10 0x7f9adc106f00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9acc009900 tx=0x7f9acc02bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:05.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.318+0000 7f9ae3522640 1 -- 172.21.15.23:0/2961843190 shutdown_connections 2024-03-28T05:21:05.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.318+0000 7f9ae3522640 1 --2- 172.21.15.23:0/2961843190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9adc107440 0x7f9adc109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.318+0000 7f9ae3522640 1 --2- 172.21.15.23:0/2961843190 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9adc104b10 0x7f9adc106f00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.318+0000 7f9ae3522640 1 --2- 172.21.15.23:0/2961843190 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9adc1021e0 0x7f9adc1045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.318+0000 7f9ae3522640 1 -- 172.21.15.23:0/2961843190 >> 172.21.15.23:0/2961843190 conn(0x7f9adc0fbb30 msgr2=0x7f9adc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:05.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.318+0000 7f9ae3522640 1 -- 172.21.15.23:0/2961843190 shutdown_connections 2024-03-28T05:21:05.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.318+0000 7f9ae3522640 1 -- 172.21.15.23:0/2961843190 wait complete. 2024-03-28T05:21:05.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.318+0000 7f9ae3522640 1 Processor -- start 2024-03-28T05:21:05.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.318+0000 7f9ae3522640 1 -- start start 2024-03-28T05:21:05.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.319+0000 7f9ae3522640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9adc1021e0 0x7f9adc197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:05.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.319+0000 7f9ae3522640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9adc104b10 0x7f9adc197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:05.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.319+0000 7f9ae3522640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9adc107440 0x7f9adc19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:05.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.319+0000 7f9ae3522640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc10d330 con 0x7f9adc104b10 2024-03-28T05:21:05.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.319+0000 7f9ae3522640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc10d1b0 con 0x7f9adc107440 2024-03-28T05:21:05.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.319+0000 7f9ae3522640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc10d4b0 con 0x7f9adc1021e0 2024-03-28T05:21:05.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.319+0000 7f9ae0abd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9adc104b10 0x7f9adc197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:05.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.319+0000 7f9ae0abd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9adc104b10 0x7f9adc197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43464/0 (socket says 172.21.15.23:43464) 2024-03-28T05:21:05.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.319+0000 7f9ae0abd640 1 -- 172.21.15.23:0/2824073711 learned_addr learned my addr 172.21.15.23:0/2824073711 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:05.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.319+0000 7f9ae12be640 1 --2- 172.21.15.23:0/2824073711 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9adc1021e0 0x7f9adc197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:05.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.319+0000 7f9ae1abf640 1 --2- 172.21.15.23:0/2824073711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9adc107440 0x7f9adc19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:05.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.319+0000 7f9ae0abd640 1 -- 172.21.15.23:0/2824073711 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9adc1021e0 msgr2=0x7f9adc197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:05.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.319+0000 7f9ae0abd640 1 --2- 172.21.15.23:0/2824073711 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9adc1021e0 0x7f9adc197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.319+0000 7f9ae0abd640 1 -- 172.21.15.23:0/2824073711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9adc107440 msgr2=0x7f9adc19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:05.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.319+0000 7f9ae0abd640 1 --2- 172.21.15.23:0/2824073711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9adc107440 0x7f9adc19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.319+0000 7f9ae0abd640 1 -- 172.21.15.23:0/2824073711 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9adc19f650 con 0x7f9adc104b10 2024-03-28T05:21:05.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.319+0000 7f9ae12be640 1 --2- 172.21.15.23:0/2824073711 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9adc1021e0 0x7f9adc197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:21:05.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.319+0000 7f9ae0abd640 1 --2- 172.21.15.23:0/2824073711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9adc104b10 0x7f9adc197f70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9acc000c00 tx=0x7f9acc005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:05.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.319+0000 7f9ad27fc640 1 -- 172.21.15.23:0/2824073711 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc041070 con 0x7f9adc104b10 2024-03-28T05:21:05.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.319+0000 7f9ad27fc640 1 -- 172.21.15.23:0/2824073711 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9acc0044d0 con 0x7f9adc104b10 2024-03-28T05:21:05.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.319+0000 7f9ae3522640 1 -- 172.21.15.23:0/2824073711 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9adc19f8e0 con 0x7f9adc104b10 2024-03-28T05:21:05.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.319+0000 7f9ad27fc640 1 -- 172.21.15.23:0/2824073711 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9acc034440 con 0x7f9adc104b10 2024-03-28T05:21:05.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.320+0000 7f9ae3522640 1 -- 172.21.15.23:0/2824073711 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9adc19fe20 con 0x7f9adc104b10 2024-03-28T05:21:05.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.320+0000 7f9ae3522640 1 -- 172.21.15.23:0/2824073711 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9adc0fd620 con 0x7f9adc104b10 2024-03-28T05:21:05.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.322+0000 7f9ad27fc640 1 -- 172.21.15.23:0/2824073711 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9acc044070 con 0x7f9adc104b10 2024-03-28T05:21:05.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.322+0000 7f9ad27fc640 1 --2- 172.21.15.23:0/2824073711 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f9ac80a2880 0x7f9ac80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:05.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.322+0000 7f9ad27fc640 1 -- 172.21.15.23:0/2824073711 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9acc0f0220 con 0x7f9adc104b10 2024-03-28T05:21:05.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.322+0000 7f9ae12be640 1 --2- 172.21.15.23:0/2824073711 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f9ac80a2880 0x7f9ac80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:05.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.323+0000 7f9ae12be640 1 --2- 172.21.15.23:0/2824073711 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f9ac80a2880 0x7f9ac80a4d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8009fc0 tx=0x7f9ad8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:05.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.326+0000 7f9ad27fc640 1 -- 172.21.15.23:0/2824073711 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9acc0bd110 con 0x7f9adc104b10 2024-03-28T05:21:05.409 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:05 smithi023 ceph-mon[31954]: mds.? [v2:172.21.15.98:6838/144113007,v1:172.21.15.98:6839/144113007] up:boot 2024-03-28T05:21:05.409 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:05 smithi023 ceph-mon[31954]: fsmap 12 up:standby 2024-03-28T05:21:05.409 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:05 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T05:21:05.409 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:05 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:05.409 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:05 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:05.409 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:05 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:05.409 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:05 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:05.409 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:05 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:05.410 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:05 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:21:05.410 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:05 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/310722785' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:21:05.410 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:05 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2989634456' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:05.450 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:05 smithi098 ceph-mon[36087]: mds.? [v2:172.21.15.98:6838/144113007,v1:172.21.15.98:6839/144113007] up:boot 2024-03-28T05:21:05.451 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:05 smithi098 ceph-mon[36087]: fsmap 12 up:standby 2024-03-28T05:21:05.451 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:05 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T05:21:05.451 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:05 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:05.451 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:05 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:05.451 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:05 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:05.451 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:05 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:05.451 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:05 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:05.451 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:05 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:21:05.451 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:05 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/310722785' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:21:05.451 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:05 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2989634456' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:05.484 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:05 smithi113 ceph-mon[37164]: mds.? [v2:172.21.15.98:6838/144113007,v1:172.21.15.98:6839/144113007] up:boot 2024-03-28T05:21:05.484 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:05 smithi113 ceph-mon[37164]: fsmap 12 up:standby 2024-03-28T05:21:05.484 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:05 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T05:21:05.484 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:05 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:05.484 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:05 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:05.484 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:05 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:05.484 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:05 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:05.485 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:05 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:05.485 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:05 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:21:05.485 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:05 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/310722785' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:21:05.485 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:05 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2989634456' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:05.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.515+0000 7f9ae3522640 1 -- 172.21.15.23:0/2824073711 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f9adc02d1f0 con 0x7f9adc104b10 2024-03-28T05:21:05.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.516+0000 7f9ad27fc640 1 -- 172.21.15.23:0/2824073711 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) v1 ==== 84+0+65 (secure 0 0 0) 0x7f9acc0c11b0 con 0x7f9adc104b10 2024-03-28T05:21:05.514 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:21:05.514 INFO:teuthology.orchestra.run.smithi023.stdout:[ 2024-03-28T05:21:05.514 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:21:05.514 INFO:teuthology.orchestra.run.smithi023.stdout: "poolnum": 1, 2024-03-28T05:21:05.515 INFO:teuthology.orchestra.run.smithi023.stdout: "poolname": ".mgr" 2024-03-28T05:21:05.515 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:21:05.515 INFO:teuthology.orchestra.run.smithi023.stdout:] 2024-03-28T05:21:05.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.518+0000 7f9ae3522640 1 -- 172.21.15.23:0/2824073711 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f9ac80a2880 msgr2=0x7f9ac80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:05.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.518+0000 7f9ae3522640 1 --2- 172.21.15.23:0/2824073711 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f9ac80a2880 0x7f9ac80a4d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8009fc0 tx=0x7f9ad8005f90 comp rx=0 tx=0).stop 2024-03-28T05:21:05.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.518+0000 7f9ae3522640 1 -- 172.21.15.23:0/2824073711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9adc104b10 msgr2=0x7f9adc197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:05.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.518+0000 7f9ae3522640 1 --2- 172.21.15.23:0/2824073711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9adc104b10 0x7f9adc197f70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9acc000c00 tx=0x7f9acc005d50 comp rx=0 tx=0).stop 2024-03-28T05:21:05.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.518+0000 7f9ae3522640 1 -- 172.21.15.23:0/2824073711 shutdown_connections 2024-03-28T05:21:05.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.518+0000 7f9ae3522640 1 --2- 172.21.15.23:0/2824073711 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f9ac80a2880 0x7f9ac80a4d40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.518+0000 7f9ae3522640 1 --2- 172.21.15.23:0/2824073711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9adc107440 0x7f9adc19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.518+0000 7f9ae3522640 1 --2- 172.21.15.23:0/2824073711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9adc104b10 0x7f9adc197f70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.518+0000 7f9ae3522640 1 --2- 172.21.15.23:0/2824073711 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9adc1021e0 0x7f9adc197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.518+0000 7f9ae3522640 1 -- 172.21.15.23:0/2824073711 >> 172.21.15.23:0/2824073711 conn(0x7f9adc0fbb30 msgr2=0x7f9adc102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:05.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.518+0000 7f9ae3522640 1 -- 172.21.15.23:0/2824073711 shutdown_connections 2024-03-28T05:21:05.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.518+0000 7f9ae3522640 1 -- 172.21.15.23:0/2824073711 wait complete. 2024-03-28T05:21:05.527 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T05:21:05.527 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T05:21:05.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.636+0000 7f0f67cd1640 1 -- 172.21.15.23:0/4236564550 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f60102150 msgr2=0x7f0f601089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:05.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.636+0000 7f0f67cd1640 1 --2- 172.21.15.23:0/4236564550 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f60102150 0x7f0f601089e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0f58009ca0 tx=0x7f0f5802c0f0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.637+0000 7f0f67cd1640 1 -- 172.21.15.23:0/4236564550 shutdown_connections 2024-03-28T05:21:05.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.637+0000 7f0f67cd1640 1 --2- 172.21.15.23:0/4236564550 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f60102150 0x7f0f601089e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.637+0000 7f0f67cd1640 1 --2- 172.21.15.23:0/4236564550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f60101790 0x7f0f60101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.637+0000 7f0f67cd1640 1 --2- 172.21.15.23:0/4236564550 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f601005a0 0x7f0f601009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.637+0000 7f0f67cd1640 1 -- 172.21.15.23:0/4236564550 >> 172.21.15.23:0/4236564550 conn(0x7f0f600fbb30 msgr2=0x7f0f600fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:05.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.637+0000 7f0f67cd1640 1 -- 172.21.15.23:0/4236564550 shutdown_connections 2024-03-28T05:21:05.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.637+0000 7f0f67cd1640 1 -- 172.21.15.23:0/4236564550 wait complete. 2024-03-28T05:21:05.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.637+0000 7f0f67cd1640 1 Processor -- start 2024-03-28T05:21:05.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.637+0000 7f0f67cd1640 1 -- start start 2024-03-28T05:21:05.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.637+0000 7f0f67cd1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f601005a0 0x7f0f60104530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:05.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.637+0000 7f0f67cd1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f60101790 0x7f0f60102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:05.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.637+0000 7f0f67cd1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f60102150 0x7f0f601030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:05.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.637+0000 7f0f67cd1640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6010c290 con 0x7f0f601005a0 2024-03-28T05:21:05.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.637+0000 7f0f67cd1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6010c110 con 0x7f0f60101790 2024-03-28T05:21:05.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.637+0000 7f0f67cd1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6010c410 con 0x7f0f60102150 2024-03-28T05:21:05.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.637+0000 7f0f65a6d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f601005a0 0x7f0f60104530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:05.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.637+0000 7f0f65a6d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f601005a0 0x7f0f60104530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43476/0 (socket says 172.21.15.23:43476) 2024-03-28T05:21:05.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.637+0000 7f0f65a6d640 1 -- 172.21.15.23:0/1487044816 learned_addr learned my addr 172.21.15.23:0/1487044816 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:05.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.638+0000 7f0f6526c640 1 --2- 172.21.15.23:0/1487044816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f60101790 0x7f0f60102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:05.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.638+0000 7f0f65a6d640 1 -- 172.21.15.23:0/1487044816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f60102150 msgr2=0x7f0f601030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:05.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.638+0000 7f0f6626e640 1 --2- 172.21.15.23:0/1487044816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f60102150 0x7f0f601030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:05.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.638+0000 7f0f65a6d640 1 --2- 172.21.15.23:0/1487044816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f60102150 0x7f0f601030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.638+0000 7f0f65a6d640 1 -- 172.21.15.23:0/1487044816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f60101790 msgr2=0x7f0f60102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:05.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.638+0000 7f0f65a6d640 1 --2- 172.21.15.23:0/1487044816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f60101790 0x7f0f60102b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:05.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.638+0000 7f0f65a6d640 1 -- 172.21.15.23:0/1487044816 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f60103940 con 0x7f0f601005a0 2024-03-28T05:21:05.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.638+0000 7f0f6526c640 1 --2- 172.21.15.23:0/1487044816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f60101790 0x7f0f60102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:21:05.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.638+0000 7f0f6626e640 1 --2- 172.21.15.23:0/1487044816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f60102150 0x7f0f601030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:21:05.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.638+0000 7f0f65a6d640 1 --2- 172.21.15.23:0/1487044816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f601005a0 0x7f0f60104530 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c00c970 tx=0x7f0f5c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:05.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.638+0000 7f0f56ffd640 1 -- 172.21.15.23:0/1487044816 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c004220 con 0x7f0f601005a0 2024-03-28T05:21:05.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.638+0000 7f0f56ffd640 1 -- 172.21.15.23:0/1487044816 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f5c0043c0 con 0x7f0f601005a0 2024-03-28T05:21:05.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.638+0000 7f0f67cd1640 1 -- 172.21.15.23:0/1487044816 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f601a1b50 con 0x7f0f601005a0 2024-03-28T05:21:05.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.638+0000 7f0f56ffd640 1 -- 172.21.15.23:0/1487044816 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c004a00 con 0x7f0f601005a0 2024-03-28T05:21:05.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.638+0000 7f0f67cd1640 1 -- 172.21.15.23:0/1487044816 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f601a20a0 con 0x7f0f601005a0 2024-03-28T05:21:05.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.639+0000 7f0f67cd1640 1 -- 172.21.15.23:0/1487044816 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f601a2270 con 0x7f0f601005a0 2024-03-28T05:21:05.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.640+0000 7f0f56ffd640 1 -- 172.21.15.23:0/1487044816 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0f5c0079b0 con 0x7f0f601005a0 2024-03-28T05:21:05.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.641+0000 7f0f56ffd640 1 --2- 172.21.15.23:0/1487044816 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f0f4c0a2880 0x7f0f4c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:05.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.641+0000 7f0f6526c640 1 --2- 172.21.15.23:0/1487044816 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f0f4c0a2880 0x7f0f4c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:05.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.641+0000 7f0f56ffd640 1 -- 172.21.15.23:0/1487044816 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0f5c0ce7c0 con 0x7f0f601005a0 2024-03-28T05:21:05.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.641+0000 7f0f6526c640 1 --2- 172.21.15.23:0/1487044816 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f0f4c0a2880 0x7f0f4c0a4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0f50006fd0 tx=0x7f0f50008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:05.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.644+0000 7f0f56ffd640 1 -- 172.21.15.23:0/1487044816 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0f5c014030 con 0x7f0f601005a0 2024-03-28T05:21:05.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:05.842+0000 7f0f67cd1640 1 -- 172.21.15.23:0/1487044816 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0f601a2600 con 0x7f0f601005a0 2024-03-28T05:21:06.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.234+0000 7f0f56ffd640 1 -- 172.21.15.23:0/1487044816 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0f5c09bd80 con 0x7f0f601005a0 2024-03-28T05:21:06.233 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_metadata' created 2024-03-28T05:21:06.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.236+0000 7f0f67cd1640 1 -- 172.21.15.23:0/1487044816 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f0f4c0a2880 msgr2=0x7f0f4c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:06.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.236+0000 7f0f67cd1640 1 --2- 172.21.15.23:0/1487044816 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f0f4c0a2880 0x7f0f4c0a4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0f50006fd0 tx=0x7f0f50008040 comp rx=0 tx=0).stop 2024-03-28T05:21:06.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.236+0000 7f0f67cd1640 1 -- 172.21.15.23:0/1487044816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f601005a0 msgr2=0x7f0f60104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:06.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.236+0000 7f0f67cd1640 1 --2- 172.21.15.23:0/1487044816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f601005a0 0x7f0f60104530 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c00c970 tx=0x7f0f5c00ce30 comp rx=0 tx=0).stop 2024-03-28T05:21:06.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.236+0000 7f0f67cd1640 1 -- 172.21.15.23:0/1487044816 shutdown_connections 2024-03-28T05:21:06.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.236+0000 7f0f67cd1640 1 --2- 172.21.15.23:0/1487044816 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f0f4c0a2880 0x7f0f4c0a4d40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:06.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.236+0000 7f0f67cd1640 1 --2- 172.21.15.23:0/1487044816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f60102150 0x7f0f601030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:06.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.236+0000 7f0f67cd1640 1 --2- 172.21.15.23:0/1487044816 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f60101790 0x7f0f60102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:06.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.236+0000 7f0f67cd1640 1 --2- 172.21.15.23:0/1487044816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f601005a0 0x7f0f60104530 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:06.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.236+0000 7f0f67cd1640 1 -- 172.21.15.23:0/1487044816 >> 172.21.15.23:0/1487044816 conn(0x7f0f600fbb30 msgr2=0x7f0f600fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:06.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.237+0000 7f0f67cd1640 1 -- 172.21.15.23:0/1487044816 shutdown_connections 2024-03-28T05:21:06.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.237+0000 7f0f67cd1640 1 -- 172.21.15.23:0/1487044816 wait complete. 2024-03-28T05:21:06.250 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-28T05:21:06.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.364+0000 7f7739348640 1 -- 172.21.15.23:0/2227788725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77340ff9b0 msgr2=0x7f77340ffdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:06.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.364+0000 7f7739348640 1 --2- 172.21.15.23:0/2227788725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77340ff9b0 0x7f77340ffdd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7724009930 tx=0x7f772402bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:06.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.364+0000 7f7739348640 1 -- 172.21.15.23:0/2227788725 shutdown_connections 2024-03-28T05:21:06.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.364+0000 7f7739348640 1 --2- 172.21.15.23:0/2227788725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77341033e0 0x7f7734109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:06.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.364+0000 7f7739348640 1 --2- 172.21.15.23:0/2227788725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7734102a20 0x7f7734102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:06.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.364+0000 7f7739348640 1 --2- 172.21.15.23:0/2227788725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77340ff9b0 0x7f77340ffdd0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:06.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.364+0000 7f7739348640 1 -- 172.21.15.23:0/2227788725 >> 172.21.15.23:0/2227788725 conn(0x7f77340fb2a0 msgr2=0x7f77340fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:06.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.364+0000 7f7739348640 1 -- 172.21.15.23:0/2227788725 shutdown_connections 2024-03-28T05:21:06.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.364+0000 7f7739348640 1 -- 172.21.15.23:0/2227788725 wait complete. 2024-03-28T05:21:06.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.364+0000 7f7739348640 1 Processor -- start 2024-03-28T05:21:06.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.364+0000 7f7739348640 1 -- start start 2024-03-28T05:21:06.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.364+0000 7f7739348640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77340ff9b0 0x7f7734197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:06.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.365+0000 7f7739348640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7734102a20 0x7f7734197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:06.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.365+0000 7f7739348640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77341033e0 0x7f773419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:06.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.365+0000 7f7739348640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773410d340 con 0x7f77341033e0 2024-03-28T05:21:06.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.365+0000 7f7739348640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773410d1c0 con 0x7f77340ff9b0 2024-03-28T05:21:06.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.365+0000 7f7739348640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773410d4c0 con 0x7f7734102a20 2024-03-28T05:21:06.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.365+0000 7f77337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77341033e0 0x7f773419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:06.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.365+0000 7f77337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77341033e0 0x7f773419f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43492/0 (socket says 172.21.15.23:43492) 2024-03-28T05:21:06.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.365+0000 7f77337fe640 1 -- 172.21.15.23:0/52460312 learned_addr learned my addr 172.21.15.23:0/52460312 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:06.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.365+0000 7f7732ffd640 1 --2- 172.21.15.23:0/52460312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77340ff9b0 0x7f7734197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:06.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.365+0000 7f772bfff640 1 --2- 172.21.15.23:0/52460312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7734102a20 0x7f7734197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:06.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.365+0000 7f77337fe640 1 -- 172.21.15.23:0/52460312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7734102a20 msgr2=0x7f7734197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:06.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.365+0000 7f77337fe640 1 --2- 172.21.15.23:0/52460312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7734102a20 0x7f7734197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:06.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.365+0000 7f77337fe640 1 -- 172.21.15.23:0/52460312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77340ff9b0 msgr2=0x7f7734197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:06.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.365+0000 7f77337fe640 1 --2- 172.21.15.23:0/52460312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77340ff9b0 0x7f7734197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:06.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.365+0000 7f77337fe640 1 -- 172.21.15.23:0/52460312 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773419f690 con 0x7f77341033e0 2024-03-28T05:21:06.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.365+0000 7f77337fe640 1 --2- 172.21.15.23:0/52460312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77341033e0 0x7f773419f020 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f772c00eb10 tx=0x7f772c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:06.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.365+0000 7f7730ff9640 1 -- 172.21.15.23:0/52460312 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c004b20 con 0x7f77341033e0 2024-03-28T05:21:06.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.365+0000 7f7739348640 1 -- 172.21.15.23:0/52460312 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f773419f980 con 0x7f77341033e0 2024-03-28T05:21:06.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.365+0000 7f7730ff9640 1 -- 172.21.15.23:0/52460312 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f772c004cc0 con 0x7f77341033e0 2024-03-28T05:21:06.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.365+0000 7f7730ff9640 1 -- 172.21.15.23:0/52460312 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c010760 con 0x7f77341033e0 2024-03-28T05:21:06.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.365+0000 7f7739348640 1 -- 172.21.15.23:0/52460312 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f773419fe80 con 0x7f77341033e0 2024-03-28T05:21:06.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.367+0000 7f7739348640 1 -- 172.21.15.23:0/52460312 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7734108bc0 con 0x7f77341033e0 2024-03-28T05:21:06.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.368+0000 7f7730ff9640 1 -- 172.21.15.23:0/52460312 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f772c005020 con 0x7f77341033e0 2024-03-28T05:21:06.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.368+0000 7f7730ff9640 1 --2- 172.21.15.23:0/52460312 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f77200a2800 0x7f77200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:06.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.368+0000 7f7732ffd640 1 --2- 172.21.15.23:0/52460312 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f77200a2800 0x7f77200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:06.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.368+0000 7f7730ff9640 1 -- 172.21.15.23:0/52460312 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f772c0ceca0 con 0x7f77341033e0 2024-03-28T05:21:06.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.369+0000 7f7732ffd640 1 --2- 172.21.15.23:0/52460312 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f77200a2800 0x7f77200a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f772402c7b0 tx=0x7f7724005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:06.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.373+0000 7f7730ff9640 1 -- 172.21.15.23:0/52460312 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f772c09c070 con 0x7f77341033e0 2024-03-28T05:21:06.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:06 smithi113 ceph-mon[37164]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:21:06.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:06 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2939284032' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:21:06.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:06 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2824073711' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T05:21:06.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:06 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1487044816' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T05:21:06.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:06.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:06.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:06.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:06 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:06.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:06 smithi023 ceph-mon[31954]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:21:06.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:06 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2939284032' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:21:06.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:06 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2824073711' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T05:21:06.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:06 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1487044816' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T05:21:06.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:06.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:06.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:06.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:06 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:06.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:06.578+0000 7f7739348640 1 -- 172.21.15.23:0/52460312 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f77340622a0 con 0x7f77341033e0 2024-03-28T05:21:06.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:06 smithi098 ceph-mon[36087]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:21:06.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:06 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2939284032' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:21:06.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:06 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2824073711' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T05:21:06.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:06 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1487044816' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T05:21:06.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:06 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:06.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:06 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:06.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:06 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:06.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:06 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:07.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.245+0000 7f7730ff9640 1 -- 172.21.15.23:0/52460312 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) v1 ==== 171+0+0 (secure 0 0 0) 0x7f772c09b970 con 0x7f77341033e0 2024-03-28T05:21:07.244 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data' created 2024-03-28T05:21:07.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.247+0000 7f7739348640 1 -- 172.21.15.23:0/52460312 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f77200a2800 msgr2=0x7f77200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:07.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.247+0000 7f7739348640 1 --2- 172.21.15.23:0/52460312 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f77200a2800 0x7f77200a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f772402c7b0 tx=0x7f7724005fd0 comp rx=0 tx=0).stop 2024-03-28T05:21:07.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.247+0000 7f7739348640 1 -- 172.21.15.23:0/52460312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77341033e0 msgr2=0x7f773419f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:07.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.247+0000 7f7739348640 1 --2- 172.21.15.23:0/52460312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77341033e0 0x7f773419f020 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f772c00eb10 tx=0x7f772c00c2d0 comp rx=0 tx=0).stop 2024-03-28T05:21:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.247+0000 7f7739348640 1 -- 172.21.15.23:0/52460312 shutdown_connections 2024-03-28T05:21:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.247+0000 7f7739348640 1 --2- 172.21.15.23:0/52460312 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f77200a2800 0x7f77200a4cc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.247+0000 7f7739348640 1 --2- 172.21.15.23:0/52460312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77341033e0 0x7f773419f020 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.247+0000 7f7739348640 1 --2- 172.21.15.23:0/52460312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7734102a20 0x7f7734197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.248+0000 7f7739348640 1 --2- 172.21.15.23:0/52460312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77340ff9b0 0x7f7734197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.248+0000 7f7739348640 1 -- 172.21.15.23:0/52460312 >> 172.21.15.23:0/52460312 conn(0x7f77340fb2a0 msgr2=0x7f7734105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.248+0000 7f7739348640 1 -- 172.21.15.23:0/52460312 shutdown_connections 2024-03-28T05:21:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.248+0000 7f7739348640 1 -- 172.21.15.23:0/52460312 wait complete. 2024-03-28T05:21:07.267 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T05:21:07.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.380+0000 7fe649ad1640 1 -- 172.21.15.23:0/2596770649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe644102150 msgr2=0x7fe6441089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.380+0000 7fe649ad1640 1 --2- 172.21.15.23:0/2596770649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe644102150 0x7fe6441089e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009ca0 tx=0x7fe63c02c0f0 comp rx=0 tx=0).stop 2024-03-28T05:21:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.380+0000 7fe649ad1640 1 -- 172.21.15.23:0/2596770649 shutdown_connections 2024-03-28T05:21:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.380+0000 7fe649ad1640 1 --2- 172.21.15.23:0/2596770649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe644102150 0x7fe6441089e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.380+0000 7fe649ad1640 1 --2- 172.21.15.23:0/2596770649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe644101790 0x7fe644101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.380+0000 7fe649ad1640 1 --2- 172.21.15.23:0/2596770649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6441005a0 0x7fe6441009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.380+0000 7fe649ad1640 1 -- 172.21.15.23:0/2596770649 >> 172.21.15.23:0/2596770649 conn(0x7fe6440fbb30 msgr2=0x7fe6440fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.380+0000 7fe649ad1640 1 -- 172.21.15.23:0/2596770649 shutdown_connections 2024-03-28T05:21:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.380+0000 7fe649ad1640 1 -- 172.21.15.23:0/2596770649 wait complete. 2024-03-28T05:21:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.380+0000 7fe649ad1640 1 Processor -- start 2024-03-28T05:21:07.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.380+0000 7fe649ad1640 1 -- start start 2024-03-28T05:21:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.380+0000 7fe649ad1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6441005a0 0x7fe6440693d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.380+0000 7fe649ad1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe644101790 0x7fe64419c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.380+0000 7fe649ad1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe644102150 0x7fe64419ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.380+0000 7fe649ad1640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64410c230 con 0x7fe644101790 2024-03-28T05:21:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.381+0000 7fe649ad1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64410c0b0 con 0x7fe644102150 2024-03-28T05:21:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.381+0000 7fe649ad1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64410c3b0 con 0x7fe6441005a0 2024-03-28T05:21:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.381+0000 7fe63bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe644101790 0x7fe64419c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.381+0000 7fe63bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe644101790 0x7fe64419c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43514/0 (socket says 172.21.15.23:43514) 2024-03-28T05:21:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.381+0000 7fe63bfff640 1 -- 172.21.15.23:0/844046750 learned_addr learned my addr 172.21.15.23:0/844046750 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.381+0000 7fe6437fe640 1 --2- 172.21.15.23:0/844046750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6441005a0 0x7fe6440693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:07.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.381+0000 7fe643fff640 1 --2- 172.21.15.23:0/844046750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe644102150 0x7fe64419ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:07.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.381+0000 7fe63bfff640 1 -- 172.21.15.23:0/844046750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6441005a0 msgr2=0x7fe6440693d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:07.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.381+0000 7fe63bfff640 1 --2- 172.21.15.23:0/844046750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6441005a0 0x7fe6440693d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:07.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.381+0000 7fe63bfff640 1 -- 172.21.15.23:0/844046750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe644102150 msgr2=0x7fe64419ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:07.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.381+0000 7fe63bfff640 1 --2- 172.21.15.23:0/844046750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe644102150 0x7fe64419ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:07.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.381+0000 7fe63bfff640 1 -- 172.21.15.23:0/844046750 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64419f4c0 con 0x7fe644101790 2024-03-28T05:21:07.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.381+0000 7fe643fff640 1 --2- 172.21.15.23:0/844046750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe644102150 0x7fe64419ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:21:07.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.381+0000 7fe63bfff640 1 --2- 172.21.15.23:0/844046750 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe644101790 0x7fe64419c590 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe62c00c920 tx=0x7fe62c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:07.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.381+0000 7fe6417fa640 1 -- 172.21.15.23:0/844046750 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62c007c70 con 0x7fe644101790 2024-03-28T05:21:07.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.381+0000 7fe6417fa640 1 -- 172.21.15.23:0/844046750 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe62c007e10 con 0x7fe644101790 2024-03-28T05:21:07.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.381+0000 7fe649ad1640 1 -- 172.21.15.23:0/844046750 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe64419f7b0 con 0x7fe644101790 2024-03-28T05:21:07.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.381+0000 7fe6417fa640 1 -- 172.21.15.23:0/844046750 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62c018890 con 0x7fe644101790 2024-03-28T05:21:07.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.381+0000 7fe649ad1640 1 -- 172.21.15.23:0/844046750 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe64419fd50 con 0x7fe644101790 2024-03-28T05:21:07.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.382+0000 7fe649ad1640 1 -- 172.21.15.23:0/844046750 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe644107860 con 0x7fe644101790 2024-03-28T05:21:07.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.383+0000 7fe6417fa640 1 -- 172.21.15.23:0/844046750 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe62c018a50 con 0x7fe644101790 2024-03-28T05:21:07.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.384+0000 7fe6417fa640 1 --2- 172.21.15.23:0/844046750 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe6300a2880 0x7fe6300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:07.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.384+0000 7fe6437fe640 1 --2- 172.21.15.23:0/844046750 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe6300a2880 0x7fe6300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:07.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.384+0000 7fe6417fa640 1 -- 172.21.15.23:0/844046750 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7879+0+0 (secure 0 0 0) 0x7fe62c0cec10 con 0x7fe644101790 2024-03-28T05:21:07.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.384+0000 7fe6437fe640 1 --2- 172.21.15.23:0/844046750 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe6300a2880 0x7fe6300a4d40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe6441015f0 tx=0x7fe634005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:07.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.388+0000 7fe6417fa640 1 -- 172.21.15.23:0/844046750 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe62c014030 con 0x7fe644101790 2024-03-28T05:21:07.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:07 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:07.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:07 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:07.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:07 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:21:07.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:07 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:21:07.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:07 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:07.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:07 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1487044816' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T05:21:07.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:07 smithi113 ceph-mon[37164]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T05:21:07.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:07 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:21:07.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:07 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/52460312' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-28T05:21:07.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:07 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:07.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:07 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:07.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:07 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:21:07.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:07 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:21:07.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:07 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:07.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:07 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1487044816' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T05:21:07.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:07 smithi023 ceph-mon[31954]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T05:21:07.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:07 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:21:07.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:07 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/52460312' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-28T05:21:07.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:07.615+0000 7fe649ad1640 1 -- 172.21.15.23:0/844046750 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe64402d070 con 0x7fe644101790 2024-03-28T05:21:07.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:07 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:07.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:07 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:07.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:07 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:21:07.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:07 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:21:07.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:07 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:07.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:07 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1487044816' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T05:21:07.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:07 smithi098 ceph-mon[36087]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T05:21:07.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:07 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:21:07.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:07 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/52460312' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-28T05:21:08.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.247+0000 7fe6417fa640 1 -- 172.21.15.23:0/844046750 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7fe62c09bf70 con 0x7fe644101790 2024-03-28T05:21:08.245 INFO:teuthology.orchestra.run.smithi023.stderr:new fs with metadata pool 2 and data pool 3 2024-03-28T05:21:08.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.249+0000 7fe649ad1640 1 -- 172.21.15.23:0/844046750 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe6300a2880 msgr2=0x7fe6300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:08.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.249+0000 7fe649ad1640 1 --2- 172.21.15.23:0/844046750 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe6300a2880 0x7fe6300a4d40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe6441015f0 tx=0x7fe634005f90 comp rx=0 tx=0).stop 2024-03-28T05:21:08.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.249+0000 7fe649ad1640 1 -- 172.21.15.23:0/844046750 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe644101790 msgr2=0x7fe64419c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:08.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.249+0000 7fe649ad1640 1 --2- 172.21.15.23:0/844046750 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe644101790 0x7fe64419c590 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe62c00c920 tx=0x7fe62c00cde0 comp rx=0 tx=0).stop 2024-03-28T05:21:08.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.249+0000 7fe649ad1640 1 -- 172.21.15.23:0/844046750 shutdown_connections 2024-03-28T05:21:08.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.249+0000 7fe649ad1640 1 --2- 172.21.15.23:0/844046750 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe6300a2880 0x7fe6300a4d40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:08.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.249+0000 7fe649ad1640 1 --2- 172.21.15.23:0/844046750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe644102150 0x7fe64419ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:08.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.249+0000 7fe649ad1640 1 --2- 172.21.15.23:0/844046750 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe644101790 0x7fe64419c590 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:08.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.249+0000 7fe649ad1640 1 --2- 172.21.15.23:0/844046750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6441005a0 0x7fe6440693d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:08.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.249+0000 7fe649ad1640 1 -- 172.21.15.23:0/844046750 >> 172.21.15.23:0/844046750 conn(0x7fe6440fbb30 msgr2=0x7fe644104a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:08.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.249+0000 7fe649ad1640 1 -- 172.21.15.23:0/844046750 shutdown_connections 2024-03-28T05:21:08.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.249+0000 7fe649ad1640 1 -- 172.21.15.23:0/844046750 wait complete. 2024-03-28T05:21:08.260 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-28T05:21:08.260 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-28T05:21:08.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.369+0000 7f492d2c6640 1 -- 172.21.15.23:0/1514934199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4928103280 msgr2=0x7f4928109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:08.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.369+0000 7f492d2c6640 1 --2- 172.21.15.23:0/1514934199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4928103280 0x7f4928109b10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4918009ca0 tx=0x7f491802c0f0 comp rx=0 tx=0).stop 2024-03-28T05:21:08.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.369+0000 7f492d2c6640 1 -- 172.21.15.23:0/1514934199 shutdown_connections 2024-03-28T05:21:08.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.369+0000 7f492d2c6640 1 --2- 172.21.15.23:0/1514934199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4928103280 0x7f4928109b10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:08.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.369+0000 7f492d2c6640 1 --2- 172.21.15.23:0/1514934199 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49281028c0 0x7f4928102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:08.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.369+0000 7f492d2c6640 1 --2- 172.21.15.23:0/1514934199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f49281016d0 0x7f4928101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:08.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.369+0000 7f492d2c6640 1 -- 172.21.15.23:0/1514934199 >> 172.21.15.23:0/1514934199 conn(0x7f49280fcc40 msgr2=0x7f49280ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:08.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.369+0000 7f492d2c6640 1 -- 172.21.15.23:0/1514934199 shutdown_connections 2024-03-28T05:21:08.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.369+0000 7f492d2c6640 1 -- 172.21.15.23:0/1514934199 wait complete. 2024-03-28T05:21:08.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.370+0000 7f492d2c6640 1 Processor -- start 2024-03-28T05:21:08.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.370+0000 7f492d2c6640 1 -- start start 2024-03-28T05:21:08.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.370+0000 7f492d2c6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49281016d0 0x7f49281979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:08.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.370+0000 7f492d2c6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49281028c0 0x7f4928197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:08.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.370+0000 7f492d2c6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4928103280 0x7f492819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:08.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.370+0000 7f492d2c6640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492810d350 con 0x7f49281016d0 2024-03-28T05:21:08.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.370+0000 7f492d2c6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492810d1d0 con 0x7f49281028c0 2024-03-28T05:21:08.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.370+0000 7f492d2c6640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492810d4d0 con 0x7f4928103280 2024-03-28T05:21:08.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.370+0000 7f4926ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49281016d0 0x7f49281979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:08.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.370+0000 7f4926ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49281016d0 0x7f49281979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43534/0 (socket says 172.21.15.23:43534) 2024-03-28T05:21:08.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.370+0000 7f4926ffd640 1 -- 172.21.15.23:0/4211634355 learned_addr learned my addr 172.21.15.23:0/4211634355 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:08.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.371+0000 7f49277fe640 1 --2- 172.21.15.23:0/4211634355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4928103280 0x7f492819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:08.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.371+0000 7f49267fc640 1 --2- 172.21.15.23:0/4211634355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49281028c0 0x7f4928197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:08.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.371+0000 7f4926ffd640 1 -- 172.21.15.23:0/4211634355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4928103280 msgr2=0x7f492819ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:08.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.371+0000 7f4926ffd640 1 --2- 172.21.15.23:0/4211634355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4928103280 0x7f492819ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:08.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.371+0000 7f4926ffd640 1 -- 172.21.15.23:0/4211634355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49281028c0 msgr2=0x7f4928197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:08.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.371+0000 7f4926ffd640 1 --2- 172.21.15.23:0/4211634355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49281028c0 0x7f4928197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:08.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.371+0000 7f4926ffd640 1 -- 172.21.15.23:0/4211634355 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492819f5e0 con 0x7f49281016d0 2024-03-28T05:21:08.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.371+0000 7f49277fe640 1 --2- 172.21.15.23:0/4211634355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4928103280 0x7f492819ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:21:08.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.371+0000 7f49267fc640 1 --2- 172.21.15.23:0/4211634355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49281028c0 0x7f4928197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:21:08.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.371+0000 7f4926ffd640 1 --2- 172.21.15.23:0/4211634355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49281016d0 0x7f49281979c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f492000e9c0 tx=0x7f492000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:08.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.371+0000 7f490ffff640 1 -- 172.21.15.23:0/4211634355 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492000cd80 con 0x7f49281016d0 2024-03-28T05:21:08.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.371+0000 7f490ffff640 1 -- 172.21.15.23:0/4211634355 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49200035f0 con 0x7f49281016d0 2024-03-28T05:21:08.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.371+0000 7f492d2c6640 1 -- 172.21.15.23:0/4211634355 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f492819f8d0 con 0x7f49281016d0 2024-03-28T05:21:08.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.371+0000 7f490ffff640 1 -- 172.21.15.23:0/4211634355 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4920010690 con 0x7f49281016d0 2024-03-28T05:21:08.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.371+0000 7f492d2c6640 1 -- 172.21.15.23:0/4211634355 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f492819fdd0 con 0x7f49281016d0 2024-03-28T05:21:08.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.372+0000 7f492d2c6640 1 -- 172.21.15.23:0/4211634355 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4928108920 con 0x7f49281016d0 2024-03-28T05:21:08.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.373+0000 7f490ffff640 1 -- 172.21.15.23:0/4211634355 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f49200037a0 con 0x7f49281016d0 2024-03-28T05:21:08.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.373+0000 7f490ffff640 1 --2- 172.21.15.23:0/4211634355 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f49100a2800 0x7f49100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:08.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.374+0000 7f49267fc640 1 --2- 172.21.15.23:0/4211634355 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f49100a2800 0x7f49100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:08.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.374+0000 7f490ffff640 1 -- 172.21.15.23:0/4211634355 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f49200cee00 con 0x7f49281016d0 2024-03-28T05:21:08.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.374+0000 7f49267fc640 1 --2- 172.21.15.23:0/4211634355 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f49100a2800 0x7f49100a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f491c005e10 tx=0x7f491c005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:08.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.378+0000 7f490ffff640 1 -- 172.21.15.23:0/4211634355 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f492009bff0 con 0x7f49281016d0 2024-03-28T05:21:08.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:08 smithi113 ceph-mon[37164]: pgmap v281: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:21:08.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:08 smithi113 ceph-mon[37164]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T05:21:08.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:08 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/52460312' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-28T05:21:08.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:08 smithi113 ceph-mon[37164]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T05:21:08.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:08 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/844046750' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T05:21:08.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:08 smithi113 ceph-mon[37164]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-03-28T05:21:08.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:08 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:08.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:08 smithi113 ceph-mon[37164]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T05:21:08.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:08 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:08.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:08 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:08.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:08 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:08.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:08 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:08.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:08 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:08.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:08 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:21:08.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:08 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:21:08.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:08 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:08.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:08 smithi023 ceph-mon[31954]: pgmap v281: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:21:08.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:08 smithi023 ceph-mon[31954]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T05:21:08.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:08 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/52460312' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-28T05:21:08.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:08 smithi023 ceph-mon[31954]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T05:21:08.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:08 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/844046750' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T05:21:08.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:08 smithi023 ceph-mon[31954]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-03-28T05:21:08.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:08 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:08.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:08 smithi023 ceph-mon[31954]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T05:21:08.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:08 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:08.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:08 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:08.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:08 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:08.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:08 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:08.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:08 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:08.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:08 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:21:08.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:08 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:21:08.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:08 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:08.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:08.571+0000 7f492d2c6640 1 -- 172.21.15.23:0/4211634355 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f492802d070 con 0x7f49281016d0 2024-03-28T05:21:08.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:08 smithi098 ceph-mon[36087]: pgmap v281: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:21:08.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:08 smithi098 ceph-mon[36087]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T05:21:08.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:08 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/52460312' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-28T05:21:08.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:08 smithi098 ceph-mon[36087]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T05:21:08.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:08 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/844046750' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T05:21:08.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:08 smithi098 ceph-mon[36087]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-03-28T05:21:08.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:08.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:08 smithi098 ceph-mon[36087]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T05:21:08.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:08.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:08.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:08.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:08.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:08.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:21:08.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:21:08.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:09.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.025+0000 7f490ffff640 1 -- 172.21.15.23:0/4211634355 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) v1 ==== 157+0+0 (secure 0 0 0) 0x7f492802d070 con 0x7f49281016d0 2024-03-28T05:21:09.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.027+0000 7f492d2c6640 1 -- 172.21.15.23:0/4211634355 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f49100a2800 msgr2=0x7f49100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:09.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.027+0000 7f492d2c6640 1 --2- 172.21.15.23:0/4211634355 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f49100a2800 0x7f49100a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f491c005e10 tx=0x7f491c005da0 comp rx=0 tx=0).stop 2024-03-28T05:21:09.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.027+0000 7f492d2c6640 1 -- 172.21.15.23:0/4211634355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49281016d0 msgr2=0x7f49281979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:09.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.027+0000 7f492d2c6640 1 --2- 172.21.15.23:0/4211634355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49281016d0 0x7f49281979c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f492000e9c0 tx=0x7f492000ee80 comp rx=0 tx=0).stop 2024-03-28T05:21:09.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.027+0000 7f492d2c6640 1 -- 172.21.15.23:0/4211634355 shutdown_connections 2024-03-28T05:21:09.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.027+0000 7f492d2c6640 1 --2- 172.21.15.23:0/4211634355 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f49100a2800 0x7f49100a4cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:09.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.027+0000 7f492d2c6640 1 --2- 172.21.15.23:0/4211634355 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4928103280 0x7f492819ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:09.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.027+0000 7f492d2c6640 1 --2- 172.21.15.23:0/4211634355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49281028c0 0x7f4928197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:09.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.027+0000 7f492d2c6640 1 --2- 172.21.15.23:0/4211634355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49281016d0 0x7f49281979c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:09.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.027+0000 7f492d2c6640 1 -- 172.21.15.23:0/4211634355 >> 172.21.15.23:0/4211634355 conn(0x7f49280fcc40 msgr2=0x7f4928105af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:09.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.027+0000 7f492d2c6640 1 -- 172.21.15.23:0/4211634355 shutdown_connections 2024-03-28T05:21:09.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.028+0000 7f492d2c6640 1 -- 172.21.15.23:0/4211634355 wait complete. 2024-03-28T05:21:09.038 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T05:21:09.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.152+0000 7f73ff04b640 1 -- 172.21.15.23:0/2681231236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73f8100640 msgr2=0x7f73f8100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:09.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.152+0000 7f73ff04b640 1 --2- 172.21.15.23:0/2681231236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73f8100640 0x7f73f8100a60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f73f4009900 tx=0x7f73f402bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:09.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.152+0000 7f73ff04b640 1 -- 172.21.15.23:0/2681231236 shutdown_connections 2024-03-28T05:21:09.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.152+0000 7f73ff04b640 1 --2- 172.21.15.23:0/2681231236 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73f81021f0 0x7f73f8108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:09.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.152+0000 7f73ff04b640 1 --2- 172.21.15.23:0/2681231236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73f8101830 0x7f73f8101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:09.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.152+0000 7f73ff04b640 1 --2- 172.21.15.23:0/2681231236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73f8100640 0x7f73f8100a60 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:09.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.152+0000 7f73ff04b640 1 -- 172.21.15.23:0/2681231236 >> 172.21.15.23:0/2681231236 conn(0x7f73f80fbbd0 msgr2=0x7f73f80fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:09.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.152+0000 7f73ff04b640 1 -- 172.21.15.23:0/2681231236 shutdown_connections 2024-03-28T05:21:09.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.152+0000 7f73ff04b640 1 -- 172.21.15.23:0/2681231236 wait complete. 2024-03-28T05:21:09.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.152+0000 7f73ff04b640 1 Processor -- start 2024-03-28T05:21:09.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.152+0000 7f73ff04b640 1 -- start start 2024-03-28T05:21:09.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73ff04b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73f8100640 0x7f73f81979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:09.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73ff04b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73f8101830 0x7f73f8197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:09.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73ff04b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73f81021f0 0x7f73f819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:09.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73ff04b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f810d3b0 con 0x7f73f81021f0 2024-03-28T05:21:09.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73ff04b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f810d230 con 0x7f73f8101830 2024-03-28T05:21:09.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73ff04b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f810d530 con 0x7f73f8100640 2024-03-28T05:21:09.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73fd5e8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73f81021f0 0x7f73f819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:09.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73fcde7640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73f8100640 0x7f73f81979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:09.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73fd5e8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73f81021f0 0x7f73f819ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43552/0 (socket says 172.21.15.23:43552) 2024-03-28T05:21:09.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73fd5e8640 1 -- 172.21.15.23:0/1338069887 learned_addr learned my addr 172.21.15.23:0/1338069887 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:09.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73fcde7640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73f8100640 0x7f73f81979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:60552/0 (socket says 172.21.15.23:60552) 2024-03-28T05:21:09.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73effff640 1 --2- 172.21.15.23:0/1338069887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73f8101830 0x7f73f8197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:09.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73fd5e8640 1 -- 172.21.15.23:0/1338069887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73f8100640 msgr2=0x7f73f81979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:09.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73fd5e8640 1 --2- 172.21.15.23:0/1338069887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73f8100640 0x7f73f81979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:09.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73fd5e8640 1 -- 172.21.15.23:0/1338069887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73f8101830 msgr2=0x7f73f8197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:09.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73fd5e8640 1 --2- 172.21.15.23:0/1338069887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73f8101830 0x7f73f8197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:09.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73fd5e8640 1 -- 172.21.15.23:0/1338069887 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f819f5d0 con 0x7f73f81021f0 2024-03-28T05:21:09.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73fd5e8640 1 --2- 172.21.15.23:0/1338069887 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73f81021f0 0x7f73f819ef60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f73f000eb10 tx=0x7f73f000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:09.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73edffb640 1 -- 172.21.15.23:0/1338069887 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f0004b20 con 0x7f73f81021f0 2024-03-28T05:21:09.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73edffb640 1 -- 172.21.15.23:0/1338069887 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73f0004cc0 con 0x7f73f81021f0 2024-03-28T05:21:09.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73ff04b640 1 -- 172.21.15.23:0/1338069887 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f819f8c0 con 0x7f73f81021f0 2024-03-28T05:21:09.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73edffb640 1 -- 172.21.15.23:0/1338069887 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f0010760 con 0x7f73f81021f0 2024-03-28T05:21:09.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.153+0000 7f73ff04b640 1 -- 172.21.15.23:0/1338069887 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f819fe10 con 0x7f73f81021f0 2024-03-28T05:21:09.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.154+0000 7f73ff04b640 1 -- 172.21.15.23:0/1338069887 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f8107900 con 0x7f73f81021f0 2024-03-28T05:21:09.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.155+0000 7f73edffb640 1 -- 172.21.15.23:0/1338069887 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f73f0005020 con 0x7f73f81021f0 2024-03-28T05:21:09.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.156+0000 7f73edffb640 1 --2- 172.21.15.23:0/1338069887 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f73e00a2800 0x7f73e00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:09.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.156+0000 7f73fcde7640 1 --2- 172.21.15.23:0/1338069887 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f73e00a2800 0x7f73e00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:09.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.156+0000 7f73edffb640 1 -- 172.21.15.23:0/1338069887 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f73f00cf220 con 0x7f73f81021f0 2024-03-28T05:21:09.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.156+0000 7f73fcde7640 1 --2- 172.21.15.23:0/1338069887 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f73e00a2800 0x7f73e00a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f73f402c170 tx=0x7f73f402df90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:09.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.160+0000 7f73edffb640 1 -- 172.21.15.23:0/1338069887 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f73f009c290 con 0x7f73f81021f0 2024-03-28T05:21:09.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:09.349+0000 7f73ff04b640 1 -- 172.21.15.23:0/1338069887 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f73f802d070 con 0x7f73f81021f0 2024-03-28T05:21:09.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:09 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/844046750' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T05:21:09.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:09 smithi023 ceph-mon[31954]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-03-28T05:21:09.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:09 smithi023 ceph-mon[31954]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-03-28T05:21:09.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:09 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/4211634355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T05:21:09.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:09 smithi023 ceph-mon[31954]: mds.? [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] up:active 2024-03-28T05:21:09.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:09 smithi023 ceph-mon[31954]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T05:21:09.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:09.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:09 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/4211634355' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-28T05:21:09.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:09 smithi023 ceph-mon[31954]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T05:21:09.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:09 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/844046750' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T05:21:09.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:09 smithi098 ceph-mon[36087]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-03-28T05:21:09.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:09 smithi098 ceph-mon[36087]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-03-28T05:21:09.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:09 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/4211634355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T05:21:09.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:09 smithi098 ceph-mon[36087]: mds.? [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] up:active 2024-03-28T05:21:09.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:09 smithi098 ceph-mon[36087]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T05:21:09.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:09.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:09 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/4211634355' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-28T05:21:09.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:09 smithi098 ceph-mon[36087]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T05:21:09.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:09 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/844046750' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T05:21:09.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:09 smithi113 ceph-mon[37164]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-03-28T05:21:09.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:09 smithi113 ceph-mon[37164]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-03-28T05:21:09.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:09 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/4211634355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T05:21:09.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:09 smithi113 ceph-mon[37164]: mds.? [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] up:active 2024-03-28T05:21:09.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:09 smithi113 ceph-mon[37164]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T05:21:09.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:21:09.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:09 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/4211634355' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-28T05:21:09.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:09 smithi113 ceph-mon[37164]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T05:21:10.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:10 smithi023 ceph-mon[31954]: pgmap v284: 97 pgs: 25 creating+peering, 34 unknown, 38 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 0 op/s 2024-03-28T05:21:10.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:10 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1338069887' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T05:21:10.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:10 smithi023 ceph-mon[31954]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T05:21:10.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:10 smithi023 ceph-mon[31954]: Cluster is now healthy 2024-03-28T05:21:10.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:10 smithi098 ceph-mon[36087]: pgmap v284: 97 pgs: 25 creating+peering, 34 unknown, 38 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 0 op/s 2024-03-28T05:21:10.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:10 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1338069887' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T05:21:10.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:10 smithi098 ceph-mon[36087]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T05:21:10.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:10 smithi098 ceph-mon[36087]: Cluster is now healthy 2024-03-28T05:21:10.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:10 smithi113 ceph-mon[37164]: pgmap v284: 97 pgs: 25 creating+peering, 34 unknown, 38 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 0 op/s 2024-03-28T05:21:10.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:10 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1338069887' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T05:21:10.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:10 smithi113 ceph-mon[37164]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T05:21:10.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:10 smithi113 ceph-mon[37164]: Cluster is now healthy 2024-03-28T05:21:11.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.262+0000 7f73edffb640 1 -- 172.21.15.23:0/1338069887 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) v1 ==== 228+0+0 (secure 0 0 0) 0x7f73f009bb90 con 0x7f73f81021f0 2024-03-28T05:21:11.261 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data_ec' created 2024-03-28T05:21:11.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.266+0000 7f73ff04b640 1 -- 172.21.15.23:0/1338069887 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f73e00a2800 msgr2=0x7f73e00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:11.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.266+0000 7f73ff04b640 1 --2- 172.21.15.23:0/1338069887 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f73e00a2800 0x7f73e00a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f73f402c170 tx=0x7f73f402df90 comp rx=0 tx=0).stop 2024-03-28T05:21:11.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.266+0000 7f73ff04b640 1 -- 172.21.15.23:0/1338069887 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73f81021f0 msgr2=0x7f73f819ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:11.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.266+0000 7f73ff04b640 1 --2- 172.21.15.23:0/1338069887 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73f81021f0 0x7f73f819ef60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f73f000eb10 tx=0x7f73f000c2d0 comp rx=0 tx=0).stop 2024-03-28T05:21:11.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.266+0000 7f73ff04b640 1 -- 172.21.15.23:0/1338069887 shutdown_connections 2024-03-28T05:21:11.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.266+0000 7f73ff04b640 1 --2- 172.21.15.23:0/1338069887 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f73e00a2800 0x7f73e00a4cc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:11.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.266+0000 7f73ff04b640 1 --2- 172.21.15.23:0/1338069887 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73f81021f0 0x7f73f819ef60 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:11.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.266+0000 7f73ff04b640 1 --2- 172.21.15.23:0/1338069887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73f8101830 0x7f73f8197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:11.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.266+0000 7f73ff04b640 1 --2- 172.21.15.23:0/1338069887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f73f8100640 0x7f73f81979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:11.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.266+0000 7f73ff04b640 1 -- 172.21.15.23:0/1338069887 >> 172.21.15.23:0/1338069887 conn(0x7f73f80fbbd0 msgr2=0x7f73f8104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:11.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.266+0000 7f73ff04b640 1 -- 172.21.15.23:0/1338069887 shutdown_connections 2024-03-28T05:21:11.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.266+0000 7f73ff04b640 1 -- 172.21.15.23:0/1338069887 wait complete. 2024-03-28T05:21:11.283 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-28T05:21:11.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.397+0000 7f56a9ff4640 1 -- 172.21.15.23:0/3523593136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a41033d0 msgr2=0x7f56a4109c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:11.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.397+0000 7f56a9ff4640 1 --2- 172.21.15.23:0/3523593136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a41033d0 0x7f56a4109c60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f569c009cd0 tx=0x7f569c02c0f0 comp rx=0 tx=0).stop 2024-03-28T05:21:11.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.397+0000 7f56a9ff4640 1 -- 172.21.15.23:0/3523593136 shutdown_connections 2024-03-28T05:21:11.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.397+0000 7f56a9ff4640 1 --2- 172.21.15.23:0/3523593136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a41033d0 0x7f56a4109c60 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:11.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.397+0000 7f56a9ff4640 1 --2- 172.21.15.23:0/3523593136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f56a4102a10 0x7f56a4102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:11.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.397+0000 7f56a9ff4640 1 --2- 172.21.15.23:0/3523593136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56a40ff9a0 0x7f56a40ffdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:11.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.397+0000 7f56a9ff4640 1 -- 172.21.15.23:0/3523593136 >> 172.21.15.23:0/3523593136 conn(0x7f56a40fb290 msgr2=0x7f56a40fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:11.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.397+0000 7f56a9ff4640 1 -- 172.21.15.23:0/3523593136 shutdown_connections 2024-03-28T05:21:11.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.397+0000 7f56a9ff4640 1 -- 172.21.15.23:0/3523593136 wait complete. 2024-03-28T05:21:11.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.398+0000 7f56a9ff4640 1 Processor -- start 2024-03-28T05:21:11.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.398+0000 7f56a9ff4640 1 -- start start 2024-03-28T05:21:11.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.398+0000 7f56a9ff4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56a40ff9a0 0x7f56a4197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:11.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.398+0000 7f56a9ff4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f56a4102a10 0x7f56a4197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:11.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.398+0000 7f56a9ff4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a41033d0 0x7f56a419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:11.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.398+0000 7f56a9ff4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a410d320 con 0x7f56a41033d0 2024-03-28T05:21:11.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.398+0000 7f56a9ff4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a410d1a0 con 0x7f56a40ff9a0 2024-03-28T05:21:11.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.398+0000 7f56a9ff4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a410d4a0 con 0x7f56a4102a10 2024-03-28T05:21:11.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.398+0000 7f56a3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a41033d0 0x7f56a419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:11.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.398+0000 7f56a3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a41033d0 0x7f56a419ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43574/0 (socket says 172.21.15.23:43574) 2024-03-28T05:21:11.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.398+0000 7f56a3fff640 1 -- 172.21.15.23:0/2696024667 learned_addr learned my addr 172.21.15.23:0/2696024667 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:11.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.398+0000 7f56a2ffd640 1 --2- 172.21.15.23:0/2696024667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f56a4102a10 0x7f56a4197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:11.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.398+0000 7f56a37fe640 1 --2- 172.21.15.23:0/2696024667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56a40ff9a0 0x7f56a4197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:11.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.399+0000 7f56a3fff640 1 -- 172.21.15.23:0/2696024667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f56a4102a10 msgr2=0x7f56a4197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:11.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.399+0000 7f56a3fff640 1 --2- 172.21.15.23:0/2696024667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f56a4102a10 0x7f56a4197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:11.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.399+0000 7f56a3fff640 1 -- 172.21.15.23:0/2696024667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56a40ff9a0 msgr2=0x7f56a4197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:11.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.399+0000 7f56a3fff640 1 --2- 172.21.15.23:0/2696024667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56a40ff9a0 0x7f56a4197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:11.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.399+0000 7f56a3fff640 1 -- 172.21.15.23:0/2696024667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a419f580 con 0x7f56a41033d0 2024-03-28T05:21:11.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.399+0000 7f56a3fff640 1 --2- 172.21.15.23:0/2696024667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a41033d0 0x7f56a419ef10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f569c009ca0 tx=0x7f569c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:11.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.399+0000 7f56a0ff9640 1 -- 172.21.15.23:0/2696024667 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c041020 con 0x7f56a41033d0 2024-03-28T05:21:11.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.399+0000 7f56a9ff4640 1 -- 172.21.15.23:0/2696024667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a419f810 con 0x7f56a41033d0 2024-03-28T05:21:11.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.399+0000 7f56a0ff9640 1 -- 172.21.15.23:0/2696024667 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f569c034cd0 con 0x7f56a41033d0 2024-03-28T05:21:11.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.399+0000 7f56a0ff9640 1 -- 172.21.15.23:0/2696024667 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c039040 con 0x7f56a41033d0 2024-03-28T05:21:11.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.399+0000 7f56a9ff4640 1 -- 172.21.15.23:0/2696024667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a419fd50 con 0x7f56a41033d0 2024-03-28T05:21:11.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.400+0000 7f56a9ff4640 1 -- 172.21.15.23:0/2696024667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56a4108bc0 con 0x7f56a41033d0 2024-03-28T05:21:11.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.401+0000 7f56a0ff9640 1 -- 172.21.15.23:0/2696024667 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f569c044020 con 0x7f56a41033d0 2024-03-28T05:21:11.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.402+0000 7f56a0ff9640 1 --2- 172.21.15.23:0/2696024667 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f568c0a2800 0x7f568c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:11.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.402+0000 7f56a0ff9640 1 -- 172.21.15.23:0/2696024667 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8770+0+0 (secure 0 0 0) 0x7f569c0f0e90 con 0x7f56a41033d0 2024-03-28T05:21:11.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.402+0000 7f56a37fe640 1 --2- 172.21.15.23:0/2696024667 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f568c0a2800 0x7f568c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:11.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.402+0000 7f56a37fe640 1 --2- 172.21.15.23:0/2696024667 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f568c0a2800 0x7f568c0a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f56a40699b0 tx=0x7f5694005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:11.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.406+0000 7f56a0ff9640 1 -- 172.21.15.23:0/2696024667 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f569c04b020 con 0x7f56a41033d0 2024-03-28T05:21:11.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:11 smithi113 ceph-mon[37164]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T05:21:11.492 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 28 05:21:11 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-i[68132]: 2024-03-28T05:21:11.153+0000 7f70f1469700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T05:21:11.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:11 smithi023 ceph-mon[31954]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T05:21:11.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:11.590+0000 7f56a9ff4640 1 -- 172.21.15.23:0/2696024667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f56a4062310 con 0x7f56a41033d0 2024-03-28T05:21:11.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:11 smithi098 ceph-mon[36087]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T05:21:12.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.268+0000 7f56a0ff9640 1 -- 172.21.15.23:0/2696024667 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) v1 ==== 169+0+0 (secure 0 0 0) 0x7f569c0bdda0 con 0x7f56a41033d0 2024-03-28T05:21:12.267 INFO:teuthology.orchestra.run.smithi023.stderr:set pool 4 allow_ec_overwrites to true 2024-03-28T05:21:12.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.271+0000 7f56a9ff4640 1 -- 172.21.15.23:0/2696024667 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f568c0a2800 msgr2=0x7f568c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:12.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.271+0000 7f56a9ff4640 1 --2- 172.21.15.23:0/2696024667 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f568c0a2800 0x7f568c0a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f56a40699b0 tx=0x7f5694005f90 comp rx=0 tx=0).stop 2024-03-28T05:21:12.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.271+0000 7f56a9ff4640 1 -- 172.21.15.23:0/2696024667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a41033d0 msgr2=0x7f56a419ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:12.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.271+0000 7f56a9ff4640 1 --2- 172.21.15.23:0/2696024667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a41033d0 0x7f56a419ef10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f569c009ca0 tx=0x7f569c032f20 comp rx=0 tx=0).stop 2024-03-28T05:21:12.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.271+0000 7f56a9ff4640 1 -- 172.21.15.23:0/2696024667 shutdown_connections 2024-03-28T05:21:12.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.271+0000 7f56a9ff4640 1 --2- 172.21.15.23:0/2696024667 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f568c0a2800 0x7f568c0a4cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:12.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.272+0000 7f56a9ff4640 1 --2- 172.21.15.23:0/2696024667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a41033d0 0x7f56a419ef10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:12.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.272+0000 7f56a9ff4640 1 --2- 172.21.15.23:0/2696024667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f56a4102a10 0x7f56a4197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:12.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.272+0000 7f56a9ff4640 1 --2- 172.21.15.23:0/2696024667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56a40ff9a0 0x7f56a4197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:12.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.272+0000 7f56a9ff4640 1 -- 172.21.15.23:0/2696024667 >> 172.21.15.23:0/2696024667 conn(0x7f56a40fb290 msgr2=0x7f56a4105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:12.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.272+0000 7f56a9ff4640 1 -- 172.21.15.23:0/2696024667 shutdown_connections 2024-03-28T05:21:12.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.272+0000 7f56a9ff4640 1 -- 172.21.15.23:0/2696024667 wait complete. 2024-03-28T05:21:12.292 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-28T05:21:12.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.405+0000 7f06a1c62640 1 -- 172.21.15.23:0/3696234670 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f069c10c0b0 msgr2=0x7f069c10c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:12.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.405+0000 7f06a1c62640 1 --2- 172.21.15.23:0/3696234670 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f069c10c0b0 0x7f069c10c4d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f068c009900 tx=0x7f068c02bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:12.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.405+0000 7f06a1c62640 1 -- 172.21.15.23:0/3696234670 shutdown_connections 2024-03-28T05:21:12.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.405+0000 7f06a1c62640 1 --2- 172.21.15.23:0/3696234670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f069c10dc60 0x7f069c1144f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:12.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.405+0000 7f06a1c62640 1 --2- 172.21.15.23:0/3696234670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f069c10d2a0 0x7f069c10d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:12.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.405+0000 7f06a1c62640 1 --2- 172.21.15.23:0/3696234670 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f069c10c0b0 0x7f069c10c4d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:12.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.405+0000 7f06a1c62640 1 -- 172.21.15.23:0/3696234670 >> 172.21.15.23:0/3696234670 conn(0x7f069c06c060 msgr2=0x7f069c06c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:12.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.405+0000 7f06a1c62640 1 -- 172.21.15.23:0/3696234670 shutdown_connections 2024-03-28T05:21:12.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.405+0000 7f06a1c62640 1 -- 172.21.15.23:0/3696234670 wait complete. 2024-03-28T05:21:12.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.406+0000 7f06a1c62640 1 Processor -- start 2024-03-28T05:21:12.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.406+0000 7f06a1c62640 1 -- start start 2024-03-28T05:21:12.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.406+0000 7f06a1c62640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f069c10c0b0 0x7f069c1a2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:12.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.406+0000 7f06a1c62640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f069c10d2a0 0x7f069c1a2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:12.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.406+0000 7f06a1c62640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f069c10dc60 0x7f069c1a99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:12.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.406+0000 7f06a1c62640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c117d40 con 0x7f069c10c0b0 2024-03-28T05:21:12.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.406+0000 7f06a1c62640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c117bc0 con 0x7f069c10dc60 2024-03-28T05:21:12.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.406+0000 7f06a1c62640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c117ec0 con 0x7f069c10d2a0 2024-03-28T05:21:12.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.406+0000 7f069b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f069c10c0b0 0x7f069c1a2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:12.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.406+0000 7f069b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f069c10c0b0 0x7f069c1a2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43596/0 (socket says 172.21.15.23:43596) 2024-03-28T05:21:12.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.406+0000 7f069b7fe640 1 -- 172.21.15.23:0/783142709 learned_addr learned my addr 172.21.15.23:0/783142709 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:12.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.406+0000 7f069affd640 1 --2- 172.21.15.23:0/783142709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f069c10d2a0 0x7f069c1a2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:12.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.406+0000 7f069bfff640 1 --2- 172.21.15.23:0/783142709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f069c10dc60 0x7f069c1a99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:12.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.407+0000 7f069b7fe640 1 -- 172.21.15.23:0/783142709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f069c10d2a0 msgr2=0x7f069c1a2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:12.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.407+0000 7f069b7fe640 1 --2- 172.21.15.23:0/783142709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f069c10d2a0 0x7f069c1a2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:12.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.407+0000 7f069b7fe640 1 -- 172.21.15.23:0/783142709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f069c10dc60 msgr2=0x7f069c1a99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:12.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.407+0000 7f069b7fe640 1 --2- 172.21.15.23:0/783142709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f069c10dc60 0x7f069c1a99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:12.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.407+0000 7f069b7fe640 1 -- 172.21.15.23:0/783142709 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069c1aa020 con 0x7f069c10c0b0 2024-03-28T05:21:12.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.407+0000 7f069bfff640 1 --2- 172.21.15.23:0/783142709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f069c10dc60 0x7f069c1a99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:21:12.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.407+0000 7f069b7fe640 1 --2- 172.21.15.23:0/783142709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f069c10c0b0 0x7f069c1a2400 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f068c000c00 tx=0x7f068c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:12.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.407+0000 7f0698ff9640 1 -- 172.21.15.23:0/783142709 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068c041070 con 0x7f069c10c0b0 2024-03-28T05:21:12.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.407+0000 7f0698ff9640 1 -- 172.21.15.23:0/783142709 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f068c0044d0 con 0x7f069c10c0b0 2024-03-28T05:21:12.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.407+0000 7f06a1c62640 1 -- 172.21.15.23:0/783142709 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069c1aa2b0 con 0x7f069c10c0b0 2024-03-28T05:21:12.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.407+0000 7f0698ff9640 1 -- 172.21.15.23:0/783142709 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068c034440 con 0x7f069c10c0b0 2024-03-28T05:21:12.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.407+0000 7f06a1c62640 1 -- 172.21.15.23:0/783142709 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f069c1aa7a0 con 0x7f069c10c0b0 2024-03-28T05:21:12.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.408+0000 7f06a1c62640 1 -- 172.21.15.23:0/783142709 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f069c113470 con 0x7f069c10c0b0 2024-03-28T05:21:12.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.409+0000 7f0698ff9640 1 -- 172.21.15.23:0/783142709 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f068c044070 con 0x7f069c10c0b0 2024-03-28T05:21:12.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.410+0000 7f0698ff9640 1 --2- 172.21.15.23:0/783142709 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f06840a2800 0x7f06840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:12.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.410+0000 7f069affd640 1 --2- 172.21.15.23:0/783142709 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f06840a2800 0x7f06840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:12.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.410+0000 7f0698ff9640 1 -- 172.21.15.23:0/783142709 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8770+0+0 (secure 0 0 0) 0x7f068c0f0820 con 0x7f069c10c0b0 2024-03-28T05:21:12.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.410+0000 7f069affd640 1 --2- 172.21.15.23:0/783142709 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f06840a2800 0x7f06840a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f06900095f0 tx=0x7f0690005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:12.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.414+0000 7f0698ff9640 1 -- 172.21.15.23:0/783142709 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f068c0bd6a0 con 0x7f069c10c0b0 2024-03-28T05:21:12.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:12 smithi023 ceph-mon[31954]: pgmap v287: 97 pgs: 25 creating+peering, 8 unknown, 64 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 3 op/s 2024-03-28T05:21:12.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:12 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1338069887' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T05:21:12.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:12 smithi023 ceph-mon[31954]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T05:21:12.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:12 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2696024667' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-28T05:21:12.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:12 smithi098 ceph-mon[36087]: pgmap v287: 97 pgs: 25 creating+peering, 8 unknown, 64 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 3 op/s 2024-03-28T05:21:12.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:12 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1338069887' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T05:21:12.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:12 smithi098 ceph-mon[36087]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T05:21:12.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:12 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2696024667' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-28T05:21:12.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:12.640+0000 7f06a1c62640 1 -- 172.21.15.23:0/783142709 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f069c02d290 con 0x7f069c10c0b0 2024-03-28T05:21:12.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:12 smithi113 ceph-mon[37164]: pgmap v287: 97 pgs: 25 creating+peering, 8 unknown, 64 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 3 op/s 2024-03-28T05:21:12.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:12 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1338069887' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T05:21:12.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:12 smithi113 ceph-mon[37164]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T05:21:12.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:12 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2696024667' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-28T05:21:13.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.277+0000 7f0698ff9640 1 -- 172.21.15.23:0/783142709 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v16) v1 ==== 137+0+0 (secure 0 0 0) 0x7f068c0bcfa0 con 0x7f069c10c0b0 2024-03-28T05:21:13.275 INFO:teuthology.orchestra.run.smithi023.stderr:added data pool 4 to fsmap 2024-03-28T05:21:13.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.280+0000 7f06a1c62640 1 -- 172.21.15.23:0/783142709 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f06840a2800 msgr2=0x7f06840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:13.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.280+0000 7f06a1c62640 1 --2- 172.21.15.23:0/783142709 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f06840a2800 0x7f06840a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f06900095f0 tx=0x7f0690005f90 comp rx=0 tx=0).stop 2024-03-28T05:21:13.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.280+0000 7f06a1c62640 1 -- 172.21.15.23:0/783142709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f069c10c0b0 msgr2=0x7f069c1a2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:13.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.280+0000 7f06a1c62640 1 --2- 172.21.15.23:0/783142709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f069c10c0b0 0x7f069c1a2400 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f068c000c00 tx=0x7f068c005d50 comp rx=0 tx=0).stop 2024-03-28T05:21:13.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.280+0000 7f06a1c62640 1 -- 172.21.15.23:0/783142709 shutdown_connections 2024-03-28T05:21:13.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.280+0000 7f06a1c62640 1 --2- 172.21.15.23:0/783142709 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f06840a2800 0x7f06840a4cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.280+0000 7f06a1c62640 1 --2- 172.21.15.23:0/783142709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f069c10dc60 0x7f069c1a99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.280+0000 7f06a1c62640 1 --2- 172.21.15.23:0/783142709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f069c10d2a0 0x7f069c1a2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.280+0000 7f06a1c62640 1 --2- 172.21.15.23:0/783142709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f069c10c0b0 0x7f069c1a2400 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.280+0000 7f06a1c62640 1 -- 172.21.15.23:0/783142709 >> 172.21.15.23:0/783142709 conn(0x7f069c06c060 msgr2=0x7f069c110640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:13.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.280+0000 7f06a1c62640 1 -- 172.21.15.23:0/783142709 shutdown_connections 2024-03-28T05:21:13.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.280+0000 7f06a1c62640 1 -- 172.21.15.23:0/783142709 wait complete. 2024-03-28T05:21:13.291 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:21:13.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.400+0000 7f9914abd640 1 -- 172.21.15.23:0/4192618921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99101017c0 msgr2=0x7f9910101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:13.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.400+0000 7f9914abd640 1 --2- 172.21.15.23:0/4192618921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99101017c0 0x7f9910101c40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f98f8009900 tx=0x7f98f802bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:13.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.400+0000 7f9914abd640 1 -- 172.21.15.23:0/4192618921 shutdown_connections 2024-03-28T05:21:13.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.400+0000 7f9914abd640 1 --2- 172.21.15.23:0/4192618921 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9910102180 0x7f9910108a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.400+0000 7f9914abd640 1 --2- 172.21.15.23:0/4192618921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99101017c0 0x7f9910101c40 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.400+0000 7f9914abd640 1 --2- 172.21.15.23:0/4192618921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99101005d0 0x7f99101009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.400+0000 7f9914abd640 1 -- 172.21.15.23:0/4192618921 >> 172.21.15.23:0/4192618921 conn(0x7f99100fbb80 msgr2=0x7f99100fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:13.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.400+0000 7f9914abd640 1 -- 172.21.15.23:0/4192618921 shutdown_connections 2024-03-28T05:21:13.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.400+0000 7f9914abd640 1 -- 172.21.15.23:0/4192618921 wait complete. 2024-03-28T05:21:13.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.401+0000 7f9914abd640 1 Processor -- start 2024-03-28T05:21:13.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.401+0000 7f9914abd640 1 -- start start 2024-03-28T05:21:13.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.401+0000 7f9914abd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99101005d0 0x7f9910199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:13.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.401+0000 7f9914abd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99101017c0 0x7f991019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:13.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.401+0000 7f9914abd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9910102180 0x7f99101a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:13.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.401+0000 7f9914abd640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991010c250 con 0x7f9910102180 2024-03-28T05:21:13.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.401+0000 7f9914abd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991010c0d0 con 0x7f99101005d0 2024-03-28T05:21:13.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.401+0000 7f9914abd640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991010c3d0 con 0x7f99101017c0 2024-03-28T05:21:13.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.401+0000 7f990ed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9910102180 0x7f99101a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:13.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.401+0000 7f990ed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9910102180 0x7f99101a11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57514/0 (socket says 172.21.15.23:57514) 2024-03-28T05:21:13.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.401+0000 7f990ed9d640 1 -- 172.21.15.23:0/3384291851 learned_addr learned my addr 172.21.15.23:0/3384291851 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:13.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.401+0000 7f9907fff640 1 --2- 172.21.15.23:0/3384291851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99101017c0 0x7f991019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:13.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.401+0000 7f990e59c640 1 --2- 172.21.15.23:0/3384291851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99101005d0 0x7f9910199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:13.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.402+0000 7f990ed9d640 1 -- 172.21.15.23:0/3384291851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99101017c0 msgr2=0x7f991019a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:13.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.402+0000 7f990ed9d640 1 --2- 172.21.15.23:0/3384291851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99101017c0 0x7f991019a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.402+0000 7f990ed9d640 1 -- 172.21.15.23:0/3384291851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99101005d0 msgr2=0x7f9910199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:13.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.402+0000 7f990ed9d640 1 --2- 172.21.15.23:0/3384291851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99101005d0 0x7f9910199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.402+0000 7f990ed9d640 1 -- 172.21.15.23:0/3384291851 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99101a1880 con 0x7f9910102180 2024-03-28T05:21:13.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.402+0000 7f9907fff640 1 --2- 172.21.15.23:0/3384291851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99101017c0 0x7f991019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:21:13.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.402+0000 7f990e59c640 1 --2- 172.21.15.23:0/3384291851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99101005d0 0x7f9910199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:21:13.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.402+0000 7f990ed9d640 1 --2- 172.21.15.23:0/3384291851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9910102180 0x7f99101a11e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f990800eb10 tx=0x7f990800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:13.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.402+0000 7f99077fe640 1 -- 172.21.15.23:0/3384291851 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9908004b20 con 0x7f9910102180 2024-03-28T05:21:13.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.402+0000 7f99077fe640 1 -- 172.21.15.23:0/3384291851 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9908004cc0 con 0x7f9910102180 2024-03-28T05:21:13.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.402+0000 7f99077fe640 1 -- 172.21.15.23:0/3384291851 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9908010760 con 0x7f9910102180 2024-03-28T05:21:13.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.402+0000 7f9914abd640 1 -- 172.21.15.23:0/3384291851 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99101a1b70 con 0x7f9910102180 2024-03-28T05:21:13.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.402+0000 7f9914abd640 1 -- 172.21.15.23:0/3384291851 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99101a2070 con 0x7f9910102180 2024-03-28T05:21:13.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.403+0000 7f9914abd640 1 -- 172.21.15.23:0/3384291851 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99101a2240 con 0x7f9910102180 2024-03-28T05:21:13.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.404+0000 7f99077fe640 1 -- 172.21.15.23:0/3384291851 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9908005000 con 0x7f9910102180 2024-03-28T05:21:13.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.405+0000 7f99077fe640 1 --2- 172.21.15.23:0/3384291851 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f98fc0a2800 0x7f98fc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:13.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.405+0000 7f990e59c640 1 --2- 172.21.15.23:0/3384291851 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f98fc0a2800 0x7f98fc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:13.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.405+0000 7f99077fe640 1 -- 172.21.15.23:0/3384291851 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f99080cf810 con 0x7f9910102180 2024-03-28T05:21:13.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.405+0000 7f990e59c640 1 --2- 172.21.15.23:0/3384291851 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f98fc0a2800 0x7f98fc0a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9900005fd0 tx=0x7f9900005ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:13.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.408+0000 7f99077fe640 1 -- 172.21.15.23:0/3384291851 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f990809c670 con 0x7f9910102180 2024-03-28T05:21:13.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:13 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2696024667' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-28T05:21:13.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:13 smithi023 ceph-mon[31954]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T05:21:13.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:13 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/783142709' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-28T05:21:13.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:13 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2696024667' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-28T05:21:13.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:13 smithi098 ceph-mon[36087]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T05:21:13.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:13 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/783142709' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-28T05:21:13.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.632+0000 7f9914abd640 1 -- 172.21.15.23:0/3384291851 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99101a2580 con 0x7f9910102180 2024-03-28T05:21:13.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.633+0000 7f99077fe640 1 -- 172.21.15.23:0/3384291851 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+11315 (secure 0 0 0) 0x7f990809bf70 con 0x7f9910102180 2024-03-28T05:21:13.631 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:21:13.631 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/733913400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":733913400},{"type":"v1","addr":"172.21.15.23:6835","nonce":733913400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1972901387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1972901387},{"type":"v1","addr":"172.21.15.23:6837","nonce":1972901387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/2390673788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2390673788},{"type":"v1","addr":"172.21.15.23:6839","nonce":2390673788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/289263156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":289263156},{"type":"v1","addr":"172.21.15.23:6841","nonce":289263156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3313932732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3313932732},{"type":"v1","addr":"172.21.15.98:6835","nonce":3313932732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2695340024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2695340024},{"type":"v1","addr":"172.21.15.113:6835","nonce":2695340024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2750367469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2750367469},{"type":"v1","addr":"172.21.15.98:6837","nonce":2750367469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/3222726274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":3222726274},{"type":"v1","addr":"172.21.15.113:6837","nonce":3222726274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24473,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/144113007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":144113007},{"type":"v1","addr":"172.21.15.98:6839","nonce":144113007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T05:21:07.616553+0000","modified":"2024-03-28T05:21:12.641348+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24487":{"gid":24487,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.113:6839/212060489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":212060489},{"type":"v1","addr":"172.21.15.113:6839","nonce":212060489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24487,"qdb_cluster":[24487]},"id":1}]} 2024-03-28T05:21:13.631 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 16 2024-03-28T05:21:13.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.635+0000 7f9914abd640 1 -- 172.21.15.23:0/3384291851 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f98fc0a2800 msgr2=0x7f98fc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:13.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.635+0000 7f9914abd640 1 --2- 172.21.15.23:0/3384291851 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f98fc0a2800 0x7f98fc0a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9900005fd0 tx=0x7f9900005ee0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.635+0000 7f9914abd640 1 -- 172.21.15.23:0/3384291851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9910102180 msgr2=0x7f99101a11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:13.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.635+0000 7f9914abd640 1 --2- 172.21.15.23:0/3384291851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9910102180 0x7f99101a11e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f990800eb10 tx=0x7f990800c2d0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.635+0000 7f9914abd640 1 -- 172.21.15.23:0/3384291851 shutdown_connections 2024-03-28T05:21:13.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.635+0000 7f9914abd640 1 --2- 172.21.15.23:0/3384291851 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f98fc0a2800 0x7f98fc0a4cc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.635+0000 7f9914abd640 1 --2- 172.21.15.23:0/3384291851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9910102180 0x7f99101a11e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.635+0000 7f9914abd640 1 --2- 172.21.15.23:0/3384291851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99101017c0 0x7f991019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.635+0000 7f9914abd640 1 --2- 172.21.15.23:0/3384291851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99101005d0 0x7f9910199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.635+0000 7f9914abd640 1 -- 172.21.15.23:0/3384291851 >> 172.21.15.23:0/3384291851 conn(0x7f99100fbb80 msgr2=0x7f99100fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:13.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.635+0000 7f9914abd640 1 -- 172.21.15.23:0/3384291851 shutdown_connections 2024-03-28T05:21:13.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.635+0000 7f9914abd640 1 -- 172.21.15.23:0/3384291851 wait complete. 2024-03-28T05:21:13.644 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T05:21:13.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:13 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2696024667' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-28T05:21:13.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:13 smithi113 ceph-mon[37164]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T05:21:13.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:13 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/783142709' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-28T05:21:13.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.748+0000 7f94d0e12640 1 -- 172.21.15.23:0/146225306 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94cc1017b0 msgr2=0x7f94cc101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:13.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.748+0000 7f94d0e12640 1 --2- 172.21.15.23:0/146225306 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94cc1017b0 0x7f94cc101c30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009900 tx=0x7f94bc02bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:13.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.749+0000 7f94d0e12640 1 -- 172.21.15.23:0/146225306 shutdown_connections 2024-03-28T05:21:13.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.749+0000 7f94d0e12640 1 --2- 172.21.15.23:0/146225306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94cc102170 0x7f94cc108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.749+0000 7f94d0e12640 1 --2- 172.21.15.23:0/146225306 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94cc1017b0 0x7f94cc101c30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.749+0000 7f94d0e12640 1 --2- 172.21.15.23:0/146225306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94cc1005c0 0x7f94cc1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.749+0000 7f94d0e12640 1 -- 172.21.15.23:0/146225306 >> 172.21.15.23:0/146225306 conn(0x7f94cc0fbb30 msgr2=0x7f94cc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:13.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.749+0000 7f94d0e12640 1 -- 172.21.15.23:0/146225306 shutdown_connections 2024-03-28T05:21:13.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.749+0000 7f94d0e12640 1 -- 172.21.15.23:0/146225306 wait complete. 2024-03-28T05:21:13.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.749+0000 7f94d0e12640 1 Processor -- start 2024-03-28T05:21:13.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.749+0000 7f94d0e12640 1 -- start start 2024-03-28T05:21:13.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.749+0000 7f94d0e12640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94cc1005c0 0x7f94cc0693d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:13.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.749+0000 7f94d0e12640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94cc1017b0 0x7f94cc19c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:13.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.749+0000 7f94d0e12640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94cc102170 0x7f94cc19ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:13.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.750+0000 7f94d0e12640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc10c220 con 0x7f94cc1017b0 2024-03-28T05:21:13.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.750+0000 7f94d0e12640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc10c0a0 con 0x7f94cc1005c0 2024-03-28T05:21:13.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.750+0000 7f94d0e12640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc10c3a0 con 0x7f94cc102170 2024-03-28T05:21:13.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.750+0000 7f94c9d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94cc1017b0 0x7f94cc19c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:13.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.750+0000 7f94c9d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94cc1017b0 0x7f94cc19c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57538/0 (socket says 172.21.15.23:57538) 2024-03-28T05:21:13.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.750+0000 7f94c9d9b640 1 -- 172.21.15.23:0/1121982811 learned_addr learned my addr 172.21.15.23:0/1121982811 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:13.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.750+0000 7f94cad9d640 1 --2- 172.21.15.23:0/1121982811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94cc102170 0x7f94cc19ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:13.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.750+0000 7f94ca59c640 1 --2- 172.21.15.23:0/1121982811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94cc1005c0 0x7f94cc0693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:13.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.750+0000 7f94c9d9b640 1 -- 172.21.15.23:0/1121982811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94cc102170 msgr2=0x7f94cc19ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:13.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.750+0000 7f94c9d9b640 1 --2- 172.21.15.23:0/1121982811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94cc102170 0x7f94cc19ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.750+0000 7f94c9d9b640 1 -- 172.21.15.23:0/1121982811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94cc1005c0 msgr2=0x7f94cc0693d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:13.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.750+0000 7f94c9d9b640 1 --2- 172.21.15.23:0/1121982811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94cc1005c0 0x7f94cc0693d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.750+0000 7f94c9d9b640 1 -- 172.21.15.23:0/1121982811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94cc19f620 con 0x7f94cc1017b0 2024-03-28T05:21:13.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.750+0000 7f94ca59c640 1 --2- 172.21.15.23:0/1121982811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94cc1005c0 0x7f94cc0693d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:21:13.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.750+0000 7f94c9d9b640 1 --2- 172.21.15.23:0/1121982811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94cc1017b0 0x7f94cc19c6f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009230 tx=0x7f94bc005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:13.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.750+0000 7f94bb7fe640 1 -- 172.21.15.23:0/1121982811 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc041070 con 0x7f94cc1017b0 2024-03-28T05:21:13.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.750+0000 7f94d0e12640 1 -- 172.21.15.23:0/1121982811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94cc19f8b0 con 0x7f94cc1017b0 2024-03-28T05:21:13.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.750+0000 7f94bb7fe640 1 -- 172.21.15.23:0/1121982811 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94bc0044d0 con 0x7f94cc1017b0 2024-03-28T05:21:13.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.750+0000 7f94bb7fe640 1 -- 172.21.15.23:0/1121982811 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc034440 con 0x7f94cc1017b0 2024-03-28T05:21:13.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.750+0000 7f94d0e12640 1 -- 172.21.15.23:0/1121982811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94cc19fda0 con 0x7f94cc1017b0 2024-03-28T05:21:13.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.751+0000 7f94d0e12640 1 -- 172.21.15.23:0/1121982811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94cc107900 con 0x7f94cc1017b0 2024-03-28T05:21:13.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.752+0000 7f94bb7fe640 1 -- 172.21.15.23:0/1121982811 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f94bc044070 con 0x7f94cc1017b0 2024-03-28T05:21:13.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.753+0000 7f94bb7fe640 1 --2- 172.21.15.23:0/1121982811 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f94b40a2800 0x7f94b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:13.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.753+0000 7f94ca59c640 1 --2- 172.21.15.23:0/1121982811 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f94b40a2800 0x7f94b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:13.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.753+0000 7f94bb7fe640 1 -- 172.21.15.23:0/1121982811 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f94bc0f0cb0 con 0x7f94cc1017b0 2024-03-28T05:21:13.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.753+0000 7f94ca59c640 1 --2- 172.21.15.23:0/1121982811 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f94b40a2800 0x7f94b40a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f94cc101610 tx=0x7f94c0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:13.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.757+0000 7f94bb7fe640 1 -- 172.21.15.23:0/1121982811 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f94bc0bdb10 con 0x7f94cc1017b0 2024-03-28T05:21:13.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.942+0000 7f94d0e12640 1 -- 172.21.15.23:0/1121982811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94cc02d280 con 0x7f94cc1017b0 2024-03-28T05:21:13.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.944+0000 7f94bb7fe640 1 -- 172.21.15.23:0/1121982811 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24082 (secure 0 0 0) 0x7f94bc0bd410 con 0x7f94cc1017b0 2024-03-28T05:21:13.942 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:21:13.943 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":89,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","created":"2024-03-28T05:13:13.453653+0000","modified":"2024-03-28T05:21:12.641185+0000","last_up_change":"2024-03-28T05:20:20.221527+0000","last_in_change":"2024-03-28T05:20:00.365478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T05:16:32.986974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T05:21:06.103894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T05:21:06.827190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T05:21:10.507418+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"18150942-59a4-4c3b-83d4-c1fc3ccf1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6803","nonce":1219956881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6805","nonce":1219956881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6809","nonce":1219956881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6807","nonce":1219956881}]},"public_addr":"172.21.15.23:6803/1219956881","cluster_addr":"172.21.15.23:6805/1219956881","heartbeat_back_addr":"172.21.15.23:6809/1219956881","heartbeat_front_addr":"172.21.15.23:6807/1219956881","state":["exists","up"]},{"osd":1,"uuid":"ba6d1a95-5135-4670-95f9-7878c0e7bd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6801","nonce":555491234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6803","nonce":555491234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6807","nonce":555491234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6805","nonce":555491234}]},"public_addr":"172.21.15.98:6801/555491234","cluster_addr":"172.21.15.98:6803/555491234","heartbeat_back_addr":"172.21.15.98:6807/555491234","heartbeat_front_addr":"172.21.15.98:6805/555491234","state":["exists","up"]},{"osd":2,"uuid":"873bdd60-2a47-403b-b1f4-ee9226a95067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6801","nonce":1535856914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6803","nonce":1535856914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6807","nonce":1535856914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6805","nonce":1535856914}]},"public_addr":"172.21.15.113:6801/1535856914","cluster_addr":"172.21.15.113:6803/1535856914","heartbeat_back_addr":"172.21.15.113:6807/1535856914","heartbeat_front_addr":"172.21.15.113:6805/1535856914","state":["exists","up"]},{"osd":3,"uuid":"83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6811","nonce":2949415020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6813","nonce":2949415020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6817","nonce":2949415020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6815","nonce":2949415020}]},"public_addr":"172.21.15.23:6811/2949415020","cluster_addr":"172.21.15.23:6813/2949415020","heartbeat_back_addr":"172.21.15.23:6817/2949415020","heartbeat_front_addr":"172.21.15.23:6815/2949415020","state":["exists","up"]},{"osd":4,"uuid":"76a90bac-b221-40a3-8238-40e8049cec36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6809","nonce":2615902819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6811","nonce":2615902819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6815","nonce":2615902819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6813","nonce":2615902819}]},"public_addr":"172.21.15.98:6809/2615902819","cluster_addr":"172.21.15.98:6811/2615902819","heartbeat_back_addr":"172.21.15.98:6815/2615902819","heartbeat_front_addr":"172.21.15.98:6813/2615902819","state":["exists","up"]},{"osd":5,"uuid":"b53fdb55-a8df-40a8-bd1b-8bcef634e26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6809","nonce":625271528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6811","nonce":625271528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6815","nonce":625271528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6813","nonce":625271528}]},"public_addr":"172.21.15.113:6809/625271528","cluster_addr":"172.21.15.113:6811/625271528","heartbeat_back_addr":"172.21.15.113:6815/625271528","heartbeat_front_addr":"172.21.15.113:6813/625271528","state":["exists","up"]},{"osd":6,"uuid":"f00283d2-8d67-4e97-a575-9a9d3b55d5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6819","nonce":4188464176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6821","nonce":4188464176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6825","nonce":4188464176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6823","nonce":4188464176}]},"public_addr":"172.21.15.23:6819/4188464176","cluster_addr":"172.21.15.23:6821/4188464176","heartbeat_back_addr":"172.21.15.23:6825/4188464176","heartbeat_front_addr":"172.21.15.23:6823/4188464176","state":["exists","up"]},{"osd":7,"uuid":"edc008a7-b07b-40a8-88b7-1614ba38a775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6817","nonce":1985546028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6819","nonce":1985546028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6823","nonce":1985546028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6821","nonce":1985546028}]},"public_addr":"172.21.15.98:6817/1985546028","cluster_addr":"172.21.15.98:6819/1985546028","heartbeat_back_addr":"172.21.15.98:6823/1985546028","heartbeat_front_addr":"172.21.15.98:6821/1985546028","state":["exists","up"]},{"osd":8,"uuid":"56e7b2d1-81f0-47b2-8638-f29832bed8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6817","nonce":3534876081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6819","nonce":3534876081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6823","nonce":3534876081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6821","nonce":3534876081}]},"public_addr":"172.21.15.113:6817/3534876081","cluster_addr":"172.21.15.113:6819/3534876081","heartbeat_back_addr":"172.21.15.113:6823/3534876081","heartbeat_front_addr":"172.21.15.113:6821/3534876081","state":["exists","up"]},{"osd":9,"uuid":"a2e90caa-6e9c-42bf-9f18-81f7f47880cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6827","nonce":672264207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6829","nonce":672264207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6833","nonce":672264207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6831","nonce":672264207}]},"public_addr":"172.21.15.23:6827/672264207","cluster_addr":"172.21.15.23:6829/672264207","heartbeat_back_addr":"172.21.15.23:6833/672264207","heartbeat_front_addr":"172.21.15.23:6831/672264207","state":["exists","up"]},{"osd":10,"uuid":"0eda917f-020c-437f-95ae-b26203f6c68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6825","nonce":2699920595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6827","nonce":2699920595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6831","nonce":2699920595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6829","nonce":2699920595}]},"public_addr":"172.21.15.98:6825/2699920595","cluster_addr":"172.21.15.98:6827/2699920595","heartbeat_back_addr":"172.21.15.98:6831/2699920595","heartbeat_front_addr":"172.21.15.98:6829/2699920595","state":["exists","up"]},{"osd":11,"uuid":"aae78501-78a8-469a-be20-3006f5ec5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6825","nonce":2113644020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6827","nonce":2113644020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6831","nonce":2113644020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6829","nonce":2113644020}]},"public_addr":"172.21.15.113:6825/2113644020","cluster_addr":"172.21.15.113:6827/2113644020","heartbeat_back_addr":"172.21.15.113:6831/2113644020","heartbeat_front_addr":"172.21.15.113:6829/2113644020","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:15:43.527475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:07.006538+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:30.594176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:55.810999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:20.595811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:45.474138+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:10.500409+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:35.787638+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:01.001188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:26.761115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:52.952752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:20:18.559832+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:6800/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1876451381":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2253559862":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2653562762":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/3337463830":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6800/1735221490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/2784771635":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2389902490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6801/1735221490":"2024-03-29T05:13:42.579607+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T05:21:13.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.946+0000 7f94d0e12640 1 -- 172.21.15.23:0/1121982811 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f94b40a2800 msgr2=0x7f94b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:13.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.946+0000 7f94d0e12640 1 --2- 172.21.15.23:0/1121982811 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f94b40a2800 0x7f94b40a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f94cc101610 tx=0x7f94c0005f90 comp rx=0 tx=0).stop 2024-03-28T05:21:13.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.946+0000 7f94d0e12640 1 -- 172.21.15.23:0/1121982811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94cc1017b0 msgr2=0x7f94cc19c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:13.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.946+0000 7f94d0e12640 1 --2- 172.21.15.23:0/1121982811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94cc1017b0 0x7f94cc19c6f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009230 tx=0x7f94bc005d50 comp rx=0 tx=0).stop 2024-03-28T05:21:13.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.946+0000 7f94d0e12640 1 -- 172.21.15.23:0/1121982811 shutdown_connections 2024-03-28T05:21:13.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.946+0000 7f94d0e12640 1 --2- 172.21.15.23:0/1121982811 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f94b40a2800 0x7f94b40a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.946+0000 7f94d0e12640 1 --2- 172.21.15.23:0/1121982811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94cc102170 0x7f94cc19ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.946+0000 7f94d0e12640 1 --2- 172.21.15.23:0/1121982811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94cc1017b0 0x7f94cc19c6f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.946+0000 7f94d0e12640 1 --2- 172.21.15.23:0/1121982811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94cc1005c0 0x7f94cc0693d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:13.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.946+0000 7f94d0e12640 1 -- 172.21.15.23:0/1121982811 >> 172.21.15.23:0/1121982811 conn(0x7f94cc0fbb30 msgr2=0x7f94cc104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:13.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.946+0000 7f94d0e12640 1 -- 172.21.15.23:0/1121982811 shutdown_connections 2024-03-28T05:21:13.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:13.946+0000 7f94d0e12640 1 -- 172.21.15.23:0/1121982811 wait complete. 2024-03-28T05:21:13.956 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T05:21:14.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.060+0000 7fd1cf79d640 1 -- 172.21.15.23:0/3136092650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1c8104a70 msgr2=0x7fd1c8106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:14.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.060+0000 7fd1cf79d640 1 --2- 172.21.15.23:0/3136092650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1c8104a70 0x7fd1c8106e60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8009900 tx=0x7fd1b802bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:14.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.060+0000 7fd1cf79d640 1 -- 172.21.15.23:0/3136092650 shutdown_connections 2024-03-28T05:21:14.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.060+0000 7fd1cf79d640 1 --2- 172.21.15.23:0/3136092650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1c81073a0 0x7fd1c81097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.061+0000 7fd1cf79d640 1 --2- 172.21.15.23:0/3136092650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1c8104a70 0x7fd1c8106e60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.061+0000 7fd1cf79d640 1 --2- 172.21.15.23:0/3136092650 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1c8102140 0x7fd1c8104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.061+0000 7fd1cf79d640 1 -- 172.21.15.23:0/3136092650 >> 172.21.15.23:0/3136092650 conn(0x7fd1c80fbb30 msgr2=0x7fd1c80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:14.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.061+0000 7fd1cf79d640 1 -- 172.21.15.23:0/3136092650 shutdown_connections 2024-03-28T05:21:14.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.061+0000 7fd1cf79d640 1 -- 172.21.15.23:0/3136092650 wait complete. 2024-03-28T05:21:14.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.061+0000 7fd1cf79d640 1 Processor -- start 2024-03-28T05:21:14.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.061+0000 7fd1cf79d640 1 -- start start 2024-03-28T05:21:14.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.061+0000 7fd1cf79d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1c8102140 0x7fd1c81979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:14.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.061+0000 7fd1cf79d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1c8104a70 0x7fd1c8197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:14.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.061+0000 7fd1cf79d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1c81073a0 0x7fd1c819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:14.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.061+0000 7fd1cf79d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c810d380 con 0x7fd1c8104a70 2024-03-28T05:21:14.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.061+0000 7fd1cf79d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c810d200 con 0x7fd1c81073a0 2024-03-28T05:21:14.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.061+0000 7fd1cf79d640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c810d500 con 0x7fd1c8102140 2024-03-28T05:21:14.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.061+0000 7fd1ccd38640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1c8104a70 0x7fd1c8197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:14.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.061+0000 7fd1ccd38640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1c8104a70 0x7fd1c8197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57564/0 (socket says 172.21.15.23:57564) 2024-03-28T05:21:14.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.061+0000 7fd1ccd38640 1 -- 172.21.15.23:0/2511399895 learned_addr learned my addr 172.21.15.23:0/2511399895 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:14.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.061+0000 7fd1cd539640 1 --2- 172.21.15.23:0/2511399895 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1c8102140 0x7fd1c81979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:14.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.062+0000 7fd1cdd3a640 1 --2- 172.21.15.23:0/2511399895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1c81073a0 0x7fd1c819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:14.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.062+0000 7fd1ccd38640 1 -- 172.21.15.23:0/2511399895 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1c8102140 msgr2=0x7fd1c81979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:14.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.062+0000 7fd1ccd38640 1 --2- 172.21.15.23:0/2511399895 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1c8102140 0x7fd1c81979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.062+0000 7fd1ccd38640 1 -- 172.21.15.23:0/2511399895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1c81073a0 msgr2=0x7fd1c819efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:14.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.062+0000 7fd1ccd38640 1 --2- 172.21.15.23:0/2511399895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1c81073a0 0x7fd1c819efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.062+0000 7fd1ccd38640 1 -- 172.21.15.23:0/2511399895 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c819f610 con 0x7fd1c8104a70 2024-03-28T05:21:14.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.062+0000 7fd1ccd38640 1 --2- 172.21.15.23:0/2511399895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1c8104a70 0x7fd1c8197f30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8000c00 tx=0x7fd1b802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:14.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.062+0000 7fd1be7fc640 1 -- 172.21.15.23:0/2511399895 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8043020 con 0x7fd1c8104a70 2024-03-28T05:21:14.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.062+0000 7fd1be7fc640 1 -- 172.21.15.23:0/2511399895 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1b80054e0 con 0x7fd1c8104a70 2024-03-28T05:21:14.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.062+0000 7fd1cf79d640 1 -- 172.21.15.23:0/2511399895 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c819f8a0 con 0x7fd1c8104a70 2024-03-28T05:21:14.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.062+0000 7fd1be7fc640 1 -- 172.21.15.23:0/2511399895 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8003ce0 con 0x7fd1c8104a70 2024-03-28T05:21:14.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.062+0000 7fd1cf79d640 1 -- 172.21.15.23:0/2511399895 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c819fe10 con 0x7fd1c8104a70 2024-03-28T05:21:14.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.063+0000 7fd1cf79d640 1 -- 172.21.15.23:0/2511399895 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1c80fd620 con 0x7fd1c8104a70 2024-03-28T05:21:14.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.064+0000 7fd1be7fc640 1 -- 172.21.15.23:0/2511399895 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd1b8046020 con 0x7fd1c8104a70 2024-03-28T05:21:14.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.064+0000 7fd1be7fc640 1 --2- 172.21.15.23:0/2511399895 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd1b40a2880 0x7fd1b40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:14.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.065+0000 7fd1cd539640 1 --2- 172.21.15.23:0/2511399895 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd1b40a2880 0x7fd1b40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:14.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.065+0000 7fd1be7fc640 1 -- 172.21.15.23:0/2511399895 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd1b80f1140 con 0x7fd1c8104a70 2024-03-28T05:21:14.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.065+0000 7fd1cd539640 1 --2- 172.21.15.23:0/2511399895 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd1b40a2880 0x7fd1b40a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4009bd0 tx=0x7fd1c4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:14.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.068+0000 7fd1be7fc640 1 -- 172.21.15.23:0/2511399895 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd1b804d020 con 0x7fd1c8104a70 2024-03-28T05:21:14.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.254+0000 7fd1cf79d640 1 -- 172.21.15.23:0/2511399895 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1c802d2e0 con 0x7fd1c8104a70 2024-03-28T05:21:14.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.255+0000 7fd1be7fc640 1 -- 172.21.15.23:0/2511399895 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24082 (secure 0 0 0) 0x7fd1b80bdfa0 con 0x7fd1c8104a70 2024-03-28T05:21:14.254 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:21:14.254 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":89,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","created":"2024-03-28T05:13:13.453653+0000","modified":"2024-03-28T05:21:12.641185+0000","last_up_change":"2024-03-28T05:20:20.221527+0000","last_in_change":"2024-03-28T05:20:00.365478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T05:16:32.986974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T05:21:06.103894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T05:21:06.827190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T05:21:10.507418+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"18150942-59a4-4c3b-83d4-c1fc3ccf1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6803","nonce":1219956881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6805","nonce":1219956881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6809","nonce":1219956881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6807","nonce":1219956881}]},"public_addr":"172.21.15.23:6803/1219956881","cluster_addr":"172.21.15.23:6805/1219956881","heartbeat_back_addr":"172.21.15.23:6809/1219956881","heartbeat_front_addr":"172.21.15.23:6807/1219956881","state":["exists","up"]},{"osd":1,"uuid":"ba6d1a95-5135-4670-95f9-7878c0e7bd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6801","nonce":555491234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6803","nonce":555491234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6807","nonce":555491234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6805","nonce":555491234}]},"public_addr":"172.21.15.98:6801/555491234","cluster_addr":"172.21.15.98:6803/555491234","heartbeat_back_addr":"172.21.15.98:6807/555491234","heartbeat_front_addr":"172.21.15.98:6805/555491234","state":["exists","up"]},{"osd":2,"uuid":"873bdd60-2a47-403b-b1f4-ee9226a95067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6801","nonce":1535856914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6803","nonce":1535856914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6807","nonce":1535856914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6805","nonce":1535856914}]},"public_addr":"172.21.15.113:6801/1535856914","cluster_addr":"172.21.15.113:6803/1535856914","heartbeat_back_addr":"172.21.15.113:6807/1535856914","heartbeat_front_addr":"172.21.15.113:6805/1535856914","state":["exists","up"]},{"osd":3,"uuid":"83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6811","nonce":2949415020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6813","nonce":2949415020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6817","nonce":2949415020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6815","nonce":2949415020}]},"public_addr":"172.21.15.23:6811/2949415020","cluster_addr":"172.21.15.23:6813/2949415020","heartbeat_back_addr":"172.21.15.23:6817/2949415020","heartbeat_front_addr":"172.21.15.23:6815/2949415020","state":["exists","up"]},{"osd":4,"uuid":"76a90bac-b221-40a3-8238-40e8049cec36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6809","nonce":2615902819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6811","nonce":2615902819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6815","nonce":2615902819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6813","nonce":2615902819}]},"public_addr":"172.21.15.98:6809/2615902819","cluster_addr":"172.21.15.98:6811/2615902819","heartbeat_back_addr":"172.21.15.98:6815/2615902819","heartbeat_front_addr":"172.21.15.98:6813/2615902819","state":["exists","up"]},{"osd":5,"uuid":"b53fdb55-a8df-40a8-bd1b-8bcef634e26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6809","nonce":625271528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6811","nonce":625271528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6815","nonce":625271528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6813","nonce":625271528}]},"public_addr":"172.21.15.113:6809/625271528","cluster_addr":"172.21.15.113:6811/625271528","heartbeat_back_addr":"172.21.15.113:6815/625271528","heartbeat_front_addr":"172.21.15.113:6813/625271528","state":["exists","up"]},{"osd":6,"uuid":"f00283d2-8d67-4e97-a575-9a9d3b55d5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6819","nonce":4188464176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6821","nonce":4188464176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6825","nonce":4188464176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6823","nonce":4188464176}]},"public_addr":"172.21.15.23:6819/4188464176","cluster_addr":"172.21.15.23:6821/4188464176","heartbeat_back_addr":"172.21.15.23:6825/4188464176","heartbeat_front_addr":"172.21.15.23:6823/4188464176","state":["exists","up"]},{"osd":7,"uuid":"edc008a7-b07b-40a8-88b7-1614ba38a775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6817","nonce":1985546028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6819","nonce":1985546028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6823","nonce":1985546028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6821","nonce":1985546028}]},"public_addr":"172.21.15.98:6817/1985546028","cluster_addr":"172.21.15.98:6819/1985546028","heartbeat_back_addr":"172.21.15.98:6823/1985546028","heartbeat_front_addr":"172.21.15.98:6821/1985546028","state":["exists","up"]},{"osd":8,"uuid":"56e7b2d1-81f0-47b2-8638-f29832bed8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6817","nonce":3534876081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6819","nonce":3534876081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6823","nonce":3534876081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6821","nonce":3534876081}]},"public_addr":"172.21.15.113:6817/3534876081","cluster_addr":"172.21.15.113:6819/3534876081","heartbeat_back_addr":"172.21.15.113:6823/3534876081","heartbeat_front_addr":"172.21.15.113:6821/3534876081","state":["exists","up"]},{"osd":9,"uuid":"a2e90caa-6e9c-42bf-9f18-81f7f47880cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6827","nonce":672264207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6829","nonce":672264207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6833","nonce":672264207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6831","nonce":672264207}]},"public_addr":"172.21.15.23:6827/672264207","cluster_addr":"172.21.15.23:6829/672264207","heartbeat_back_addr":"172.21.15.23:6833/672264207","heartbeat_front_addr":"172.21.15.23:6831/672264207","state":["exists","up"]},{"osd":10,"uuid":"0eda917f-020c-437f-95ae-b26203f6c68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6825","nonce":2699920595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6827","nonce":2699920595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6831","nonce":2699920595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6829","nonce":2699920595}]},"public_addr":"172.21.15.98:6825/2699920595","cluster_addr":"172.21.15.98:6827/2699920595","heartbeat_back_addr":"172.21.15.98:6831/2699920595","heartbeat_front_addr":"172.21.15.98:6829/2699920595","state":["exists","up"]},{"osd":11,"uuid":"aae78501-78a8-469a-be20-3006f5ec5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6825","nonce":2113644020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6827","nonce":2113644020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6831","nonce":2113644020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6829","nonce":2113644020}]},"public_addr":"172.21.15.113:6825/2113644020","cluster_addr":"172.21.15.113:6827/2113644020","heartbeat_back_addr":"172.21.15.113:6831/2113644020","heartbeat_front_addr":"172.21.15.113:6829/2113644020","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:15:43.527475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:07.006538+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:30.594176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:55.810999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:20.595811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:45.474138+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:10.500409+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:35.787638+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:01.001188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:26.761115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:52.952752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:20:18.559832+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:6800/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1876451381":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2253559862":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2653562762":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/3337463830":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6800/1735221490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/2784771635":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2389902490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6801/1735221490":"2024-03-29T05:13:42.579607+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T05:21:14.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.257+0000 7fd1cf79d640 1 -- 172.21.15.23:0/2511399895 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd1b40a2880 msgr2=0x7fd1b40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:14.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.257+0000 7fd1cf79d640 1 --2- 172.21.15.23:0/2511399895 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd1b40a2880 0x7fd1b40a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4009bd0 tx=0x7fd1c4005f90 comp rx=0 tx=0).stop 2024-03-28T05:21:14.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.257+0000 7fd1cf79d640 1 -- 172.21.15.23:0/2511399895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1c8104a70 msgr2=0x7fd1c8197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:14.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.257+0000 7fd1cf79d640 1 --2- 172.21.15.23:0/2511399895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1c8104a70 0x7fd1c8197f30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8000c00 tx=0x7fd1b802c6c0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.257+0000 7fd1cf79d640 1 -- 172.21.15.23:0/2511399895 shutdown_connections 2024-03-28T05:21:14.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.257+0000 7fd1cf79d640 1 --2- 172.21.15.23:0/2511399895 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd1b40a2880 0x7fd1b40a4d40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.257+0000 7fd1cf79d640 1 --2- 172.21.15.23:0/2511399895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1c81073a0 0x7fd1c819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.257+0000 7fd1cf79d640 1 --2- 172.21.15.23:0/2511399895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1c8104a70 0x7fd1c8197f30 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.257+0000 7fd1cf79d640 1 --2- 172.21.15.23:0/2511399895 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1c8102140 0x7fd1c81979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.257+0000 7fd1cf79d640 1 -- 172.21.15.23:0/2511399895 >> 172.21.15.23:0/2511399895 conn(0x7fd1c80fbb30 msgr2=0x7fd1c8102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:14.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.257+0000 7fd1cf79d640 1 -- 172.21.15.23:0/2511399895 shutdown_connections 2024-03-28T05:21:14.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.257+0000 7fd1cf79d640 1 -- 172.21.15.23:0/2511399895 wait complete. 2024-03-28T05:21:14.282 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:21:14.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.386+0000 7f2fa9f1d640 1 -- 172.21.15.23:0/630540150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fa4101b60 msgr2=0x7f2fa4101f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:14.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.386+0000 7f2fa9f1d640 1 --2- 172.21.15.23:0/630540150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fa4101b60 0x7f2fa4101f80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2f94009900 tx=0x7f2f9402bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:14.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.386+0000 7f2fa9f1d640 1 -- 172.21.15.23:0/630540150 shutdown_connections 2024-03-28T05:21:14.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.386+0000 7f2fa9f1d640 1 --2- 172.21.15.23:0/630540150 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fa4101250 0x7f2fa4108af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.386+0000 7f2fa9f1d640 1 --2- 172.21.15.23:0/630540150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fa4100890 0x7f2fa4100d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.386+0000 7f2fa9f1d640 1 --2- 172.21.15.23:0/630540150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fa4101b60 0x7f2fa4101f80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.386+0000 7f2fa9f1d640 1 -- 172.21.15.23:0/630540150 >> 172.21.15.23:0/630540150 conn(0x7f2fa40fb2c0 msgr2=0x7f2fa40fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:14.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.386+0000 7f2fa9f1d640 1 -- 172.21.15.23:0/630540150 shutdown_connections 2024-03-28T05:21:14.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.386+0000 7f2fa9f1d640 1 -- 172.21.15.23:0/630540150 wait complete. 2024-03-28T05:21:14.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.387+0000 7f2fa9f1d640 1 Processor -- start 2024-03-28T05:21:14.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.387+0000 7f2fa9f1d640 1 -- start start 2024-03-28T05:21:14.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.387+0000 7f2fa9f1d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fa4100890 0x7f2fa41979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:14.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.387+0000 7f2fa9f1d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fa4101250 0x7f2fa4197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:14.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.387+0000 7f2fa9f1d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fa4101b60 0x7f2fa419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:14.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.387+0000 7f2fa9f1d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa410d380 con 0x7f2fa4100890 2024-03-28T05:21:14.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.387+0000 7f2fa9f1d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa410d200 con 0x7f2fa4101b60 2024-03-28T05:21:14.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.387+0000 7f2fa9f1d640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa410d500 con 0x7f2fa4101250 2024-03-28T05:21:14.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.387+0000 7f2fa37fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fa4100890 0x7f2fa41979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:14.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.387+0000 7f2fa37fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fa4100890 0x7f2fa41979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57584/0 (socket says 172.21.15.23:57584) 2024-03-28T05:21:14.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.387+0000 7f2fa37fe640 1 -- 172.21.15.23:0/1474127142 learned_addr learned my addr 172.21.15.23:0/1474127142 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:14.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.387+0000 7f2fa2ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fa4101250 0x7f2fa4197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:14.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.387+0000 7f2fa3fff640 1 --2- 172.21.15.23:0/1474127142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fa4101b60 0x7f2fa419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:14.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.387+0000 7f2fa37fe640 1 -- 172.21.15.23:0/1474127142 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fa4101250 msgr2=0x7f2fa4197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:14.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.387+0000 7f2fa37fe640 1 --2- 172.21.15.23:0/1474127142 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fa4101250 0x7f2fa4197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.388+0000 7f2fa37fe640 1 -- 172.21.15.23:0/1474127142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fa4101b60 msgr2=0x7f2fa419ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:14.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.388+0000 7f2fa37fe640 1 --2- 172.21.15.23:0/1474127142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fa4101b60 0x7f2fa419ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.388+0000 7f2fa37fe640 1 -- 172.21.15.23:0/1474127142 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa419f600 con 0x7f2fa4100890 2024-03-28T05:21:14.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.388+0000 7f2fa3fff640 1 --2- 172.21.15.23:0/1474127142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fa4101b60 0x7f2fa419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:21:14.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.388+0000 7f2fa37fe640 1 --2- 172.21.15.23:0/1474127142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fa4100890 0x7f2fa41979e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2f9402efd0 tx=0x7f2f9402c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:14.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.388+0000 7f2fa0ff9640 1 -- 172.21.15.23:0/1474127142 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f94041070 con 0x7f2fa4100890 2024-03-28T05:21:14.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.388+0000 7f2fa0ff9640 1 -- 172.21.15.23:0/1474127142 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f94004450 con 0x7f2fa4100890 2024-03-28T05:21:14.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.388+0000 7f2fa9f1d640 1 -- 172.21.15.23:0/1474127142 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa419f890 con 0x7f2fa4100890 2024-03-28T05:21:14.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.388+0000 7f2fa0ff9640 1 -- 172.21.15.23:0/1474127142 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f94034440 con 0x7f2fa4100890 2024-03-28T05:21:14.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.388+0000 7f2fa9f1d640 1 -- 172.21.15.23:0/1474127142 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa419fdd0 con 0x7f2fa4100890 2024-03-28T05:21:14.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.389+0000 7f2fa9f1d640 1 -- 172.21.15.23:0/1474127142 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fa4107a40 con 0x7f2fa4100890 2024-03-28T05:21:14.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.390+0000 7f2fa0ff9640 1 -- 172.21.15.23:0/1474127142 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2f94044070 con 0x7f2fa4100890 2024-03-28T05:21:14.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.390+0000 7f2fa0ff9640 1 --2- 172.21.15.23:0/1474127142 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2f8c0a2880 0x7f2f8c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:14.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.391+0000 7f2fa2ffd640 1 --2- 172.21.15.23:0/1474127142 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2f8c0a2880 0x7f2f8c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:14.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.391+0000 7f2fa0ff9640 1 -- 172.21.15.23:0/1474127142 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f2f940f08b0 con 0x7f2fa4100890 2024-03-28T05:21:14.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.391+0000 7f2fa2ffd640 1 --2- 172.21.15.23:0/1474127142 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2f8c0a2880 0x7f2f8c0a4d40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4198ef0 tx=0x7f2f9800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:14.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.395+0000 7f2fa0ff9640 1 -- 172.21.15.23:0/1474127142 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2f940bd710 con 0x7f2fa4100890 2024-03-28T05:21:14.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:14 smithi023 ceph-mon[31954]: pgmap v290: 129 pgs: 2 creating+peering, 22 unknown, 105 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 8 op/s 2024-03-28T05:21:14.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:14 smithi023 ceph-mon[31954]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T05:21:14.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:14 smithi023 ceph-mon[31954]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T05:21:14.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:14 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/783142709' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-28T05:21:14.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:14 smithi023 ceph-mon[31954]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T05:21:14.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:14 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3384291851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:21:14.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:14 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1121982811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:14.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:14 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2511399895' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:14.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:14 smithi098 ceph-mon[36087]: pgmap v290: 129 pgs: 2 creating+peering, 22 unknown, 105 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 8 op/s 2024-03-28T05:21:14.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:14 smithi098 ceph-mon[36087]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T05:21:14.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:14 smithi098 ceph-mon[36087]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T05:21:14.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:14 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/783142709' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-28T05:21:14.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:14 smithi098 ceph-mon[36087]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T05:21:14.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:14 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3384291851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:21:14.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:14 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1121982811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:14.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:14 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2511399895' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:14.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.619+0000 7f2fa9f1d640 1 -- 172.21.15.23:0/1474127142 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fa402d070 con 0x7f2fa4100890 2024-03-28T05:21:14.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.620+0000 7f2fa0ff9640 1 -- 172.21.15.23:0/1474127142 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+11315 (secure 0 0 0) 0x7f2f940bd010 con 0x7f2fa4100890 2024-03-28T05:21:14.618 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:21:14.618 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/733913400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":733913400},{"type":"v1","addr":"172.21.15.23:6835","nonce":733913400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1972901387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1972901387},{"type":"v1","addr":"172.21.15.23:6837","nonce":1972901387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/2390673788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2390673788},{"type":"v1","addr":"172.21.15.23:6839","nonce":2390673788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/289263156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":289263156},{"type":"v1","addr":"172.21.15.23:6841","nonce":289263156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3313932732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3313932732},{"type":"v1","addr":"172.21.15.98:6835","nonce":3313932732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2695340024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2695340024},{"type":"v1","addr":"172.21.15.113:6835","nonce":2695340024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2750367469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2750367469},{"type":"v1","addr":"172.21.15.98:6837","nonce":2750367469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/3222726274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":3222726274},{"type":"v1","addr":"172.21.15.113:6837","nonce":3222726274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24473,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/144113007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":144113007},{"type":"v1","addr":"172.21.15.98:6839","nonce":144113007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T05:21:07.616553+0000","modified":"2024-03-28T05:21:12.641348+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24487":{"gid":24487,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.113:6839/212060489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":212060489},{"type":"v1","addr":"172.21.15.113:6839","nonce":212060489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24487,"qdb_cluster":[24487]},"id":1}]} 2024-03-28T05:21:14.618 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 16 2024-03-28T05:21:14.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.622+0000 7f2fa9f1d640 1 -- 172.21.15.23:0/1474127142 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2f8c0a2880 msgr2=0x7f2f8c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:14.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.622+0000 7f2fa9f1d640 1 --2- 172.21.15.23:0/1474127142 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2f8c0a2880 0x7f2f8c0a4d40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4198ef0 tx=0x7f2f9800b390 comp rx=0 tx=0).stop 2024-03-28T05:21:14.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.622+0000 7f2fa9f1d640 1 -- 172.21.15.23:0/1474127142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fa4100890 msgr2=0x7f2fa41979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:14.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.622+0000 7f2fa9f1d640 1 --2- 172.21.15.23:0/1474127142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fa4100890 0x7f2fa41979e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2f9402efd0 tx=0x7f2f9402c6c0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.622+0000 7f2fa9f1d640 1 -- 172.21.15.23:0/1474127142 shutdown_connections 2024-03-28T05:21:14.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.622+0000 7f2fa9f1d640 1 --2- 172.21.15.23:0/1474127142 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2f8c0a2880 0x7f2f8c0a4d40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.622+0000 7f2fa9f1d640 1 --2- 172.21.15.23:0/1474127142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fa4101b60 0x7f2fa419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.622+0000 7f2fa9f1d640 1 --2- 172.21.15.23:0/1474127142 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2fa4101250 0x7f2fa4197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.622+0000 7f2fa9f1d640 1 --2- 172.21.15.23:0/1474127142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fa4100890 0x7f2fa41979e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.622+0000 7f2fa9f1d640 1 -- 172.21.15.23:0/1474127142 >> 172.21.15.23:0/1474127142 conn(0x7f2fa40fb2c0 msgr2=0x7f2fa4104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:14.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.622+0000 7f2fa9f1d640 1 -- 172.21.15.23:0/1474127142 shutdown_connections 2024-03-28T05:21:14.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.622+0000 7f2fa9f1d640 1 -- 172.21.15.23:0/1474127142 wait complete. 2024-03-28T05:21:14.631 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T05:21:14.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.736+0000 7f3c07a75640 1 -- 172.21.15.23:0/3512350179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c00101830 msgr2=0x7f3c00101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:14.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.736+0000 7f3c07a75640 1 --2- 172.21.15.23:0/3512350179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c00101830 0x7f3c00101cb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc009900 tx=0x7f3bfc02bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:14.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.737+0000 7f3c07a75640 1 -- 172.21.15.23:0/3512350179 shutdown_connections 2024-03-28T05:21:14.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.737+0000 7f3c07a75640 1 --2- 172.21.15.23:0/3512350179 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c001021f0 0x7f3c00108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.737+0000 7f3c07a75640 1 --2- 172.21.15.23:0/3512350179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c00101830 0x7f3c00101cb0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.737+0000 7f3c07a75640 1 --2- 172.21.15.23:0/3512350179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c00100640 0x7f3c00100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.737+0000 7f3c07a75640 1 -- 172.21.15.23:0/3512350179 >> 172.21.15.23:0/3512350179 conn(0x7f3c000fbbd0 msgr2=0x7f3c000fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:14.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.737+0000 7f3c07a75640 1 -- 172.21.15.23:0/3512350179 shutdown_connections 2024-03-28T05:21:14.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.737+0000 7f3c07a75640 1 -- 172.21.15.23:0/3512350179 wait complete. 2024-03-28T05:21:14.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.737+0000 7f3c07a75640 1 Processor -- start 2024-03-28T05:21:14.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.737+0000 7f3c07a75640 1 -- start start 2024-03-28T05:21:14.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.737+0000 7f3c07a75640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c00100640 0x7f3c0010afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:14.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.737+0000 7f3c07a75640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c00101830 0x7f3c00109600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:14.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.737+0000 7f3c07a75640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c001021f0 0x7f3c00109b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:14.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.737+0000 7f3c07a75640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0010d370 con 0x7f3c00101830 2024-03-28T05:21:14.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.737+0000 7f3c07a75640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0010d1f0 con 0x7f3c001021f0 2024-03-28T05:21:14.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.737+0000 7f3c07a75640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0010d4f0 con 0x7f3c00100640 2024-03-28T05:21:14.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.738+0000 7f3c05010640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c00101830 0x7f3c00109600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:14.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.738+0000 7f3c05010640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c00101830 0x7f3c00109600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57606/0 (socket says 172.21.15.23:57606) 2024-03-28T05:21:14.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.738+0000 7f3c05010640 1 -- 172.21.15.23:0/2381611783 learned_addr learned my addr 172.21.15.23:0/2381611783 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:14.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.738+0000 7f3c05811640 1 --2- 172.21.15.23:0/2381611783 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c00100640 0x7f3c0010afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:14.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.738+0000 7f3c06012640 1 --2- 172.21.15.23:0/2381611783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c001021f0 0x7f3c00109b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:14.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.738+0000 7f3c05010640 1 -- 172.21.15.23:0/2381611783 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c00100640 msgr2=0x7f3c0010afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:14.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.738+0000 7f3c05010640 1 --2- 172.21.15.23:0/2381611783 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c00100640 0x7f3c0010afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.738+0000 7f3c05010640 1 -- 172.21.15.23:0/2381611783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c001021f0 msgr2=0x7f3c00109b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:14.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.738+0000 7f3c05010640 1 --2- 172.21.15.23:0/2381611783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c001021f0 0x7f3c00109b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.738+0000 7f3c05010640 1 -- 172.21.15.23:0/2381611783 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0010a3d0 con 0x7f3c00101830 2024-03-28T05:21:14.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.738+0000 7f3c05010640 1 --2- 172.21.15.23:0/2381611783 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c00101830 0x7f3c00109600 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc009230 tx=0x7f3bfc005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:14.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.738+0000 7f3bf2ffd640 1 -- 172.21.15.23:0/2381611783 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc041070 con 0x7f3c00101830 2024-03-28T05:21:14.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.738+0000 7f3bf2ffd640 1 -- 172.21.15.23:0/2381611783 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bfc0044d0 con 0x7f3c00101830 2024-03-28T05:21:14.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.738+0000 7f3bf2ffd640 1 -- 172.21.15.23:0/2381611783 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc034440 con 0x7f3c00101830 2024-03-28T05:21:14.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.738+0000 7f3c07a75640 1 -- 172.21.15.23:0/2381611783 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c001a1a90 con 0x7f3c00101830 2024-03-28T05:21:14.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.738+0000 7f3c07a75640 1 -- 172.21.15.23:0/2381611783 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c001a2030 con 0x7f3c00101830 2024-03-28T05:21:14.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.739+0000 7f3c07a75640 1 -- 172.21.15.23:0/2381611783 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c001079e0 con 0x7f3c00101830 2024-03-28T05:21:14.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.740+0000 7f3bf2ffd640 1 -- 172.21.15.23:0/2381611783 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3bfc044070 con 0x7f3c00101830 2024-03-28T05:21:14.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.741+0000 7f3bf2ffd640 1 --2- 172.21.15.23:0/2381611783 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3bec0a2880 0x7f3bec0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:14.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.741+0000 7f3c05811640 1 --2- 172.21.15.23:0/2381611783 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3bec0a2880 0x7f3bec0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:14.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.741+0000 7f3bf2ffd640 1 -- 172.21.15.23:0/2381611783 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f3bfc0f0b70 con 0x7f3c00101830 2024-03-28T05:21:14.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.741+0000 7f3c05811640 1 --2- 172.21.15.23:0/2381611783 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3bec0a2880 0x7f3bec0a4d40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3c00101690 tx=0x7f3bf400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:14.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:14 smithi113 ceph-mon[37164]: pgmap v290: 129 pgs: 2 creating+peering, 22 unknown, 105 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 8 op/s 2024-03-28T05:21:14.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:14 smithi113 ceph-mon[37164]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T05:21:14.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:14 smithi113 ceph-mon[37164]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T05:21:14.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:14 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/783142709' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-28T05:21:14.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:14 smithi113 ceph-mon[37164]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T05:21:14.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:14 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3384291851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:21:14.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:14 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1121982811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:14.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:14 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2511399895' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:14.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.745+0000 7f3bf2ffd640 1 -- 172.21.15.23:0/2381611783 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3bfc0bd9d0 con 0x7f3c00101830 2024-03-28T05:21:14.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.929+0000 7f3c07a75640 1 -- 172.21.15.23:0/2381611783 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c0002d240 con 0x7f3c00101830 2024-03-28T05:21:14.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.930+0000 7f3bf2ffd640 1 -- 172.21.15.23:0/2381611783 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24082 (secure 0 0 0) 0x7f3bfc0bd2d0 con 0x7f3c00101830 2024-03-28T05:21:14.929 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:21:14.929 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":89,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","created":"2024-03-28T05:13:13.453653+0000","modified":"2024-03-28T05:21:12.641185+0000","last_up_change":"2024-03-28T05:20:20.221527+0000","last_in_change":"2024-03-28T05:20:00.365478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T05:16:32.986974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T05:21:06.103894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T05:21:06.827190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T05:21:10.507418+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"18150942-59a4-4c3b-83d4-c1fc3ccf1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6803","nonce":1219956881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6805","nonce":1219956881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6809","nonce":1219956881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6807","nonce":1219956881}]},"public_addr":"172.21.15.23:6803/1219956881","cluster_addr":"172.21.15.23:6805/1219956881","heartbeat_back_addr":"172.21.15.23:6809/1219956881","heartbeat_front_addr":"172.21.15.23:6807/1219956881","state":["exists","up"]},{"osd":1,"uuid":"ba6d1a95-5135-4670-95f9-7878c0e7bd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6801","nonce":555491234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6803","nonce":555491234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6807","nonce":555491234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6805","nonce":555491234}]},"public_addr":"172.21.15.98:6801/555491234","cluster_addr":"172.21.15.98:6803/555491234","heartbeat_back_addr":"172.21.15.98:6807/555491234","heartbeat_front_addr":"172.21.15.98:6805/555491234","state":["exists","up"]},{"osd":2,"uuid":"873bdd60-2a47-403b-b1f4-ee9226a95067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6801","nonce":1535856914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6803","nonce":1535856914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6807","nonce":1535856914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6805","nonce":1535856914}]},"public_addr":"172.21.15.113:6801/1535856914","cluster_addr":"172.21.15.113:6803/1535856914","heartbeat_back_addr":"172.21.15.113:6807/1535856914","heartbeat_front_addr":"172.21.15.113:6805/1535856914","state":["exists","up"]},{"osd":3,"uuid":"83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6811","nonce":2949415020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6813","nonce":2949415020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6817","nonce":2949415020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6815","nonce":2949415020}]},"public_addr":"172.21.15.23:6811/2949415020","cluster_addr":"172.21.15.23:6813/2949415020","heartbeat_back_addr":"172.21.15.23:6817/2949415020","heartbeat_front_addr":"172.21.15.23:6815/2949415020","state":["exists","up"]},{"osd":4,"uuid":"76a90bac-b221-40a3-8238-40e8049cec36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6809","nonce":2615902819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6811","nonce":2615902819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6815","nonce":2615902819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6813","nonce":2615902819}]},"public_addr":"172.21.15.98:6809/2615902819","cluster_addr":"172.21.15.98:6811/2615902819","heartbeat_back_addr":"172.21.15.98:6815/2615902819","heartbeat_front_addr":"172.21.15.98:6813/2615902819","state":["exists","up"]},{"osd":5,"uuid":"b53fdb55-a8df-40a8-bd1b-8bcef634e26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6809","nonce":625271528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6811","nonce":625271528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6815","nonce":625271528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6813","nonce":625271528}]},"public_addr":"172.21.15.113:6809/625271528","cluster_addr":"172.21.15.113:6811/625271528","heartbeat_back_addr":"172.21.15.113:6815/625271528","heartbeat_front_addr":"172.21.15.113:6813/625271528","state":["exists","up"]},{"osd":6,"uuid":"f00283d2-8d67-4e97-a575-9a9d3b55d5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6819","nonce":4188464176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6821","nonce":4188464176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6825","nonce":4188464176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6823","nonce":4188464176}]},"public_addr":"172.21.15.23:6819/4188464176","cluster_addr":"172.21.15.23:6821/4188464176","heartbeat_back_addr":"172.21.15.23:6825/4188464176","heartbeat_front_addr":"172.21.15.23:6823/4188464176","state":["exists","up"]},{"osd":7,"uuid":"edc008a7-b07b-40a8-88b7-1614ba38a775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6817","nonce":1985546028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6819","nonce":1985546028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6823","nonce":1985546028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6821","nonce":1985546028}]},"public_addr":"172.21.15.98:6817/1985546028","cluster_addr":"172.21.15.98:6819/1985546028","heartbeat_back_addr":"172.21.15.98:6823/1985546028","heartbeat_front_addr":"172.21.15.98:6821/1985546028","state":["exists","up"]},{"osd":8,"uuid":"56e7b2d1-81f0-47b2-8638-f29832bed8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6817","nonce":3534876081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6819","nonce":3534876081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6823","nonce":3534876081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6821","nonce":3534876081}]},"public_addr":"172.21.15.113:6817/3534876081","cluster_addr":"172.21.15.113:6819/3534876081","heartbeat_back_addr":"172.21.15.113:6823/3534876081","heartbeat_front_addr":"172.21.15.113:6821/3534876081","state":["exists","up"]},{"osd":9,"uuid":"a2e90caa-6e9c-42bf-9f18-81f7f47880cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6827","nonce":672264207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6829","nonce":672264207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6833","nonce":672264207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6831","nonce":672264207}]},"public_addr":"172.21.15.23:6827/672264207","cluster_addr":"172.21.15.23:6829/672264207","heartbeat_back_addr":"172.21.15.23:6833/672264207","heartbeat_front_addr":"172.21.15.23:6831/672264207","state":["exists","up"]},{"osd":10,"uuid":"0eda917f-020c-437f-95ae-b26203f6c68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6825","nonce":2699920595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6827","nonce":2699920595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6831","nonce":2699920595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6829","nonce":2699920595}]},"public_addr":"172.21.15.98:6825/2699920595","cluster_addr":"172.21.15.98:6827/2699920595","heartbeat_back_addr":"172.21.15.98:6831/2699920595","heartbeat_front_addr":"172.21.15.98:6829/2699920595","state":["exists","up"]},{"osd":11,"uuid":"aae78501-78a8-469a-be20-3006f5ec5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6825","nonce":2113644020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6827","nonce":2113644020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6831","nonce":2113644020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6829","nonce":2113644020}]},"public_addr":"172.21.15.113:6825/2113644020","cluster_addr":"172.21.15.113:6827/2113644020","heartbeat_back_addr":"172.21.15.113:6831/2113644020","heartbeat_front_addr":"172.21.15.113:6829/2113644020","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:15:43.527475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:07.006538+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:30.594176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:55.810999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:20.595811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:45.474138+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:10.500409+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:35.787638+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:01.001188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:26.761115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:52.952752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:20:18.559832+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:6800/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1876451381":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2253559862":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2653562762":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/3337463830":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6800/1735221490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/2784771635":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2389902490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6801/1735221490":"2024-03-29T05:13:42.579607+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T05:21:14.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.932+0000 7f3c07a75640 1 -- 172.21.15.23:0/2381611783 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3bec0a2880 msgr2=0x7f3bec0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:14.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.932+0000 7f3c07a75640 1 --2- 172.21.15.23:0/2381611783 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3bec0a2880 0x7f3bec0a4d40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3c00101690 tx=0x7f3bf400a2d0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.932+0000 7f3c07a75640 1 -- 172.21.15.23:0/2381611783 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c00101830 msgr2=0x7f3c00109600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:14.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.932+0000 7f3c07a75640 1 --2- 172.21.15.23:0/2381611783 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c00101830 0x7f3c00109600 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc009230 tx=0x7f3bfc005d50 comp rx=0 tx=0).stop 2024-03-28T05:21:14.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.932+0000 7f3c07a75640 1 -- 172.21.15.23:0/2381611783 shutdown_connections 2024-03-28T05:21:14.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.932+0000 7f3c07a75640 1 --2- 172.21.15.23:0/2381611783 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3bec0a2880 0x7f3bec0a4d40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.932+0000 7f3c07a75640 1 --2- 172.21.15.23:0/2381611783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c001021f0 0x7f3c00109b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.932+0000 7f3c07a75640 1 --2- 172.21.15.23:0/2381611783 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c00101830 0x7f3c00109600 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.932+0000 7f3c07a75640 1 --2- 172.21.15.23:0/2381611783 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3c00100640 0x7f3c0010afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:14.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.932+0000 7f3c07a75640 1 -- 172.21.15.23:0/2381611783 >> 172.21.15.23:0/2381611783 conn(0x7f3c000fbbd0 msgr2=0x7f3c00104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:14.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.932+0000 7f3c07a75640 1 -- 172.21.15.23:0/2381611783 shutdown_connections 2024-03-28T05:21:14.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:14.932+0000 7f3c07a75640 1 -- 172.21.15.23:0/2381611783 wait complete. 2024-03-28T05:21:14.942 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T05:21:07.616553+0000', 'modified': '2024-03-28T05:21:12.641348+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24487}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24487': {'gid': 24487, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.113:6839/212060489', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6838', 'nonce': 212060489}, {'type': 'v1', 'addr': '172.21.15.113:6839', 'nonce': 212060489}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24487, 'qdb_cluster': [24487]} 2024-03-28T05:21:14.942 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T05:21:14.942 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-28T05:21:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.048+0000 7f76a56c6640 1 -- 172.21.15.23:0/2216626772 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76a01016c0 msgr2=0x7f76a0101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.048+0000 7f76a56c6640 1 --2- 172.21.15.23:0/2216626772 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76a01016c0 0x7f76a0101ae0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7690009900 tx=0x7f769002bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.048+0000 7f76a56c6640 1 -- 172.21.15.23:0/2216626772 shutdown_connections 2024-03-28T05:21:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.048+0000 7f76a56c6640 1 --2- 172.21.15.23:0/2216626772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76a0103270 0x7f76a0109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.048+0000 7f76a56c6640 1 --2- 172.21.15.23:0/2216626772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76a01028b0 0x7f76a0102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.048+0000 7f76a56c6640 1 --2- 172.21.15.23:0/2216626772 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76a01016c0 0x7f76a0101ae0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.048+0000 7f76a56c6640 1 -- 172.21.15.23:0/2216626772 >> 172.21.15.23:0/2216626772 conn(0x7f76a00fcc10 msgr2=0x7f76a00ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:15.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.048+0000 7f76a56c6640 1 -- 172.21.15.23:0/2216626772 shutdown_connections 2024-03-28T05:21:15.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.048+0000 7f76a56c6640 1 -- 172.21.15.23:0/2216626772 wait complete. 2024-03-28T05:21:15.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.048+0000 7f76a56c6640 1 Processor -- start 2024-03-28T05:21:15.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.048+0000 7f76a56c6640 1 -- start start 2024-03-28T05:21:15.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.049+0000 7f76a56c6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76a01016c0 0x7f76a01979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:15.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.049+0000 7f76a56c6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76a01028b0 0x7f76a0197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:15.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.049+0000 7f76a56c6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76a0103270 0x7f76a019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:15.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.049+0000 7f76a56c6640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a010d340 con 0x7f76a0103270 2024-03-28T05:21:15.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.049+0000 7f76a56c6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a010d1c0 con 0x7f76a01016c0 2024-03-28T05:21:15.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.049+0000 7f76a56c6640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a010d4c0 con 0x7f76a01028b0 2024-03-28T05:21:15.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.049+0000 7f769f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76a0103270 0x7f76a019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:15.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.049+0000 7f769f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76a0103270 0x7f76a019ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57632/0 (socket says 172.21.15.23:57632) 2024-03-28T05:21:15.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.049+0000 7f769f7fe640 1 -- 172.21.15.23:0/3574963163 learned_addr learned my addr 172.21.15.23:0/3574963163 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:15.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.049+0000 7f7697fff640 1 --2- 172.21.15.23:0/3574963163 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76a01028b0 0x7f76a0197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:15.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.049+0000 7f769effd640 1 --2- 172.21.15.23:0/3574963163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76a01016c0 0x7f76a01979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:15.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.049+0000 7f769f7fe640 1 -- 172.21.15.23:0/3574963163 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76a01028b0 msgr2=0x7f76a0197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:15.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.049+0000 7f769f7fe640 1 --2- 172.21.15.23:0/3574963163 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76a01028b0 0x7f76a0197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.049+0000 7f769f7fe640 1 -- 172.21.15.23:0/3574963163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76a01016c0 msgr2=0x7f76a01979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:15.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.049+0000 7f769f7fe640 1 --2- 172.21.15.23:0/3574963163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76a01016c0 0x7f76a01979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.049+0000 7f769f7fe640 1 -- 172.21.15.23:0/3574963163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a019f5f0 con 0x7f76a0103270 2024-03-28T05:21:15.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.049+0000 7f769f7fe640 1 --2- 172.21.15.23:0/3574963163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76a0103270 0x7f76a019ef80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f769800eb10 tx=0x7f769800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:15.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.049+0000 7f769cff9640 1 -- 172.21.15.23:0/3574963163 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7698004b20 con 0x7f76a0103270 2024-03-28T05:21:15.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.049+0000 7f76a56c6640 1 -- 172.21.15.23:0/3574963163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a019f8e0 con 0x7f76a0103270 2024-03-28T05:21:15.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.049+0000 7f769cff9640 1 -- 172.21.15.23:0/3574963163 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7698004cc0 con 0x7f76a0103270 2024-03-28T05:21:15.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.049+0000 7f769cff9640 1 -- 172.21.15.23:0/3574963163 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7698010760 con 0x7f76a0103270 2024-03-28T05:21:15.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.049+0000 7f76a56c6640 1 -- 172.21.15.23:0/3574963163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a019fe30 con 0x7f76a0103270 2024-03-28T05:21:15.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.051+0000 7f769cff9640 1 -- 172.21.15.23:0/3574963163 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7698005020 con 0x7f76a0103270 2024-03-28T05:21:15.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.052+0000 7f76a8391740 1 Processor -- start 2024-03-28T05:21:15.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.052+0000 7f76a8391740 1 -- start start 2024-03-28T05:21:15.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.052+0000 7f769cff9640 1 --2- 172.21.15.23:0/3574963163 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f768c0a2800 0x7f768c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:15.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.052+0000 7f7697fff640 1 --2- 172.21.15.23:0/3574963163 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f768c0a2800 0x7f768c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:15.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.052+0000 7f769cff9640 1 -- 172.21.15.23:0/3574963163 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f76980cf380 con 0x7f76a0103270 2024-03-28T05:21:15.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.052+0000 7f76a8391740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ca8821e740 0x55ca8821eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:15.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.052+0000 7f76a8391740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca8821f130 0x55ca88223c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:15.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.052+0000 7f76a8391740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca882242a0 0x55ca88226730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:15.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.052+0000 7f76a8391740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca88183c20 con 0x55ca882242a0 2024-03-28T05:21:15.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.052+0000 7f76a8391740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca8812ca40 con 0x55ca8821f130 2024-03-28T05:21:15.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.052+0000 7f76a8391740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca8817fcc0 con 0x55ca8821e740 2024-03-28T05:21:15.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.052+0000 7f769f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca882242a0 0x55ca88226730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:15.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.052+0000 7f769f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca882242a0 0x55ca88226730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57634/0 (socket says 172.21.15.23:57634) 2024-03-28T05:21:15.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.052+0000 7f769f7fe640 1 -- 172.21.15.23:0/4174150834 learned_addr learned my addr 172.21.15.23:0/4174150834 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:15.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.052+0000 7f769effd640 1 --2- 172.21.15.23:0/4174150834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ca8821e740 0x55ca8821eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:15.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.053+0000 7f7697fff640 1 --2- 172.21.15.23:0/4174150834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca8821f130 0x55ca88223c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:15.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.053+0000 7f7697fff640 1 --2- 172.21.15.23:0/3574963163 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f768c0a2800 0x7f768c0a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f76a0198ee0 tx=0x7f7688011440 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:15.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.053+0000 7f769f7fe640 1 -- 172.21.15.23:0/4174150834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ca8821e740 msgr2=0x55ca8821eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:21:15.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.053+0000 7f769f7fe640 1 --2- 172.21.15.23:0/4174150834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ca8821e740 0x55ca8821eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.053+0000 7f769f7fe640 1 -- 172.21.15.23:0/4174150834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca8821f130 msgr2=0x55ca88223c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:21:15.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.053+0000 7f769f7fe640 1 --2- 172.21.15.23:0/4174150834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca8821f130 0x55ca88223c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.053+0000 7f769f7fe640 1 -- 172.21.15.23:0/4174150834 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ca8814ef90 con 0x55ca882242a0 2024-03-28T05:21:15.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.053+0000 7f769effd640 1 --2- 172.21.15.23:0/4174150834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ca8821e740 0x55ca8821eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:21:15.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.053+0000 7f7697fff640 1 --2- 172.21.15.23:0/4174150834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca8821f130 0x55ca88223c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:21:15.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.053+0000 7f769f7fe640 1 --2- 172.21.15.23:0/4174150834 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca882242a0 0x55ca88226730 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7698000f80 tx=0x7f769809b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=552ccdb68c393838 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:15.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.053+0000 7f76957fa640 1 -- 172.21.15.23:0/4174150834 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769809bae0 con 0x55ca882242a0 2024-03-28T05:21:15.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.053+0000 7f76957fa640 1 -- 172.21.15.23:0/4174150834 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f769809fc30 con 0x55ca882242a0 2024-03-28T05:21:15.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.053+0000 7f76957fa640 1 -- 172.21.15.23:0/4174150834 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7698005570 con 0x55ca882242a0 2024-03-28T05:21:15.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.054+0000 7f76a8391740 1 -- 172.21.15.23:0/4174150834 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca882242a0 msgr2=0x55ca88226730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:15.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.054+0000 7f76a8391740 1 --2- 172.21.15.23:0/4174150834 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca882242a0 0x55ca88226730 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7698000f80 tx=0x7f769809b970 comp rx=0 tx=0).stop 2024-03-28T05:21:15.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.054+0000 7f76a8391740 1 -- 172.21.15.23:0/4174150834 shutdown_connections 2024-03-28T05:21:15.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.054+0000 7f76a8391740 1 --2- 172.21.15.23:0/4174150834 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca882242a0 0x55ca88226730 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.054+0000 7f76a8391740 1 --2- 172.21.15.23:0/4174150834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca8821f130 0x55ca88223c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.054+0000 7f76a8391740 1 --2- 172.21.15.23:0/4174150834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ca8821e740 0x55ca8821eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.054+0000 7f76a8391740 1 -- 172.21.15.23:0/4174150834 >> 172.21.15.23:0/4174150834 conn(0x55ca881e6a60 msgr2=0x55ca8821a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:15.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.054+0000 7f76a8391740 1 -- 172.21.15.23:0/4174150834 shutdown_connections 2024-03-28T05:21:15.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.054+0000 7f76a8391740 1 -- 172.21.15.23:0/4174150834 wait complete. 2024-03-28T05:21:15.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.054+0000 7f76a8391740 1 -- start start 2024-03-28T05:21:15.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.054+0000 7f76a8391740 1 Processor -- start 2024-03-28T05:21:15.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.054+0000 7f76a8391740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca8821e740 0x55ca8822cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:15.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.054+0000 7f76a8391740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ca8821f130 0x55ca8822d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:15.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.054+0000 7f76a8391740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca882242a0 0x55ca88234530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:15.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.054+0000 7f76a8391740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca8822b660 con 0x55ca8821e740 2024-03-28T05:21:15.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.054+0000 7f76a8391740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca8813fe90 con 0x55ca882242a0 2024-03-28T05:21:15.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.054+0000 7f76a8391740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca8822b7e0 con 0x55ca8821f130 2024-03-28T05:21:15.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.054+0000 7f76a8391740 10 client.0 _ll_register_callbacks cb 0x55ca87e97ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:21:15.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.054+0000 7f769effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca8821e740 0x55ca8822cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:15.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.054+0000 7f769effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca8821e740 0x55ca8822cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57646/0 (socket says 172.21.15.23:57646) 2024-03-28T05:21:15.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.055+0000 7f769effd640 1 -- 172.21.15.23:0/3801892339 learned_addr learned my addr 172.21.15.23:0/3801892339 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:15.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.055+0000 7f7697fff640 1 --2- 172.21.15.23:0/3801892339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ca8821f130 0x55ca8822d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:15.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.055+0000 7f769effd640 1 -- 172.21.15.23:0/3801892339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ca8821f130 msgr2=0x55ca8822d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:21:15.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.055+0000 7f769f7fe640 1 --2- 172.21.15.23:0/3801892339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca882242a0 0x55ca88234530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:15.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.055+0000 7f769effd640 1 --2- 172.21.15.23:0/3801892339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ca8821f130 0x55ca8822d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.055+0000 7f769effd640 1 -- 172.21.15.23:0/3801892339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca882242a0 msgr2=0x55ca88234530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:21:15.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.055+0000 7f769effd640 1 --2- 172.21.15.23:0/3801892339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca882242a0 0x55ca88234530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.055+0000 7f769effd640 1 -- 172.21.15.23:0/3801892339 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ca8814c9d0 con 0x55ca8821e740 2024-03-28T05:21:15.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.055+0000 7f769effd640 1 --2- 172.21.15.23:0/3801892339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca8821e740 0x55ca8822cfb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f769002cfd0 tx=0x7f7690004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbb659beb2a5ce03 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:15.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.055+0000 7f76867fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:21:15.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.055+0000 7f76867fc640 1 -- 172.21.15.23:0/3801892339 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7690005d50 con 0x55ca8821e740 2024-03-28T05:21:15.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.055+0000 7f76a8391740 1 -- 172.21.15.23:0/3801892339 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f769809fc30 con 0x55ca8821e740 2024-03-28T05:21:15.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.055+0000 7f76867fc640 1 -- 172.21.15.23:0/3801892339 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f7690005230 con 0x55ca8821e740 2024-03-28T05:21:15.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.055+0000 7f76867fc640 1 -- 172.21.15.23:0/3801892339 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7690005430 con 0x55ca8821e740 2024-03-28T05:21:15.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.055+0000 7f76a8391740 10 client.14766 fetch_fsmap learned FSMap version 16 2024-03-28T05:21:15.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.055+0000 7f76a8391740 1 -- 172.21.15.23:0/3801892339 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) v3 -- 0x55ca87e982d0 con 0x55ca8821e740 2024-03-28T05:21:15.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.055+0000 7f76867fc640 1 -- 172.21.15.23:0/3801892339 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f769809fc30 con 0x55ca8821e740 2024-03-28T05:21:15.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.056+0000 7f76867fc640 1 -- 172.21.15.23:0/3801892339 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 16) v1 ==== 7922+0+0 (secure 0 0 0) 0x7f7690003850 con 0x55ca8821e740 2024-03-28T05:21:15.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.056+0000 7f76a8391740 10 client.14766 fetch_fsmap finished waiting for FSMap version 16 2024-03-28T05:21:15.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.056+0000 7f76a8391740 10 client.14766 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T05:21:15.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.056+0000 7f76a8391740 20 client.14766 populate_metadata read hostname 'smithi023' 2024-03-28T05:21:15.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.056+0000 7f76a8391740 1 --2- 172.21.15.23:0/3801892339 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55ca882397f0 0x55ca88238870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:15.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.056+0000 7f76a8391740 4 client.14766 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:21:15.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.056+0000 7f76a8391740 1 -- 172.21.15.23:0/3801892339 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ca88191160 con 0x55ca882397f0 2024-03-28T05:21:15.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.056+0000 7f7697fff640 1 --2- 172.21.15.23:0/3801892339 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55ca882397f0 0x55ca88238870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:15.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.056+0000 7f7697fff640 1 --2- 172.21.15.23:0/3801892339 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55ca882397f0 0x55ca88238870 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4b4b4685490b852 server_cookie=17eaf09c230b86e in_seq=0 out_seq=0 2024-03-28T05:21:15.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.056+0000 7f76867fc640 10 client.14766 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:21:15.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.057+0000 7f76867fc640 1 -- 172.21.15.23:0/3801892339 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ca88191160 con 0x55ca882397f0 2024-03-28T05:21:15.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.057+0000 7f76867fc640 10 client.14766 handle_command_reply: tid=0 2024-03-28T05:21:15.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.057+0000 7f76a8391740 1 client.14766 shutdown 2024-03-28T05:21:15.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.057+0000 7f76a8391740 1 -- 172.21.15.23:0/3801892339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca8821e740 msgr2=0x55ca8822cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:15.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.057+0000 7f76a8391740 1 --2- 172.21.15.23:0/3801892339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca8821e740 0x55ca8822cfb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f769002cfd0 tx=0x7f7690004250 comp rx=0 tx=0).stop 2024-03-28T05:21:15.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.057+0000 7f76a8391740 1 -- 172.21.15.23:0/3801892339 shutdown_connections 2024-03-28T05:21:15.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.057+0000 7f76a8391740 1 --2- 172.21.15.23:0/3801892339 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55ca882397f0 0x55ca88238870 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.057+0000 7f76a8391740 1 --2- 172.21.15.23:0/3801892339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca882242a0 0x55ca88234530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.057+0000 7f76a8391740 1 --2- 172.21.15.23:0/3801892339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ca8821f130 0x55ca8822d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.057+0000 7f76867fc640 0 client.14766 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:21:15.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.057+0000 7f76a8391740 1 --2- 172.21.15.23:0/3801892339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca8821e740 0x55ca8822cfb0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.057+0000 7f76a8391740 1 -- 172.21.15.23:0/3801892339 >> 172.21.15.23:0/3801892339 conn(0x55ca881e6a60 msgr2=0x55ca882192f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:15.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.057+0000 7f76a8391740 1 -- 172.21.15.23:0/3801892339 shutdown_connections 2024-03-28T05:21:15.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.057+0000 7f76a8391740 1 -- 172.21.15.23:0/3801892339 wait complete. 2024-03-28T05:21:15.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.057+0000 7f76a8391740 20 client.14766 trim_cache size 0 max 16384 2024-03-28T05:21:15.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.074+0000 7f76a8391740 1 Processor -- start 2024-03-28T05:21:15.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.074+0000 7f76a8391740 1 -- start start 2024-03-28T05:21:15.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.074+0000 7f76a8391740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca8821e740 0x55ca8821fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:15.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.074+0000 7f76a8391740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca8821f130 0x55ca88234450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:15.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.074+0000 7f76a8391740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ca882242a0 0x55ca8821d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:15.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.074+0000 7f76a8391740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca88239010 con 0x55ca8821f130 2024-03-28T05:21:15.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.074+0000 7f76a8391740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca88239670 con 0x55ca8821e740 2024-03-28T05:21:15.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.074+0000 7f76a8391740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca88239cd0 con 0x55ca882242a0 2024-03-28T05:21:15.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.074+0000 7f7697fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca8821f130 0x55ca88234450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:15.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.074+0000 7f7697fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca8821f130 0x55ca88234450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57658/0 (socket says 172.21.15.23:57658) 2024-03-28T05:21:15.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.074+0000 7f7697fff640 1 -- 172.21.15.23:0/3560849341 learned_addr learned my addr 172.21.15.23:0/3560849341 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:15.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.074+0000 7f769f7fe640 1 --2- 172.21.15.23:0/3560849341 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ca882242a0 0x55ca8821d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:15.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.074+0000 7f769effd640 1 --2- 172.21.15.23:0/3560849341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca8821e740 0x55ca8821fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:15.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.074+0000 7f7697fff640 1 -- 172.21.15.23:0/3560849341 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ca882242a0 msgr2=0x55ca8821d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:21:15.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.074+0000 7f7697fff640 1 --2- 172.21.15.23:0/3560849341 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ca882242a0 0x55ca8821d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.074+0000 7f7697fff640 1 -- 172.21.15.23:0/3560849341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca8821e740 msgr2=0x55ca8821fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:21:15.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.074+0000 7f7697fff640 1 --2- 172.21.15.23:0/3560849341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca8821e740 0x55ca8821fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.074+0000 7f7697fff640 1 -- 172.21.15.23:0/3560849341 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ca87f626b0 con 0x55ca8821f130 2024-03-28T05:21:15.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.075+0000 7f7697fff640 1 --2- 172.21.15.23:0/3560849341 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca8821f130 0x55ca88234450 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x55ca8821f530 tx=0x7f768801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ead3d03644dc4d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:15.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.075+0000 7f7694ff9640 1 -- 172.21.15.23:0/3560849341 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7688017040 con 0x55ca8821f130 2024-03-28T05:21:15.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.075+0000 7f7694ff9640 1 -- 172.21.15.23:0/3560849341 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7688019050 con 0x55ca8821f130 2024-03-28T05:21:15.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.075+0000 7f7694ff9640 1 -- 172.21.15.23:0/3560849341 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76880047a0 con 0x55ca8821f130 2024-03-28T05:21:15.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.075+0000 7f76a8391740 1 -- 172.21.15.23:0/3560849341 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca8821f130 msgr2=0x55ca88234450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:15.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.075+0000 7f76a8391740 1 --2- 172.21.15.23:0/3560849341 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca8821f130 0x55ca88234450 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x55ca8821f530 tx=0x7f768801c000 comp rx=0 tx=0).stop 2024-03-28T05:21:15.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.075+0000 7f76a8391740 1 -- 172.21.15.23:0/3560849341 shutdown_connections 2024-03-28T05:21:15.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.075+0000 7f76a8391740 1 --2- 172.21.15.23:0/3560849341 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ca882242a0 0x55ca8821d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.075+0000 7f76a8391740 1 --2- 172.21.15.23:0/3560849341 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca8821f130 0x55ca88234450 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.075+0000 7f76a8391740 1 --2- 172.21.15.23:0/3560849341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca8821e740 0x55ca8821fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.075+0000 7f76a8391740 1 -- 172.21.15.23:0/3560849341 >> 172.21.15.23:0/3560849341 conn(0x55ca881e6a60 msgr2=0x55ca88219bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:15.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.075+0000 7f76a8391740 1 -- 172.21.15.23:0/3560849341 shutdown_connections 2024-03-28T05:21:15.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.075+0000 7f76a8391740 1 -- 172.21.15.23:0/3560849341 wait complete. 2024-03-28T05:21:15.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.076+0000 7f76a8391740 1 -- start start 2024-03-28T05:21:15.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.076+0000 7f76a8391740 1 Processor -- start 2024-03-28T05:21:15.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.076+0000 7f76a8391740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ca8821e740 0x55ca882305c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:15.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.076+0000 7f76a8391740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca8821f130 0x55ca88230b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:15.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.076+0000 7f76a8391740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca882242a0 0x55ca88233b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:15.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.076+0000 7f76a8391740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca8823a990 con 0x55ca882242a0 2024-03-28T05:21:15.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.076+0000 7f76a8391740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca8823a330 con 0x55ca8821f130 2024-03-28T05:21:15.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.076+0000 7f76a8391740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca8823aff0 con 0x55ca8821e740 2024-03-28T05:21:15.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.076+0000 7f76a8391740 10 client.0 _ll_register_callbacks cb 0x7f769002c650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:21:15.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.076+0000 7f769f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca882242a0 0x55ca88233b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:15.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.076+0000 7f769f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca882242a0 0x55ca88233b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57662/0 (socket says 172.21.15.23:57662) 2024-03-28T05:21:15.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.076+0000 7f769effd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ca8821e740 0x55ca882305c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:15.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.076+0000 7f769f7fe640 1 -- 172.21.15.23:0/1830314202 learned_addr learned my addr 172.21.15.23:0/1830314202 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:15.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.076+0000 7f769f7fe640 1 -- 172.21.15.23:0/1830314202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ca8821e740 msgr2=0x55ca882305c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:21:15.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.076+0000 7f769f7fe640 1 --2- 172.21.15.23:0/1830314202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ca8821e740 0x55ca882305c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.076+0000 7f769f7fe640 1 -- 172.21.15.23:0/1830314202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca8821f130 msgr2=0x55ca88230b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:21:15.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.076+0000 7f769f7fe640 1 --2- 172.21.15.23:0/1830314202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca8821f130 0x55ca88230b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.076+0000 7f769f7fe640 1 -- 172.21.15.23:0/1830314202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ca88234110 con 0x55ca882242a0 2024-03-28T05:21:15.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.076+0000 7f769f7fe640 1 --2- 172.21.15.23:0/1830314202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca882242a0 0x55ca88233b20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f76980ccfb0 tx=0x7f7698097e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7c237fe00e8e03 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:15.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.076+0000 7f76867fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:21:15.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.077+0000 7f76867fc640 1 -- 172.21.15.23:0/1830314202 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769809bae0 con 0x55ca882242a0 2024-03-28T05:21:15.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.077+0000 7f76867fc640 1 -- 172.21.15.23:0/1830314202 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f76980b4ba0 con 0x55ca882242a0 2024-03-28T05:21:15.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.077+0000 7f76a8391740 1 -- 172.21.15.23:0/1830314202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7688019050 con 0x55ca882242a0 2024-03-28T05:21:15.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.077+0000 7f76867fc640 1 -- 172.21.15.23:0/1830314202 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f769809fc30 con 0x55ca882242a0 2024-03-28T05:21:15.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.077+0000 7f76a8391740 10 client.14772 fetch_fsmap learned FSMap version 16 2024-03-28T05:21:15.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.077+0000 7f76a8391740 1 -- 172.21.15.23:0/1830314202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) v3 -- 0x55ca882951d0 con 0x55ca882242a0 2024-03-28T05:21:15.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.077+0000 7f76867fc640 1 -- 172.21.15.23:0/1830314202 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7688019050 con 0x55ca882242a0 2024-03-28T05:21:15.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.077+0000 7f76867fc640 1 -- 172.21.15.23:0/1830314202 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 16) v1 ==== 7922+0+0 (secure 0 0 0) 0x7f769809af30 con 0x55ca882242a0 2024-03-28T05:21:15.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.077+0000 7f76a8391740 10 client.14772 fetch_fsmap finished waiting for FSMap version 16 2024-03-28T05:21:15.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.077+0000 7f76a8391740 10 client.14772 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T05:21:15.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.077+0000 7f76a8391740 20 client.14772 populate_metadata read hostname 'smithi023' 2024-03-28T05:21:15.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.077+0000 7f76a8391740 1 --2- 172.21.15.23:0/1830314202 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55ca88299440 0x55ca882953a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:15.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.077+0000 7f76a8391740 4 client.14772 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T05:21:15.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.077+0000 7f76a8391740 1 -- 172.21.15.23:0/1830314202 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ca8818b4b0 con 0x55ca88299440 2024-03-28T05:21:15.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.077+0000 7f769effd640 1 --2- 172.21.15.23:0/1830314202 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55ca88299440 0x55ca882953a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:15.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.078+0000 7f76867fc640 10 client.14772 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:21:15.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.078+0000 7f769effd640 1 --2- 172.21.15.23:0/1830314202 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55ca88299440 0x55ca882953a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb4177e6c14afd1b server_cookie=43443fe25e6c68ec in_seq=0 out_seq=0 2024-03-28T05:21:15.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.078+0000 7f76867fc640 1 -- 172.21.15.23:0/1830314202 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0x55ca8818b4b0 con 0x55ca88299440 2024-03-28T05:21:15.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.078+0000 7f76867fc640 10 client.14772 handle_command_reply: tid=0 2024-03-28T05:21:15.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.078+0000 7f76a8391740 1 client.14772 shutdown 2024-03-28T05:21:15.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.078+0000 7f76a8391740 1 -- 172.21.15.23:0/1830314202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca882242a0 msgr2=0x55ca88233b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:15.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.078+0000 7f76a8391740 1 --2- 172.21.15.23:0/1830314202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca882242a0 0x55ca88233b20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f76980ccfb0 tx=0x7f7698097e80 comp rx=0 tx=0).stop 2024-03-28T05:21:15.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.078+0000 7f76a8391740 1 -- 172.21.15.23:0/1830314202 shutdown_connections 2024-03-28T05:21:15.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.078+0000 7f76a8391740 1 --2- 172.21.15.23:0/1830314202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ca882242a0 0x55ca88233b20 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.078+0000 7f76a8391740 1 --2- 172.21.15.23:0/1830314202 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55ca88299440 0x55ca882953a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.078+0000 7f76a8391740 1 --2- 172.21.15.23:0/1830314202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ca8821f130 0x55ca88230b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.078+0000 7f76867fc640 0 client.14772 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:21:15.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.078+0000 7f76a8391740 1 --2- 172.21.15.23:0/1830314202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ca8821e740 0x55ca882305c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.078+0000 7f76a8391740 1 -- 172.21.15.23:0/1830314202 >> 172.21.15.23:0/1830314202 conn(0x55ca881e6a60 msgr2=0x55ca8821d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:15.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.078+0000 7f76a8391740 1 -- 172.21.15.23:0/1830314202 shutdown_connections 2024-03-28T05:21:15.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.078+0000 7f76a8391740 1 -- 172.21.15.23:0/1830314202 wait complete. 2024-03-28T05:21:15.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.078+0000 7f76a8391740 20 client.14772 trim_cache size 0 max 16384 2024-03-28T05:21:15.085 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T05:21:15.085 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "bf37ebc4-ecc1-11ee-95d0-87774f69a715", 2024-03-28T05:21:15.085 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 0, 2024-03-28T05:21:15.086 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24487, 2024-03-28T05:21:15.086 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-28T05:21:15.086 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-28T05:21:15.086 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-28T05:21:15.086 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 6.830054842, 2024-03-28T05:21:15.086 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 16, 2024-03-28T05:21:15.086 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 89, 2024-03-28T05:21:15.086 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 89, 2024-03-28T05:21:15.086 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 19.583700124 2024-03-28T05:21:15.086 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T05:21:15.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.079+0000 7f76a56c6640 1 -- 172.21.15.23:0/3574963163 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f768c0a2800 msgr2=0x7f768c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:15.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.079+0000 7f76a56c6640 1 --2- 172.21.15.23:0/3574963163 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f768c0a2800 0x7f768c0a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f76a0198ee0 tx=0x7f7688011440 comp rx=0 tx=0).stop 2024-03-28T05:21:15.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.079+0000 7f76a56c6640 1 -- 172.21.15.23:0/3574963163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76a0103270 msgr2=0x7f76a019ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:15.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.079+0000 7f76a56c6640 1 --2- 172.21.15.23:0/3574963163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76a0103270 0x7f76a019ef80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f769800eb10 tx=0x7f769800c2d0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.079+0000 7f76a56c6640 1 -- 172.21.15.23:0/3574963163 shutdown_connections 2024-03-28T05:21:15.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.079+0000 7f76a56c6640 1 --2- 172.21.15.23:0/3574963163 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f768c0a2800 0x7f768c0a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.079+0000 7f76a56c6640 1 --2- 172.21.15.23:0/3574963163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76a0103270 0x7f76a019ef80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.079+0000 7f76a56c6640 1 --2- 172.21.15.23:0/3574963163 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f76a01028b0 0x7f76a0197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.079+0000 7f76a56c6640 1 --2- 172.21.15.23:0/3574963163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76a01016c0 0x7f76a01979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:15.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.079+0000 7f76a56c6640 1 -- 172.21.15.23:0/3574963163 >> 172.21.15.23:0/3574963163 conn(0x7f76a00fcc10 msgr2=0x7f76a0105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:15.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.079+0000 7f76a56c6640 1 -- 172.21.15.23:0/3574963163 shutdown_connections 2024-03-28T05:21:15.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:15.079+0000 7f76a56c6640 1 -- 172.21.15.23:0/3574963163 wait complete. 2024-03-28T05:21:15.089 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T05:21:15.089 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-28T05:21:15.089 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph fs ls 2024-03-28T05:21:15.331 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:21:15.560 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:15 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1474127142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:21:15.560 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:15 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2381611783' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:15.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:15 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1474127142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:21:15.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:15 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2381611783' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:15.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:15 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1474127142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:21:15.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:15 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2381611783' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:16.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:16 smithi098 ceph-mon[36087]: pgmap v292: 129 pgs: 2 creating+peering, 3 unknown, 124 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 688 B/s wr, 6 op/s 2024-03-28T05:21:16.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:16 smithi098 ceph-mon[36087]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T05:21:16.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:16 smithi098 ceph-mon[36087]: Cluster is now healthy 2024-03-28T05:21:16.631 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:16 smithi023 ceph-mon[31954]: pgmap v292: 129 pgs: 2 creating+peering, 3 unknown, 124 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 688 B/s wr, 6 op/s 2024-03-28T05:21:16.631 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:16 smithi023 ceph-mon[31954]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T05:21:16.631 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:16 smithi023 ceph-mon[31954]: Cluster is now healthy 2024-03-28T05:21:16.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:16 smithi113 ceph-mon[37164]: pgmap v292: 129 pgs: 2 creating+peering, 3 unknown, 124 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 688 B/s wr, 6 op/s 2024-03-28T05:21:16.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:16 smithi113 ceph-mon[37164]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T05:21:16.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:16 smithi113 ceph-mon[37164]: Cluster is now healthy 2024-03-28T05:21:16.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.977+0000 7f3720f82700 1 -- 172.21.15.23:0/2367022377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f371c107510 msgr2=0x7f371c10f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:16.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.977+0000 7f3720f82700 1 --2- 172.21.15.23:0/2367022377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f371c107510 0x7f371c10f090 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f37140099c0 tx=0x7f3714009cd0 comp rx=0 tx=0).stop 2024-03-28T05:21:16.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.977+0000 7f3720f82700 1 -- 172.21.15.23:0/2367022377 shutdown_connections 2024-03-28T05:21:16.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.977+0000 7f3720f82700 1 --2- 172.21.15.23:0/2367022377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f371c107510 0x7f371c10f090 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:16.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.977+0000 7f3720f82700 1 --2- 172.21.15.23:0/2367022377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f371c06a930 0x7f371c06ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:16.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.977+0000 7f3720f82700 1 --2- 172.21.15.23:0/2367022377 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f371c06b2e0 0x7f371c106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:16.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.977+0000 7f3720f82700 1 -- 172.21.15.23:0/2367022377 >> 172.21.15.23:0/2367022377 conn(0x7f371c0770e0 msgr2=0x7f371c0774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:16.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.977+0000 7f3720f82700 1 -- 172.21.15.23:0/2367022377 shutdown_connections 2024-03-28T05:21:16.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.977+0000 7f3720f82700 1 -- 172.21.15.23:0/2367022377 wait complete. 2024-03-28T05:21:16.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.978+0000 7f3720f82700 1 Processor -- start 2024-03-28T05:21:16.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.978+0000 7f3720f82700 1 -- start start 2024-03-28T05:21:16.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.978+0000 7f3720f82700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f371c06a930 0x7f371c19e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:16.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.978+0000 7f3720f82700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f371c06b2e0 0x7f371c19eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:16.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.978+0000 7f3720f82700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f371c107510 0x7f371c1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:16.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.978+0000 7f3720f82700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f371c1124b0 con 0x7f371c107510 2024-03-28T05:21:16.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.978+0000 7f3720f82700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f371c112330 con 0x7f371c06a930 2024-03-28T05:21:16.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.978+0000 7f3720f82700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f371c112630 con 0x7f371c06b2e0 2024-03-28T05:21:16.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.978+0000 7f371ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f371c107510 0x7f371c1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:16.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.978+0000 7f3719d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f371c06b2e0 0x7f371c19eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:16.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.978+0000 7f3719d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f371c06b2e0 0x7f371c19eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:32830/0 (socket says 172.21.15.23:32830) 2024-03-28T05:21:16.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.978+0000 7f371ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f371c107510 0x7f371c1987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57678/0 (socket says 172.21.15.23:57678) 2024-03-28T05:21:16.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.978+0000 7f3719d9b700 1 -- 172.21.15.23:0/2657433491 learned_addr learned my addr 172.21.15.23:0/2657433491 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:16.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.978+0000 7f371a59c700 1 --2- 172.21.15.23:0/2657433491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f371c06a930 0x7f371c19e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:16.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.979+0000 7f3719d9b700 1 -- 172.21.15.23:0/2657433491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f371c06a930 msgr2=0x7f371c19e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:16.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.979+0000 7f3719d9b700 1 --2- 172.21.15.23:0/2657433491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f371c06a930 0x7f371c19e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:16.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.979+0000 7f3719d9b700 1 -- 172.21.15.23:0/2657433491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f371c107510 msgr2=0x7f371c1987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:16.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.979+0000 7f3719d9b700 1 --2- 172.21.15.23:0/2657433491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f371c107510 0x7f371c1987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:16.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.979+0000 7f3719d9b700 1 -- 172.21.15.23:0/2657433491 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f371c198f90 con 0x7f371c06b2e0 2024-03-28T05:21:16.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.979+0000 7f3719d9b700 1 --2- 172.21.15.23:0/2657433491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f371c06b2e0 0x7f371c19eb70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f370400e940 tx=0x7f370400ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:16.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.979+0000 7f37137fe700 1 -- 172.21.15.23:0/2657433491 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370400c9e0 con 0x7f371c06b2e0 2024-03-28T05:21:16.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.979+0000 7f37137fe700 1 -- 172.21.15.23:0/2657433491 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f370400cb80 con 0x7f371c06b2e0 2024-03-28T05:21:16.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.979+0000 7f3720f82700 1 -- 172.21.15.23:0/2657433491 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f371c199280 con 0x7f371c06b2e0 2024-03-28T05:21:16.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.979+0000 7f37137fe700 1 -- 172.21.15.23:0/2657433491 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3704018660 con 0x7f371c06b2e0 2024-03-28T05:21:16.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.979+0000 7f3720f82700 1 -- 172.21.15.23:0/2657433491 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f371c1a5490 con 0x7f371c06b2e0 2024-03-28T05:21:16.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.980+0000 7f3720f82700 1 -- 172.21.15.23:0/2657433491 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f371c1038c0 con 0x7f371c06b2e0 2024-03-28T05:21:16.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.981+0000 7f37137fe700 1 -- 172.21.15.23:0/2657433491 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3704010430 con 0x7f371c06b2e0 2024-03-28T05:21:16.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.982+0000 7f37137fe700 1 --2- 172.21.15.23:0/2657433491 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f37080a2890 0x7f37080a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:16.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.982+0000 7f371a59c700 1 --2- 172.21.15.23:0/2657433491 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f37080a2890 0x7f37080a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:16.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.982+0000 7f37137fe700 1 -- 172.21.15.23:0/2657433491 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f37040ced90 con 0x7f371c06b2e0 2024-03-28T05:21:16.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.982+0000 7f371a59c700 1 --2- 172.21.15.23:0/2657433491 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f37080a2890 0x7f37080a4d50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f370c009550 tx=0x7f370c0091b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:16.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:16.986+0000 7f37137fe700 1 -- 172.21.15.23:0/2657433491 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f370409bbf0 con 0x7f371c06b2e0 2024-03-28T05:21:17.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:17.239+0000 7f3720f82700 1 -- 172.21.15.23:0/2657433491 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f371c02da30 con 0x7f371c06b2e0 2024-03-28T05:21:17.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:17.239+0000 7f37137fe700 1 -- 172.21.15.23:0/2657433491 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+88 (secure 0 0 0) 0x7f370409b4f0 con 0x7f371c06b2e0 2024-03-28T05:21:17.237 INFO:teuthology.orchestra.run.smithi023.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-28T05:21:17.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:17.242+0000 7f3720f82700 1 -- 172.21.15.23:0/2657433491 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f37080a2890 msgr2=0x7f37080a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:17.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:17.242+0000 7f3720f82700 1 --2- 172.21.15.23:0/2657433491 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f37080a2890 0x7f37080a4d50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f370c009550 tx=0x7f370c0091b0 comp rx=0 tx=0).stop 2024-03-28T05:21:17.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:17.242+0000 7f3720f82700 1 -- 172.21.15.23:0/2657433491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f371c06b2e0 msgr2=0x7f371c19eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:17.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:17.242+0000 7f3720f82700 1 --2- 172.21.15.23:0/2657433491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f371c06b2e0 0x7f371c19eb70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f370400e940 tx=0x7f370400ec50 comp rx=0 tx=0).stop 2024-03-28T05:21:17.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:17.242+0000 7f3720f82700 1 -- 172.21.15.23:0/2657433491 shutdown_connections 2024-03-28T05:21:17.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:17.242+0000 7f3720f82700 1 --2- 172.21.15.23:0/2657433491 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f37080a2890 0x7f37080a4d50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:17.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:17.242+0000 7f3720f82700 1 --2- 172.21.15.23:0/2657433491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f371c107510 0x7f371c1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:17.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:17.242+0000 7f3720f82700 1 --2- 172.21.15.23:0/2657433491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f371c06a930 0x7f371c19e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:17.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:17.242+0000 7f3720f82700 1 --2- 172.21.15.23:0/2657433491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f371c06b2e0 0x7f371c19eb70 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:17.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:17.242+0000 7f3720f82700 1 -- 172.21.15.23:0/2657433491 >> 172.21.15.23:0/2657433491 conn(0x7f371c0770e0 msgr2=0x7f371c0ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:17.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:17.242+0000 7f3720f82700 1 -- 172.21.15.23:0/2657433491 shutdown_connections 2024-03-28T05:21:17.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:17.242+0000 7f3720f82700 1 -- 172.21.15.23:0/2657433491 wait complete. 2024-03-28T05:21:17.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:17 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2657433491' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T05:21:17.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:17 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2657433491' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T05:21:17.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:17 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2657433491' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T05:21:17.923 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-28T05:21:17.923 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T05:21:17.923 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi023.front.sepia.ceph.com 2024-03-28T05:21:17.923 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-28T05:21:17.923 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-28T05:21:17.923 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-28T05:21:17.923 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T05:21:17.923 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T05:21:17.923 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-28T05:21:17.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:21:17.924 DEBUG:teuthology.orchestra.run.smithi023:> ip addr 2024-03-28T05:21:17.940 INFO:teuthology.orchestra.run.smithi023.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T05:21:17.941 INFO:teuthology.orchestra.run.smithi023.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T05:21:17.941 INFO:teuthology.orchestra.run.smithi023.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T05:21:17.941 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-28T05:21:17.941 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 ::1/128 scope host 2024-03-28T05:21:17.941 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-28T05:21:17.941 INFO:teuthology.orchestra.run.smithi023.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T05:21:17.941 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:ae brd ff:ff:ff:ff:ff:ff 2024-03-28T05:21:17.941 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f0 2024-03-28T05:21:17.941 INFO:teuthology.orchestra.run.smithi023.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T05:21:17.941 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:af brd ff:ff:ff:ff:ff:ff 2024-03-28T05:21:17.942 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f1 2024-03-28T05:21:17.942 INFO:teuthology.orchestra.run.smithi023.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T05:21:17.942 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:42 brd ff:ff:ff:ff:ff:ff 2024-03-28T05:21:17.942 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f0 2024-03-28T05:21:17.942 INFO:teuthology.orchestra.run.smithi023.stdout: inet 172.21.15.23/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-28T05:21:17.942 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft 42229sec preferred_lft 42229sec 2024-03-28T05:21:17.942 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 fe80::ec4:7aff:fe59:ce42/64 scope link noprefixroute 2024-03-28T05:21:17.942 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-28T05:21:17.942 INFO:teuthology.orchestra.run.smithi023.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T05:21:17.942 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:43 brd ff:ff:ff:ff:ff:ff 2024-03-28T05:21:17.942 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f1 2024-03-28T05:21:17.943 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T05:21:17.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:21:17.944 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-28T05:21:17.944 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-28T05:21:17.944 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add name ceph-brx type bridge 2024-03-28T05:21:17.944 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr flush dev ceph-brx 2024-03-28T05:21:17.944 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set ceph-brx up 2024-03-28T05:21:17.944 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T05:21:17.944 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-28T05:21:18.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:21:18.096 DEBUG:teuthology.orchestra.run.smithi023:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T05:21:18.124 INFO:teuthology.orchestra.run.smithi023.stdout:1 2024-03-28T05:21:18.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:21:18.126 DEBUG:teuthology.orchestra.run.smithi023:> route 2024-03-28T05:21:18.198 INFO:teuthology.orchestra.run.smithi023.stdout:Kernel IP routing table 2024-03-28T05:21:18.199 INFO:teuthology.orchestra.run.smithi023.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T05:21:18.199 INFO:teuthology.orchestra.run.smithi023.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-28T05:21:18.199 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-28T05:21:18.199 INFO:teuthology.orchestra.run.smithi023.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T05:21:18.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:21:18.199 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-28T05:21:18.200 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-28T05:21:18.200 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-28T05:21:18.200 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-28T05:21:18.200 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-28T05:21:18.200 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-28T05:21:18.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:21:18.455 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2024-03-28T05:21:18.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:21:18.472 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list-id 2024-03-28T05:21:18.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:21:18.527 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-28T05:21:18.527 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-28T05:21:18.527 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T05:21:18.527 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-28T05:21:18.527 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-28T05:21:18.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:18 smithi023 ceph-mon[31954]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 4 op/s 2024-03-28T05:21:18.573 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-28T05:21:18.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:21:18.573 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-28T05:21:18.573 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-28T05:21:18.573 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-28T05:21:18.573 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T05:21:18.573 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-28T05:21:18.573 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-28T05:21:18.573 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-28T05:21:18.574 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-28T05:21:18.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:18 smithi098 ceph-mon[36087]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 4 op/s 2024-03-28T05:21:18.643 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:18 smithi113 ceph-mon[37164]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 4 op/s 2024-03-28T05:21:18.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:21:18.709 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-28T05:21:18.709 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-28T05:21:18.709 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set brx.0 up 2024-03-28T05:21:18.709 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T05:21:18.709 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-28T05:21:18.751 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-28T05:21:18.751 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T05:21:18.751 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-28T05:21:18.810 INFO:teuthology.orchestra.run.smithi023.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-28T05:21:18.810 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T05:21:18.810 DEBUG:teuthology.orchestra.run.smithi023:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-28T05:21:18.865 DEBUG:teuthology.orchestra.run.smithi023:> sudo modprobe fuse 2024-03-28T05:21:18.932 DEBUG:teuthology.orchestra.run.smithi023:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T05:21:18.988 INFO:teuthology.orchestra.run.smithi023.stdout:/proc 2024-03-28T05:21:18.988 INFO:teuthology.orchestra.run.smithi023.stdout:/sys 2024-03-28T05:21:18.988 INFO:teuthology.orchestra.run.smithi023.stdout:/dev 2024-03-28T05:21:18.988 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/security 2024-03-28T05:21:18.989 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/shm 2024-03-28T05:21:18.989 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/pts 2024-03-28T05:21:18.989 INFO:teuthology.orchestra.run.smithi023.stdout:/run 2024-03-28T05:21:18.989 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/cgroup 2024-03-28T05:21:18.989 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/pstore 2024-03-28T05:21:18.989 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/bpf 2024-03-28T05:21:18.989 INFO:teuthology.orchestra.run.smithi023.stdout:/ 2024-03-28T05:21:18.989 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/selinux 2024-03-28T05:21:18.989 INFO:teuthology.orchestra.run.smithi023.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T05:21:18.989 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/hugepages 2024-03-28T05:21:18.989 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/mqueue 2024-03-28T05:21:18.989 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/debug 2024-03-28T05:21:18.989 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/tracing 2024-03-28T05:21:18.990 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/fuse/connections 2024-03-28T05:21:18.990 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/config 2024-03-28T05:21:18.990 INFO:teuthology.orchestra.run.smithi023.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T05:21:18.990 INFO:teuthology.orchestra.run.smithi023.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T05:21:18.990 INFO:teuthology.orchestra.run.smithi023.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T05:21:18.990 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T05:21:18.990 INFO:teuthology.orchestra.run.smithi023.stdout:/run/user/1000 2024-03-28T05:21:18.990 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/ceph 2024-03-28T05:21:18.990 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay 2024-03-28T05:21:18.990 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/703b66cf13816c182aa76c711f27f59b0b1d7896eb12c3496b6f929f761fc729/merged 2024-03-28T05:21:18.990 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/d1458088b34fc15ff7b558d7ba485ef0aa45a43dbe3b04ab66aae69ae7b25027/merged 2024-03-28T05:21:18.990 INFO:teuthology.orchestra.run.smithi023.stdout:/run/user/0 2024-03-28T05:21:18.991 INFO:teuthology.orchestra.run.smithi023.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T05:21:18.991 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/f900b8abeeec125189e4ae9634b2e725411f435a329d9c5e397401103e2bc217/merged 2024-03-28T05:21:18.991 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/faa92384e48bc443e99513e2466fdbe50092041e938cdd3773198c333f49dc41/merged 2024-03-28T05:21:18.991 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/cc5082e6a7aea75202bb10b25231efca21c3f316bc17ed6e6e580684301a9288/merged 2024-03-28T05:21:18.991 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/a8434bf6eda19ade43c795761227b938f49d90516fef4a63fd4277e90e786bb3/merged 2024-03-28T05:21:18.991 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/36986dc4df68b9b133cbe658b561033ea06595c6c3d57aa95d347c589fb8ae87/merged 2024-03-28T05:21:18.991 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/e8c551c055a3498b2445c66e8d75500662bbdcb917f21fe5c2e6ff9a30c86af9/merged 2024-03-28T05:21:18.991 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/f24cc8a0bbbe392f2391658423ca37872dd35f5c1875ea0eafc3daea596ceba8/merged 2024-03-28T05:21:18.991 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/0e3dba8dfa0a21fffd26c4c9af2bc240321379e4c14c183c0822928e34ef2b53/merged 2024-03-28T05:21:18.991 INFO:teuthology.orchestra.run.smithi023.stdout:/run/netns 2024-03-28T05:21:18.991 INFO:teuthology.orchestra.run.smithi023.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T05:21:18.991 INFO:teuthology.orchestra.run.smithi023.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T05:21:18.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:21:18.992 DEBUG:teuthology.orchestra.run.smithi023:> ls /sys/fs/fuse/connections 2024-03-28T05:21:19.043 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-28T05:21:19.043 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-28T05:21:19.086 DEBUG:teuthology.orchestra.run.smithi023:> sudo modprobe fuse 2024-03-28T05:21:19.113 DEBUG:teuthology.orchestra.run.smithi023:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T05:21:19.169 INFO:teuthology.orchestra.run.smithi023.stdout:/proc 2024-03-28T05:21:19.169 INFO:teuthology.orchestra.run.smithi023.stdout:/sys 2024-03-28T05:21:19.169 INFO:teuthology.orchestra.run.smithi023.stdout:/dev 2024-03-28T05:21:19.169 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/security 2024-03-28T05:21:19.169 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/shm 2024-03-28T05:21:19.170 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/pts 2024-03-28T05:21:19.170 INFO:teuthology.orchestra.run.smithi023.stdout:/run 2024-03-28T05:21:19.170 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/cgroup 2024-03-28T05:21:19.170 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/pstore 2024-03-28T05:21:19.170 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/bpf 2024-03-28T05:21:19.170 INFO:teuthology.orchestra.run.smithi023.stdout:/ 2024-03-28T05:21:19.170 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/selinux 2024-03-28T05:21:19.170 INFO:teuthology.orchestra.run.smithi023.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T05:21:19.170 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/hugepages 2024-03-28T05:21:19.170 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/mqueue 2024-03-28T05:21:19.170 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/debug 2024-03-28T05:21:19.170 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/tracing 2024-03-28T05:21:19.170 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/fuse/connections 2024-03-28T05:21:19.170 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/config 2024-03-28T05:21:19.171 INFO:teuthology.orchestra.run.smithi023.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T05:21:19.171 INFO:teuthology.orchestra.run.smithi023.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T05:21:19.171 INFO:teuthology.orchestra.run.smithi023.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T05:21:19.171 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T05:21:19.171 INFO:teuthology.orchestra.run.smithi023.stdout:/run/user/1000 2024-03-28T05:21:19.171 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/ceph 2024-03-28T05:21:19.171 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay 2024-03-28T05:21:19.171 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/703b66cf13816c182aa76c711f27f59b0b1d7896eb12c3496b6f929f761fc729/merged 2024-03-28T05:21:19.171 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/d1458088b34fc15ff7b558d7ba485ef0aa45a43dbe3b04ab66aae69ae7b25027/merged 2024-03-28T05:21:19.171 INFO:teuthology.orchestra.run.smithi023.stdout:/run/user/0 2024-03-28T05:21:19.171 INFO:teuthology.orchestra.run.smithi023.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T05:21:19.171 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/f900b8abeeec125189e4ae9634b2e725411f435a329d9c5e397401103e2bc217/merged 2024-03-28T05:21:19.171 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/faa92384e48bc443e99513e2466fdbe50092041e938cdd3773198c333f49dc41/merged 2024-03-28T05:21:19.171 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/cc5082e6a7aea75202bb10b25231efca21c3f316bc17ed6e6e580684301a9288/merged 2024-03-28T05:21:19.171 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/a8434bf6eda19ade43c795761227b938f49d90516fef4a63fd4277e90e786bb3/merged 2024-03-28T05:21:19.172 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/36986dc4df68b9b133cbe658b561033ea06595c6c3d57aa95d347c589fb8ae87/merged 2024-03-28T05:21:19.172 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/e8c551c055a3498b2445c66e8d75500662bbdcb917f21fe5c2e6ff9a30c86af9/merged 2024-03-28T05:21:19.172 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/f24cc8a0bbbe392f2391658423ca37872dd35f5c1875ea0eafc3daea596ceba8/merged 2024-03-28T05:21:19.172 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/0e3dba8dfa0a21fffd26c4c9af2bc240321379e4c14c183c0822928e34ef2b53/merged 2024-03-28T05:21:19.172 INFO:teuthology.orchestra.run.smithi023.stdout:/run/netns 2024-03-28T05:21:19.172 INFO:teuthology.orchestra.run.smithi023.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T05:21:19.172 INFO:teuthology.orchestra.run.smithi023.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T05:21:19.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:21:19.172 DEBUG:teuthology.orchestra.run.smithi023:> ls /sys/fs/fuse/connections 2024-03-28T05:21:19.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:2024-03-28T05:21:19.184+0000 7fa327443480 -1 init, newargv = 0x55c8df19ce20 newargc=15 2024-03-28T05:21:19.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:ceph-fuse[75997]: starting ceph client 2024-03-28T05:21:19.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:ceph-fuse[75997]: starting fuse 2024-03-28T05:21:19.197 INFO:teuthology.orchestra.run.smithi023.stdout:119 2024-03-28T05:21:19.197 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-28T05:21:19.197 DEBUG:teuthology.orchestra.run.smithi023:> sudo stdin-killer -- python3 -c ' 2024-03-28T05:21:19.197 DEBUG:teuthology.orchestra.run.smithi023:> import glob 2024-03-28T05:21:19.197 DEBUG:teuthology.orchestra.run.smithi023:> import re 2024-03-28T05:21:19.197 DEBUG:teuthology.orchestra.run.smithi023:> import os 2024-03-28T05:21:19.197 DEBUG:teuthology.orchestra.run.smithi023:> import subprocess 2024-03-28T05:21:19.198 DEBUG:teuthology.orchestra.run.smithi023:> 2024-03-28T05:21:19.198 DEBUG:teuthology.orchestra.run.smithi023:> def _find_admin_socket(client_name): 2024-03-28T05:21:19.198 DEBUG:teuthology.orchestra.run.smithi023:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-28T05:21:19.198 DEBUG:teuthology.orchestra.run.smithi023:> files = glob.glob(asok_path) 2024-03-28T05:21:19.198 DEBUG:teuthology.orchestra.run.smithi023:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-28T05:21:19.198 DEBUG:teuthology.orchestra.run.smithi023:> 2024-03-28T05:21:19.198 DEBUG:teuthology.orchestra.run.smithi023:> # Given a non-glob path, it better be there 2024-03-28T05:21:19.198 DEBUG:teuthology.orchestra.run.smithi023:> if "*" not in asok_path: 2024-03-28T05:21:19.198 DEBUG:teuthology.orchestra.run.smithi023:> assert(len(files) == 1) 2024-03-28T05:21:19.198 DEBUG:teuthology.orchestra.run.smithi023:> return files[0] 2024-03-28T05:21:19.198 DEBUG:teuthology.orchestra.run.smithi023:> 2024-03-28T05:21:19.198 DEBUG:teuthology.orchestra.run.smithi023:> for f in files: 2024-03-28T05:21:19.198 DEBUG:teuthology.orchestra.run.smithi023:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-28T05:21:19.199 DEBUG:teuthology.orchestra.run.smithi023:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-28T05:21:19.199 DEBUG:teuthology.orchestra.run.smithi023:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-28T05:21:19.199 DEBUG:teuthology.orchestra.run.smithi023:> contents = proc_f.read() 2024-03-28T05:21:19.199 DEBUG:teuthology.orchestra.run.smithi023:> if mountpoint in contents: 2024-03-28T05:21:19.199 DEBUG:teuthology.orchestra.run.smithi023:> return f 2024-03-28T05:21:19.199 DEBUG:teuthology.orchestra.run.smithi023:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-28T05:21:19.199 DEBUG:teuthology.orchestra.run.smithi023:> 2024-03-28T05:21:19.199 DEBUG:teuthology.orchestra.run.smithi023:> print(_find_admin_socket("client.admin")) 2024-03-28T05:21:19.199 DEBUG:teuthology.orchestra.run.smithi023:> ' 2024-03-28T05:21:19.318 INFO:teuthology.orchestra.run.smithi023.stdout:/var/run/ceph/ceph-client.admin.75997.asok 2024-03-28T05:21:19.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T05:21:19.328 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.75997.asok 2024-03-28T05:21:19.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:21:19.328 DEBUG:teuthology.orchestra.run.smithi023:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.75997.asok status 2024-03-28T05:21:19.466 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T05:21:19.466 INFO:teuthology.orchestra.run.smithi023.stdout: "metadata": { 2024-03-28T05:21:19.466 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-28T05:21:19.466 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T05:21:19.466 INFO:teuthology.orchestra.run.smithi023.stdout: "entity_id": "admin", 2024-03-28T05:21:19.466 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": "smithi023", 2024-03-28T05:21:19.466 INFO:teuthology.orchestra.run.smithi023.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-28T05:21:19.466 INFO:teuthology.orchestra.run.smithi023.stdout: "pid": "75997", 2024-03-28T05:21:19.466 INFO:teuthology.orchestra.run.smithi023.stdout: "root": "/" 2024-03-28T05:21:19.466 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:21:19.466 INFO:teuthology.orchestra.run.smithi023.stdout: "dentry_count": 0, 2024-03-28T05:21:19.467 INFO:teuthology.orchestra.run.smithi023.stdout: "dentry_pinned_count": 0, 2024-03-28T05:21:19.467 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 14784, 2024-03-28T05:21:19.467 INFO:teuthology.orchestra.run.smithi023.stdout: "inst": { 2024-03-28T05:21:19.467 INFO:teuthology.orchestra.run.smithi023.stdout: "name": { 2024-03-28T05:21:19.467 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "client", 2024-03-28T05:21:19.467 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 14784 2024-03-28T05:21:19.467 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:21:19.467 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": { 2024-03-28T05:21:19.467 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-28T05:21:19.467 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "192.168.0.1:0", 2024-03-28T05:21:19.467 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 403337624 2024-03-28T05:21:19.467 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:21:19.467 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:21:19.467 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": { 2024-03-28T05:21:19.467 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-28T05:21:19.467 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "192.168.0.1:0", 2024-03-28T05:21:19.468 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 403337624 2024-03-28T05:21:19.468 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:21:19.468 INFO:teuthology.orchestra.run.smithi023.stdout: "inst_str": "client.14784 192.168.0.1:0/403337624", 2024-03-28T05:21:19.468 INFO:teuthology.orchestra.run.smithi023.stdout: "addr_str": "192.168.0.1:0/403337624", 2024-03-28T05:21:19.468 INFO:teuthology.orchestra.run.smithi023.stdout: "inode_count": 1, 2024-03-28T05:21:19.468 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_epoch": 16, 2024-03-28T05:21:19.468 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_epoch": 89, 2024-03-28T05:21:19.468 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_epoch_barrier": 0, 2024-03-28T05:21:19.468 INFO:teuthology.orchestra.run.smithi023.stdout: "blocklisted": false, 2024-03-28T05:21:19.468 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs" 2024-03-28T05:21:19.468 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T05:21:19.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:21:19.475 DEBUG:teuthology.orchestra.run.smithi023:> stat --file-system '--printf=%T 2024-03-28T05:21:19.475 DEBUG:teuthology.orchestra.run.smithi023:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-28T05:21:19.490 INFO:teuthology.orchestra.run.smithi023.stdout:fuseblk 2024-03-28T05:21:19.490 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-28T05:21:19.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:21:19.490 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-28T05:21:19.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:21:19.557 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-28T05:21:19.636 INFO:teuthology.orchestra.run.smithi023.stdout:# file: . 2024-03-28T05:21:19.636 INFO:teuthology.orchestra.run.smithi023.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-28T05:21:19.636 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:21:19.636 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T05:21:19.636 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T05:21:19.692 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T05:21:19.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.837+0000 7ff2aa577640 1 -- 172.21.15.23:0/4194378028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2a4102a20 msgr2=0x7ff2a4102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:19.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.837+0000 7ff2aa577640 1 --2- 172.21.15.23:0/4194378028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2a4102a20 0x7ff2a4102ea0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff294009930 tx=0x7ff29402bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:19.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.837+0000 7ff2aa577640 1 -- 172.21.15.23:0/4194378028 shutdown_connections 2024-03-28T05:21:19.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.837+0000 7ff2aa577640 1 --2- 172.21.15.23:0/4194378028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a41033e0 0x7ff2a4109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:19.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.837+0000 7ff2aa577640 1 --2- 172.21.15.23:0/4194378028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2a4102a20 0x7ff2a4102ea0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:19.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.837+0000 7ff2aa577640 1 --2- 172.21.15.23:0/4194378028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2a40ff9b0 0x7ff2a40ffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:19.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.837+0000 7ff2aa577640 1 -- 172.21.15.23:0/4194378028 >> 172.21.15.23:0/4194378028 conn(0x7ff2a40fb2c0 msgr2=0x7ff2a40fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:19.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.837+0000 7ff2aa577640 1 -- 172.21.15.23:0/4194378028 shutdown_connections 2024-03-28T05:21:19.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.837+0000 7ff2aa577640 1 -- 172.21.15.23:0/4194378028 wait complete. 2024-03-28T05:21:19.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.837+0000 7ff2aa577640 1 Processor -- start 2024-03-28T05:21:19.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.837+0000 7ff2aa577640 1 -- start start 2024-03-28T05:21:19.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.838+0000 7ff2aa577640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2a40ff9b0 0x7ff2a4197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:19.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.838+0000 7ff2aa577640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2a4102a20 0x7ff2a4197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:19.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.838+0000 7ff2aa577640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a41033e0 0x7ff2a419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:19.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.838+0000 7ff2aa577640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a410d360 con 0x7ff2a40ff9b0 2024-03-28T05:21:19.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.838+0000 7ff2aa577640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a410d1e0 con 0x7ff2a41033e0 2024-03-28T05:21:19.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.838+0000 7ff2aa577640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a410d4e0 con 0x7ff2a4102a20 2024-03-28T05:21:19.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.838+0000 7ff2a3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2a40ff9b0 0x7ff2a4197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:19.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.838+0000 7ff2a3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2a40ff9b0 0x7ff2a4197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57696/0 (socket says 172.21.15.23:57696) 2024-03-28T05:21:19.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.838+0000 7ff2a3fff640 1 -- 172.21.15.23:0/4220816502 learned_addr learned my addr 172.21.15.23:0/4220816502 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:19.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.838+0000 7ff2a37fe640 1 --2- 172.21.15.23:0/4220816502 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2a4102a20 0x7ff2a4197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:19.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.838+0000 7ff2a8b14640 1 --2- 172.21.15.23:0/4220816502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a41033e0 0x7ff2a419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:19.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.838+0000 7ff2a3fff640 1 -- 172.21.15.23:0/4220816502 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2a4102a20 msgr2=0x7ff2a4197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:19.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.838+0000 7ff2a3fff640 1 --2- 172.21.15.23:0/4220816502 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2a4102a20 0x7ff2a4197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:19.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.838+0000 7ff2a3fff640 1 -- 172.21.15.23:0/4220816502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a41033e0 msgr2=0x7ff2a419f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:19.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.838+0000 7ff2a3fff640 1 --2- 172.21.15.23:0/4220816502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a41033e0 0x7ff2a419f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:19.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.838+0000 7ff2a3fff640 1 -- 172.21.15.23:0/4220816502 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a419f680 con 0x7ff2a40ff9b0 2024-03-28T05:21:19.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.838+0000 7ff2a3fff640 1 --2- 172.21.15.23:0/4220816502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2a40ff9b0 0x7ff2a4197a60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff28c00e9c0 tx=0x7ff28c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:19.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.839+0000 7ff2a17fa640 1 -- 172.21.15.23:0/4220816502 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28c00cd80 con 0x7ff2a40ff9b0 2024-03-28T05:21:19.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.839+0000 7ff2a17fa640 1 -- 172.21.15.23:0/4220816502 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff28c0035f0 con 0x7ff2a40ff9b0 2024-03-28T05:21:19.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.839+0000 7ff2aa577640 1 -- 172.21.15.23:0/4220816502 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a419f970 con 0x7ff2a40ff9b0 2024-03-28T05:21:19.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.839+0000 7ff2a17fa640 1 -- 172.21.15.23:0/4220816502 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28c010690 con 0x7ff2a40ff9b0 2024-03-28T05:21:19.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.839+0000 7ff2aa577640 1 -- 172.21.15.23:0/4220816502 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a419fe90 con 0x7ff2a40ff9b0 2024-03-28T05:21:19.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.840+0000 7ff2aa577640 1 -- 172.21.15.23:0/4220816502 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a4108c10 con 0x7ff2a40ff9b0 2024-03-28T05:21:19.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.841+0000 7ff2a17fa640 1 -- 172.21.15.23:0/4220816502 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff28c0037a0 con 0x7ff2a40ff9b0 2024-03-28T05:21:19.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.841+0000 7ff2a17fa640 1 --2- 172.21.15.23:0/4220816502 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff2900a2880 0x7ff2900a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:19.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.841+0000 7ff2a37fe640 1 --2- 172.21.15.23:0/4220816502 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff2900a2880 0x7ff2900a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:19.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.841+0000 7ff2a17fa640 1 -- 172.21.15.23:0/4220816502 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7ff28c0cef70 con 0x7ff2a40ff9b0 2024-03-28T05:21:19.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.842+0000 7ff2a37fe640 1 --2- 172.21.15.23:0/4220816502 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff2900a2880 0x7ff2900a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff294000c00 tx=0x7ff294005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:19.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:19.846+0000 7ff2a17fa640 1 -- 172.21.15.23:0/4220816502 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff28c09bdd0 con 0x7ff2a40ff9b0 2024-03-28T05:21:20.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.033+0000 7ff2aa577640 1 -- 172.21.15.23:0/4220816502 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2a40622a0 con 0x7ff2a40ff9b0 2024-03-28T05:21:20.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.034+0000 7ff2a17fa640 1 -- 172.21.15.23:0/4220816502 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24082 (secure 0 0 0) 0x7ff28c0860e0 con 0x7ff2a40ff9b0 2024-03-28T05:21:20.033 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:21:20.033 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":89,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","created":"2024-03-28T05:13:13.453653+0000","modified":"2024-03-28T05:21:12.641185+0000","last_up_change":"2024-03-28T05:20:20.221527+0000","last_in_change":"2024-03-28T05:20:00.365478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T05:16:32.986974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T05:21:06.103894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T05:21:06.827190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T05:21:10.507418+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"18150942-59a4-4c3b-83d4-c1fc3ccf1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6803","nonce":1219956881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6805","nonce":1219956881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6809","nonce":1219956881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6807","nonce":1219956881}]},"public_addr":"172.21.15.23:6803/1219956881","cluster_addr":"172.21.15.23:6805/1219956881","heartbeat_back_addr":"172.21.15.23:6809/1219956881","heartbeat_front_addr":"172.21.15.23:6807/1219956881","state":["exists","up"]},{"osd":1,"uuid":"ba6d1a95-5135-4670-95f9-7878c0e7bd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6801","nonce":555491234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6803","nonce":555491234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6807","nonce":555491234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6805","nonce":555491234}]},"public_addr":"172.21.15.98:6801/555491234","cluster_addr":"172.21.15.98:6803/555491234","heartbeat_back_addr":"172.21.15.98:6807/555491234","heartbeat_front_addr":"172.21.15.98:6805/555491234","state":["exists","up"]},{"osd":2,"uuid":"873bdd60-2a47-403b-b1f4-ee9226a95067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6801","nonce":1535856914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6803","nonce":1535856914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6807","nonce":1535856914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6805","nonce":1535856914}]},"public_addr":"172.21.15.113:6801/1535856914","cluster_addr":"172.21.15.113:6803/1535856914","heartbeat_back_addr":"172.21.15.113:6807/1535856914","heartbeat_front_addr":"172.21.15.113:6805/1535856914","state":["exists","up"]},{"osd":3,"uuid":"83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6811","nonce":2949415020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6813","nonce":2949415020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6817","nonce":2949415020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6815","nonce":2949415020}]},"public_addr":"172.21.15.23:6811/2949415020","cluster_addr":"172.21.15.23:6813/2949415020","heartbeat_back_addr":"172.21.15.23:6817/2949415020","heartbeat_front_addr":"172.21.15.23:6815/2949415020","state":["exists","up"]},{"osd":4,"uuid":"76a90bac-b221-40a3-8238-40e8049cec36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6809","nonce":2615902819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6811","nonce":2615902819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6815","nonce":2615902819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6813","nonce":2615902819}]},"public_addr":"172.21.15.98:6809/2615902819","cluster_addr":"172.21.15.98:6811/2615902819","heartbeat_back_addr":"172.21.15.98:6815/2615902819","heartbeat_front_addr":"172.21.15.98:6813/2615902819","state":["exists","up"]},{"osd":5,"uuid":"b53fdb55-a8df-40a8-bd1b-8bcef634e26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6809","nonce":625271528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6811","nonce":625271528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6815","nonce":625271528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6813","nonce":625271528}]},"public_addr":"172.21.15.113:6809/625271528","cluster_addr":"172.21.15.113:6811/625271528","heartbeat_back_addr":"172.21.15.113:6815/625271528","heartbeat_front_addr":"172.21.15.113:6813/625271528","state":["exists","up"]},{"osd":6,"uuid":"f00283d2-8d67-4e97-a575-9a9d3b55d5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6819","nonce":4188464176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6821","nonce":4188464176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6825","nonce":4188464176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6823","nonce":4188464176}]},"public_addr":"172.21.15.23:6819/4188464176","cluster_addr":"172.21.15.23:6821/4188464176","heartbeat_back_addr":"172.21.15.23:6825/4188464176","heartbeat_front_addr":"172.21.15.23:6823/4188464176","state":["exists","up"]},{"osd":7,"uuid":"edc008a7-b07b-40a8-88b7-1614ba38a775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6817","nonce":1985546028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6819","nonce":1985546028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6823","nonce":1985546028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6821","nonce":1985546028}]},"public_addr":"172.21.15.98:6817/1985546028","cluster_addr":"172.21.15.98:6819/1985546028","heartbeat_back_addr":"172.21.15.98:6823/1985546028","heartbeat_front_addr":"172.21.15.98:6821/1985546028","state":["exists","up"]},{"osd":8,"uuid":"56e7b2d1-81f0-47b2-8638-f29832bed8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6817","nonce":3534876081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6819","nonce":3534876081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6823","nonce":3534876081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6821","nonce":3534876081}]},"public_addr":"172.21.15.113:6817/3534876081","cluster_addr":"172.21.15.113:6819/3534876081","heartbeat_back_addr":"172.21.15.113:6823/3534876081","heartbeat_front_addr":"172.21.15.113:6821/3534876081","state":["exists","up"]},{"osd":9,"uuid":"a2e90caa-6e9c-42bf-9f18-81f7f47880cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6827","nonce":672264207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6829","nonce":672264207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6833","nonce":672264207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6831","nonce":672264207}]},"public_addr":"172.21.15.23:6827/672264207","cluster_addr":"172.21.15.23:6829/672264207","heartbeat_back_addr":"172.21.15.23:6833/672264207","heartbeat_front_addr":"172.21.15.23:6831/672264207","state":["exists","up"]},{"osd":10,"uuid":"0eda917f-020c-437f-95ae-b26203f6c68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6825","nonce":2699920595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6827","nonce":2699920595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6831","nonce":2699920595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6829","nonce":2699920595}]},"public_addr":"172.21.15.98:6825/2699920595","cluster_addr":"172.21.15.98:6827/2699920595","heartbeat_back_addr":"172.21.15.98:6831/2699920595","heartbeat_front_addr":"172.21.15.98:6829/2699920595","state":["exists","up"]},{"osd":11,"uuid":"aae78501-78a8-469a-be20-3006f5ec5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6825","nonce":2113644020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6827","nonce":2113644020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6831","nonce":2113644020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6829","nonce":2113644020}]},"public_addr":"172.21.15.113:6825/2113644020","cluster_addr":"172.21.15.113:6827/2113644020","heartbeat_back_addr":"172.21.15.113:6831/2113644020","heartbeat_front_addr":"172.21.15.113:6829/2113644020","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:15:43.527475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:07.006538+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:30.594176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:55.810999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:20.595811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:45.474138+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:10.500409+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:35.787638+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:01.001188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:26.761115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:52.952752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:20:18.559832+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:6800/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1876451381":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2253559862":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2653562762":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/3337463830":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6800/1735221490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/2784771635":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2389902490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6801/1735221490":"2024-03-29T05:13:42.579607+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T05:21:20.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.036+0000 7ff2aa577640 1 -- 172.21.15.23:0/4220816502 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff2900a2880 msgr2=0x7ff2900a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:20.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.036+0000 7ff2aa577640 1 --2- 172.21.15.23:0/4220816502 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff2900a2880 0x7ff2900a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff294000c00 tx=0x7ff294005fd0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.036+0000 7ff2aa577640 1 -- 172.21.15.23:0/4220816502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2a40ff9b0 msgr2=0x7ff2a4197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:20.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.036+0000 7ff2aa577640 1 --2- 172.21.15.23:0/4220816502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2a40ff9b0 0x7ff2a4197a60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff28c00e9c0 tx=0x7ff28c00ee80 comp rx=0 tx=0).stop 2024-03-28T05:21:20.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.036+0000 7ff2aa577640 1 -- 172.21.15.23:0/4220816502 shutdown_connections 2024-03-28T05:21:20.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.036+0000 7ff2aa577640 1 --2- 172.21.15.23:0/4220816502 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff2900a2880 0x7ff2900a4d40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.036+0000 7ff2aa577640 1 --2- 172.21.15.23:0/4220816502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2a41033e0 0x7ff2a419f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.036+0000 7ff2aa577640 1 --2- 172.21.15.23:0/4220816502 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2a4102a20 0x7ff2a4197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.036+0000 7ff2aa577640 1 --2- 172.21.15.23:0/4220816502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2a40ff9b0 0x7ff2a4197a60 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.037+0000 7ff2aa577640 1 -- 172.21.15.23:0/4220816502 >> 172.21.15.23:0/4220816502 conn(0x7ff2a40fb2c0 msgr2=0x7ff2a4105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:20.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.037+0000 7ff2aa577640 1 -- 172.21.15.23:0/4220816502 shutdown_connections 2024-03-28T05:21:20.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.037+0000 7ff2aa577640 1 -- 172.21.15.23:0/4220816502 wait complete. 2024-03-28T05:21:20.047 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T05:16:32.986974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T05:21:06.103894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T05:21:06.827190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T05:21:10.507418+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T05:21:20.048 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T05:21:20.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.155+0000 7fb21255b640 1 -- 172.21.15.23:0/979366702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb20c1005a0 msgr2=0x7fb20c1009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:20.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.155+0000 7fb21255b640 1 --2- 172.21.15.23:0/979366702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb20c1005a0 0x7fb20c1009c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc009900 tx=0x7fb1fc02bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:20.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.156+0000 7fb21255b640 1 -- 172.21.15.23:0/979366702 shutdown_connections 2024-03-28T05:21:20.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.156+0000 7fb21255b640 1 --2- 172.21.15.23:0/979366702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb20c102150 0x7fb20c1089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.156+0000 7fb21255b640 1 --2- 172.21.15.23:0/979366702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb20c101790 0x7fb20c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.156+0000 7fb21255b640 1 --2- 172.21.15.23:0/979366702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb20c1005a0 0x7fb20c1009c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.156+0000 7fb21255b640 1 -- 172.21.15.23:0/979366702 >> 172.21.15.23:0/979366702 conn(0x7fb20c0fbb30 msgr2=0x7fb20c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:20.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.156+0000 7fb21255b640 1 -- 172.21.15.23:0/979366702 shutdown_connections 2024-03-28T05:21:20.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.156+0000 7fb21255b640 1 -- 172.21.15.23:0/979366702 wait complete. 2024-03-28T05:21:20.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.156+0000 7fb21255b640 1 Processor -- start 2024-03-28T05:21:20.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.156+0000 7fb21255b640 1 -- start start 2024-03-28T05:21:20.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.156+0000 7fb21255b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb20c1005a0 0x7fb20c197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:20.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.156+0000 7fb21255b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb20c101790 0x7fb20c197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:20.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.156+0000 7fb21255b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb20c102150 0x7fb20c19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:20.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.156+0000 7fb21255b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c10c260 con 0x7fb20c102150 2024-03-28T05:21:20.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.156+0000 7fb21255b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c10c0e0 con 0x7fb20c1005a0 2024-03-28T05:21:20.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.156+0000 7fb21255b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c10c3e0 con 0x7fb20c101790 2024-03-28T05:21:20.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.157+0000 7fb210af8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb20c102150 0x7fb20c19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:20.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.157+0000 7fb210af8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb20c102150 0x7fb20c19f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57708/0 (socket says 172.21.15.23:57708) 2024-03-28T05:21:20.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.157+0000 7fb210af8640 1 -- 172.21.15.23:0/1933994673 learned_addr learned my addr 172.21.15.23:0/1933994673 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:20.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.157+0000 7fb20b7fe640 1 --2- 172.21.15.23:0/1933994673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb20c101790 0x7fb20c197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:20.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.157+0000 7fb20bfff640 1 --2- 172.21.15.23:0/1933994673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb20c1005a0 0x7fb20c197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:20.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.157+0000 7fb210af8640 1 -- 172.21.15.23:0/1933994673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb20c101790 msgr2=0x7fb20c197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:20.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.157+0000 7fb210af8640 1 --2- 172.21.15.23:0/1933994673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb20c101790 0x7fb20c197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.157+0000 7fb210af8640 1 -- 172.21.15.23:0/1933994673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb20c1005a0 msgr2=0x7fb20c197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:20.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.157+0000 7fb210af8640 1 --2- 172.21.15.23:0/1933994673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb20c1005a0 0x7fb20c197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.157+0000 7fb210af8640 1 -- 172.21.15.23:0/1933994673 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20c19f680 con 0x7fb20c102150 2024-03-28T05:21:20.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.157+0000 7fb210af8640 1 --2- 172.21.15.23:0/1933994673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb20c102150 0x7fb20c19f010 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb204009fd0 tx=0x7fb20400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:20.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.157+0000 7fb2097fa640 1 -- 172.21.15.23:0/1933994673 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb204017070 con 0x7fb20c102150 2024-03-28T05:21:20.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.157+0000 7fb2097fa640 1 -- 172.21.15.23:0/1933994673 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb204003e50 con 0x7fb20c102150 2024-03-28T05:21:20.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.157+0000 7fb21255b640 1 -- 172.21.15.23:0/1933994673 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb20c19f970 con 0x7fb20c102150 2024-03-28T05:21:20.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.157+0000 7fb2097fa640 1 -- 172.21.15.23:0/1933994673 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2040106c0 con 0x7fb20c102150 2024-03-28T05:21:20.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.157+0000 7fb21255b640 1 -- 172.21.15.23:0/1933994673 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb20c19fe70 con 0x7fb20c102150 2024-03-28T05:21:20.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.158+0000 7fb21255b640 1 -- 172.21.15.23:0/1933994673 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20c1a0040 con 0x7fb20c102150 2024-03-28T05:21:20.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.159+0000 7fb2097fa640 1 -- 172.21.15.23:0/1933994673 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb204003970 con 0x7fb20c102150 2024-03-28T05:21:20.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.160+0000 7fb2097fa640 1 --2- 172.21.15.23:0/1933994673 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb1f80a2800 0x7fb1f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:20.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.160+0000 7fb20bfff640 1 --2- 172.21.15.23:0/1933994673 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb1f80a2800 0x7fb1f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:20.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.161+0000 7fb2097fa640 1 -- 172.21.15.23:0/1933994673 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb2040ce980 con 0x7fb20c102150 2024-03-28T05:21:20.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.161+0000 7fb20bfff640 1 --2- 172.21.15.23:0/1933994673 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb1f80a2800 0x7fb1f80a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc02c170 tx=0x7fb1fc02df90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:20.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.164+0000 7fb2097fa640 1 -- 172.21.15.23:0/1933994673 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb20409b190 con 0x7fb20c102150 2024-03-28T05:21:20.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.350+0000 7fb21255b640 1 -- 172.21.15.23:0/1933994673 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb20c1a0210 con 0x7fb20c102150 2024-03-28T05:21:20.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.350+0000 7fb2097fa640 1 -- 172.21.15.23:0/1933994673 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb20409f230 con 0x7fb20c102150 2024-03-28T05:21:20.348 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-28T05:21:20.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.352+0000 7fb21255b640 1 -- 172.21.15.23:0/1933994673 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb1f80a2800 msgr2=0x7fb1f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:20.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.352+0000 7fb21255b640 1 --2- 172.21.15.23:0/1933994673 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb1f80a2800 0x7fb1f80a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc02c170 tx=0x7fb1fc02df90 comp rx=0 tx=0).stop 2024-03-28T05:21:20.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.352+0000 7fb21255b640 1 -- 172.21.15.23:0/1933994673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb20c102150 msgr2=0x7fb20c19f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:20.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.352+0000 7fb21255b640 1 --2- 172.21.15.23:0/1933994673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb20c102150 0x7fb20c19f010 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb204009fd0 tx=0x7fb20400b260 comp rx=0 tx=0).stop 2024-03-28T05:21:20.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.352+0000 7fb21255b640 1 -- 172.21.15.23:0/1933994673 shutdown_connections 2024-03-28T05:21:20.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.352+0000 7fb21255b640 1 --2- 172.21.15.23:0/1933994673 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb1f80a2800 0x7fb1f80a4cc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.352+0000 7fb21255b640 1 --2- 172.21.15.23:0/1933994673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb20c102150 0x7fb20c19f010 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.352+0000 7fb21255b640 1 --2- 172.21.15.23:0/1933994673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb20c101790 0x7fb20c197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.352+0000 7fb21255b640 1 --2- 172.21.15.23:0/1933994673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb20c1005a0 0x7fb20c197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.352+0000 7fb21255b640 1 -- 172.21.15.23:0/1933994673 >> 172.21.15.23:0/1933994673 conn(0x7fb20c0fbb30 msgr2=0x7fb20c0fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:20.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.352+0000 7fb21255b640 1 -- 172.21.15.23:0/1933994673 shutdown_connections 2024-03-28T05:21:20.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.352+0000 7fb21255b640 1 -- 172.21.15.23:0/1933994673 wait complete. 2024-03-28T05:21:20.361 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T05:21:20.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.468+0000 7fb94fe59640 1 -- 172.21.15.23:0/3477311316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9481016b0 msgr2=0x7fb948101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:20.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.468+0000 7fb94fe59640 1 --2- 172.21.15.23:0/3477311316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9481016b0 0x7fb948101ad0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb944009900 tx=0x7fb94402bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:20.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.468+0000 7fb94fe59640 1 -- 172.21.15.23:0/3477311316 shutdown_connections 2024-03-28T05:21:20.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.468+0000 7fb94fe59640 1 --2- 172.21.15.23:0/3477311316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb948103260 0x7fb948109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.468+0000 7fb94fe59640 1 --2- 172.21.15.23:0/3477311316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9481028a0 0x7fb948102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.468+0000 7fb94fe59640 1 --2- 172.21.15.23:0/3477311316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9481016b0 0x7fb948101ad0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.468+0000 7fb94fe59640 1 -- 172.21.15.23:0/3477311316 >> 172.21.15.23:0/3477311316 conn(0x7fb9480fcc40 msgr2=0x7fb9480ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:20.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.469+0000 7fb94fe59640 1 -- 172.21.15.23:0/3477311316 shutdown_connections 2024-03-28T05:21:20.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.469+0000 7fb94fe59640 1 -- 172.21.15.23:0/3477311316 wait complete. 2024-03-28T05:21:20.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.469+0000 7fb94fe59640 1 Processor -- start 2024-03-28T05:21:20.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.469+0000 7fb94fe59640 1 -- start start 2024-03-28T05:21:20.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.469+0000 7fb94fe59640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9481016b0 0x7fb948197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:20.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.469+0000 7fb94fe59640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9481028a0 0x7fb948197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:20.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.469+0000 7fb94fe59640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb948103260 0x7fb94819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:20.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.469+0000 7fb94fe59640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94810d330 con 0x7fb9481016b0 2024-03-28T05:21:20.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.469+0000 7fb94fe59640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94810d1b0 con 0x7fb9481028a0 2024-03-28T05:21:20.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.469+0000 7fb94fe59640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94810d4b0 con 0x7fb948103260 2024-03-28T05:21:20.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.469+0000 7fb94dbf5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9481016b0 0x7fb948197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:20.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.469+0000 7fb94dbf5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9481016b0 0x7fb948197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57712/0 (socket says 172.21.15.23:57712) 2024-03-28T05:21:20.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.469+0000 7fb94dbf5640 1 -- 172.21.15.23:0/3970896721 learned_addr learned my addr 172.21.15.23:0/3970896721 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:20.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.469+0000 7fb94e3f6640 1 --2- 172.21.15.23:0/3970896721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb948103260 0x7fb94819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:20.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.469+0000 7fb94d3f4640 1 --2- 172.21.15.23:0/3970896721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9481028a0 0x7fb948197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:20.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.470+0000 7fb94dbf5640 1 -- 172.21.15.23:0/3970896721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb948103260 msgr2=0x7fb94819f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:20.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.470+0000 7fb94dbf5640 1 --2- 172.21.15.23:0/3970896721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb948103260 0x7fb94819f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.470+0000 7fb94dbf5640 1 -- 172.21.15.23:0/3970896721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9481028a0 msgr2=0x7fb948197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:20.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.470+0000 7fb94dbf5640 1 --2- 172.21.15.23:0/3970896721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9481028a0 0x7fb948197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.470+0000 7fb94dbf5640 1 -- 172.21.15.23:0/3970896721 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94819f690 con 0x7fb9481016b0 2024-03-28T05:21:20.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.470+0000 7fb94e3f6640 1 --2- 172.21.15.23:0/3970896721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb948103260 0x7fb94819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:21:20.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.470+0000 7fb94dbf5640 1 --2- 172.21.15.23:0/3970896721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9481016b0 0x7fb948197a70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb944000c00 tx=0x7fb94402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:20.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.470+0000 7fb93effd640 1 -- 172.21.15.23:0/3970896721 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb944041070 con 0x7fb9481016b0 2024-03-28T05:21:20.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.470+0000 7fb93effd640 1 -- 172.21.15.23:0/3970896721 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb944004450 con 0x7fb9481016b0 2024-03-28T05:21:20.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.470+0000 7fb94fe59640 1 -- 172.21.15.23:0/3970896721 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb94819f920 con 0x7fb9481016b0 2024-03-28T05:21:20.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.470+0000 7fb93effd640 1 -- 172.21.15.23:0/3970896721 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb944003df0 con 0x7fb9481016b0 2024-03-28T05:21:20.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.470+0000 7fb94fe59640 1 -- 172.21.15.23:0/3970896721 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb94819fe60 con 0x7fb9481016b0 2024-03-28T05:21:20.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.471+0000 7fb94fe59640 1 -- 172.21.15.23:0/3970896721 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb948108af0 con 0x7fb9481016b0 2024-03-28T05:21:20.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.472+0000 7fb93effd640 1 -- 172.21.15.23:0/3970896721 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb944044070 con 0x7fb9481016b0 2024-03-28T05:21:20.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.473+0000 7fb93effd640 1 --2- 172.21.15.23:0/3970896721 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb9340a2880 0x7fb9340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:20.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.473+0000 7fb94d3f4640 1 --2- 172.21.15.23:0/3970896721 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb9340a2880 0x7fb9340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:20.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.473+0000 7fb93effd640 1 -- 172.21.15.23:0/3970896721 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb9440f06f0 con 0x7fb9481016b0 2024-03-28T05:21:20.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.473+0000 7fb94d3f4640 1 --2- 172.21.15.23:0/3970896721 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb9340a2880 0x7fb9340a4d40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb948198f80 tx=0x7fb938005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:20.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.477+0000 7fb93effd640 1 -- 172.21.15.23:0/3970896721 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb9440bd5e0 con 0x7fb9481016b0 2024-03-28T05:21:20.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:20 smithi098 ceph-mon[36087]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 412 B/s wr, 3 op/s 2024-03-28T05:21:20.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:20 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/4220816502' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:20.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.659+0000 7fb94fe59640 1 -- 172.21.15.23:0/3970896721 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb9480622a0 con 0x7fb9481016b0 2024-03-28T05:21:20.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.659+0000 7fb93effd640 1 -- 172.21.15.23:0/3970896721 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb9440bcee0 con 0x7fb9481016b0 2024-03-28T05:21:20.658 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T05:21:20.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.661+0000 7fb94fe59640 1 -- 172.21.15.23:0/3970896721 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb9340a2880 msgr2=0x7fb9340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:20.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.661+0000 7fb94fe59640 1 --2- 172.21.15.23:0/3970896721 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb9340a2880 0x7fb9340a4d40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb948198f80 tx=0x7fb938005f90 comp rx=0 tx=0).stop 2024-03-28T05:21:20.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.661+0000 7fb94fe59640 1 -- 172.21.15.23:0/3970896721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9481016b0 msgr2=0x7fb948197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:20.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.661+0000 7fb94fe59640 1 --2- 172.21.15.23:0/3970896721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9481016b0 0x7fb948197a70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb944000c00 tx=0x7fb94402c640 comp rx=0 tx=0).stop 2024-03-28T05:21:20.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.661+0000 7fb94fe59640 1 -- 172.21.15.23:0/3970896721 shutdown_connections 2024-03-28T05:21:20.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.662+0000 7fb94fe59640 1 --2- 172.21.15.23:0/3970896721 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb9340a2880 0x7fb9340a4d40 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.662+0000 7fb94fe59640 1 --2- 172.21.15.23:0/3970896721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb948103260 0x7fb94819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.662+0000 7fb94fe59640 1 --2- 172.21.15.23:0/3970896721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9481028a0 0x7fb948197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.662+0000 7fb94fe59640 1 --2- 172.21.15.23:0/3970896721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9481016b0 0x7fb948197a70 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.662+0000 7fb94fe59640 1 -- 172.21.15.23:0/3970896721 >> 172.21.15.23:0/3970896721 conn(0x7fb9480fcc40 msgr2=0x7fb948105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:20.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.662+0000 7fb94fe59640 1 -- 172.21.15.23:0/3970896721 shutdown_connections 2024-03-28T05:21:20.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.662+0000 7fb94fe59640 1 -- 172.21.15.23:0/3970896721 wait complete. 2024-03-28T05:21:20.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:20 smithi023 ceph-mon[31954]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 412 B/s wr, 3 op/s 2024-03-28T05:21:20.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:20 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/4220816502' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:20.670 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T05:21:20.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:20 smithi113 ceph-mon[37164]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 412 B/s wr, 3 op/s 2024-03-28T05:21:20.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:20 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/4220816502' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:20.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.778+0000 7fa3f6bb9640 1 -- 172.21.15.23:0/1170657035 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3f0101790 msgr2=0x7fa3f0101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:20.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.778+0000 7fa3f6bb9640 1 --2- 172.21.15.23:0/1170657035 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3f0101790 0x7fa3f0101c10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc009900 tx=0x7fa3dc02bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:20.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.778+0000 7fa3f6bb9640 1 -- 172.21.15.23:0/1170657035 shutdown_connections 2024-03-28T05:21:20.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.778+0000 7fa3f6bb9640 1 --2- 172.21.15.23:0/1170657035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3f0102150 0x7fa3f01089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.778+0000 7fa3f6bb9640 1 --2- 172.21.15.23:0/1170657035 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3f0101790 0x7fa3f0101c10 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.778+0000 7fa3f6bb9640 1 --2- 172.21.15.23:0/1170657035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3f01005a0 0x7fa3f01009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.778+0000 7fa3f6bb9640 1 -- 172.21.15.23:0/1170657035 >> 172.21.15.23:0/1170657035 conn(0x7fa3f00fbb30 msgr2=0x7fa3f00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:20.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.778+0000 7fa3f6bb9640 1 -- 172.21.15.23:0/1170657035 shutdown_connections 2024-03-28T05:21:20.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.779+0000 7fa3f6bb9640 1 -- 172.21.15.23:0/1170657035 wait complete. 2024-03-28T05:21:20.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.779+0000 7fa3f6bb9640 1 Processor -- start 2024-03-28T05:21:20.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.779+0000 7fa3f6bb9640 1 -- start start 2024-03-28T05:21:20.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.779+0000 7fa3f6bb9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3f01005a0 0x7fa3f0197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:20.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.779+0000 7fa3f6bb9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3f0101790 0x7fa3f0197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:20.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.779+0000 7fa3f6bb9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3f0102150 0x7fa3f019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:20.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.779+0000 7fa3f6bb9640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f010c2b0 con 0x7fa3f0102150 2024-03-28T05:21:20.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.779+0000 7fa3f6bb9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f010c130 con 0x7fa3f01005a0 2024-03-28T05:21:20.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.779+0000 7fa3f6bb9640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f010c430 con 0x7fa3f0101790 2024-03-28T05:21:20.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.779+0000 7fa3f5156640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3f0102150 0x7fa3f019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:20.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.779+0000 7fa3f5156640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3f0102150 0x7fa3f019f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57726/0 (socket says 172.21.15.23:57726) 2024-03-28T05:21:20.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.779+0000 7fa3effff640 1 --2- 172.21.15.23:0/3943368445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3f0101790 0x7fa3f0197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:20.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.779+0000 7fa3f5156640 1 -- 172.21.15.23:0/3943368445 learned_addr learned my addr 172.21.15.23:0/3943368445 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:20.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.779+0000 7fa3f4955640 1 --2- 172.21.15.23:0/3943368445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3f01005a0 0x7fa3f0197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:20.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.780+0000 7fa3f5156640 1 -- 172.21.15.23:0/3943368445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3f0101790 msgr2=0x7fa3f0197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:20.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.780+0000 7fa3f5156640 1 --2- 172.21.15.23:0/3943368445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3f0101790 0x7fa3f0197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.780+0000 7fa3f5156640 1 -- 172.21.15.23:0/3943368445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3f01005a0 msgr2=0x7fa3f0197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:20.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.780+0000 7fa3f5156640 1 --2- 172.21.15.23:0/3943368445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3f01005a0 0x7fa3f0197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.780+0000 7fa3f5156640 1 -- 172.21.15.23:0/3943368445 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f019f6a0 con 0x7fa3f0102150 2024-03-28T05:21:20.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.780+0000 7fa3effff640 1 --2- 172.21.15.23:0/3943368445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3f0101790 0x7fa3f0197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:21:20.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.780+0000 7fa3f5156640 1 --2- 172.21.15.23:0/3943368445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3f0102150 0x7fa3f019f030 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa3e800eb10 tx=0x7fa3e800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:20.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.780+0000 7fa3edffb640 1 -- 172.21.15.23:0/3943368445 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e800ce20 con 0x7fa3f0102150 2024-03-28T05:21:20.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.780+0000 7fa3f6bb9640 1 -- 172.21.15.23:0/3943368445 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f019f990 con 0x7fa3f0102150 2024-03-28T05:21:20.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.780+0000 7fa3edffb640 1 -- 172.21.15.23:0/3943368445 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3e8005020 con 0x7fa3f0102150 2024-03-28T05:21:20.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.780+0000 7fa3edffb640 1 -- 172.21.15.23:0/3943368445 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e8010690 con 0x7fa3f0102150 2024-03-28T05:21:20.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.780+0000 7fa3f6bb9640 1 -- 172.21.15.23:0/3943368445 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f019feb0 con 0x7fa3f0102150 2024-03-28T05:21:20.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.781+0000 7fa3f6bb9640 1 -- 172.21.15.23:0/3943368445 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f01a0140 con 0x7fa3f0102150 2024-03-28T05:21:20.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.782+0000 7fa3edffb640 1 -- 172.21.15.23:0/3943368445 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa3e8005540 con 0x7fa3f0102150 2024-03-28T05:21:20.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.783+0000 7fa3edffb640 1 --2- 172.21.15.23:0/3943368445 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa3e00a2800 0x7fa3e00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:20.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.783+0000 7fa3edffb640 1 -- 172.21.15.23:0/3943368445 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fa3e80cf360 con 0x7fa3f0102150 2024-03-28T05:21:20.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.783+0000 7fa3f4955640 1 --2- 172.21.15.23:0/3943368445 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa3e00a2800 0x7fa3e00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:20.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.783+0000 7fa3f4955640 1 --2- 172.21.15.23:0/3943368445 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa3e00a2800 0x7fa3e00a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa3f01015f0 tx=0x7fa3e4009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:20.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.787+0000 7fa3edffb640 1 -- 172.21.15.23:0/3943368445 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa3e809c1c0 con 0x7fa3f0102150 2024-03-28T05:21:20.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.972+0000 7fa3f6bb9640 1 -- 172.21.15.23:0/3943368445 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa3f01a0310 con 0x7fa3f0102150 2024-03-28T05:21:20.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.972+0000 7fa3edffb640 1 -- 172.21.15.23:0/3943368445 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa3e809bac0 con 0x7fa3f0102150 2024-03-28T05:21:20.970 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-28T05:21:20.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.974+0000 7fa3f6bb9640 1 -- 172.21.15.23:0/3943368445 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa3e00a2800 msgr2=0x7fa3e00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:20.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.974+0000 7fa3f6bb9640 1 --2- 172.21.15.23:0/3943368445 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa3e00a2800 0x7fa3e00a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa3f01015f0 tx=0x7fa3e4009110 comp rx=0 tx=0).stop 2024-03-28T05:21:20.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.974+0000 7fa3f6bb9640 1 -- 172.21.15.23:0/3943368445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3f0102150 msgr2=0x7fa3f019f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:20.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.974+0000 7fa3f6bb9640 1 --2- 172.21.15.23:0/3943368445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3f0102150 0x7fa3f019f030 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa3e800eb10 tx=0x7fa3e800ef10 comp rx=0 tx=0).stop 2024-03-28T05:21:20.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.974+0000 7fa3f6bb9640 1 -- 172.21.15.23:0/3943368445 shutdown_connections 2024-03-28T05:21:20.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.974+0000 7fa3f6bb9640 1 --2- 172.21.15.23:0/3943368445 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa3e00a2800 0x7fa3e00a4cc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.974+0000 7fa3f6bb9640 1 --2- 172.21.15.23:0/3943368445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3f0102150 0x7fa3f019f030 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.974+0000 7fa3f6bb9640 1 --2- 172.21.15.23:0/3943368445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3f0101790 0x7fa3f0197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.974+0000 7fa3f6bb9640 1 --2- 172.21.15.23:0/3943368445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3f01005a0 0x7fa3f0197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:20.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.974+0000 7fa3f6bb9640 1 -- 172.21.15.23:0/3943368445 >> 172.21.15.23:0/3943368445 conn(0x7fa3f00fbb30 msgr2=0x7fa3f00fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:20.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.974+0000 7fa3f6bb9640 1 -- 172.21.15.23:0/3943368445 shutdown_connections 2024-03-28T05:21:20.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:20.974+0000 7fa3f6bb9640 1 -- 172.21.15.23:0/3943368445 wait complete. 2024-03-28T05:21:20.983 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T05:21:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.090+0000 7f881ff90640 1 -- 172.21.15.23:0/1750353138 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88181016c0 msgr2=0x7f8818101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.090+0000 7f881ff90640 1 --2- 172.21.15.23:0/1750353138 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88181016c0 0x7f8818101ae0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8814009900 tx=0x7f881402bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.090+0000 7f881ff90640 1 -- 172.21.15.23:0/1750353138 shutdown_connections 2024-03-28T05:21:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.090+0000 7f881ff90640 1 --2- 172.21.15.23:0/1750353138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8818103270 0x7f8818109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.090+0000 7f881ff90640 1 --2- 172.21.15.23:0/1750353138 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f88181028b0 0x7f8818102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.090+0000 7f881ff90640 1 --2- 172.21.15.23:0/1750353138 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88181016c0 0x7f8818101ae0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.090+0000 7f881ff90640 1 -- 172.21.15.23:0/1750353138 >> 172.21.15.23:0/1750353138 conn(0x7f88180fcc30 msgr2=0x7f88180ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.090+0000 7f881ff90640 1 -- 172.21.15.23:0/1750353138 shutdown_connections 2024-03-28T05:21:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.090+0000 7f881ff90640 1 -- 172.21.15.23:0/1750353138 wait complete. 2024-03-28T05:21:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.091+0000 7f881ff90640 1 Processor -- start 2024-03-28T05:21:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.091+0000 7f881ff90640 1 -- start start 2024-03-28T05:21:21.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.091+0000 7f881ff90640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88181016c0 0x7f8818075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:21.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.091+0000 7f881ff90640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88181028b0 0x7f8818073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:21.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.091+0000 7f881ff90640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8818103270 0x7f8818073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:21.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.091+0000 7f881ff90640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881810d310 con 0x7f88181016c0 2024-03-28T05:21:21.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.091+0000 7f881ff90640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881810d190 con 0x7f88181028b0 2024-03-28T05:21:21.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.091+0000 7f881ff90640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881810d490 con 0x7f8818103270 2024-03-28T05:21:21.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.091+0000 7f881dd2c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88181016c0 0x7f8818075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:21.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.091+0000 7f881dd2c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88181016c0 0x7f8818075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57742/0 (socket says 172.21.15.23:57742) 2024-03-28T05:21:21.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.091+0000 7f881dd2c640 1 -- 172.21.15.23:0/3278695341 learned_addr learned my addr 172.21.15.23:0/3278695341 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:21.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.091+0000 7f881d52b640 1 --2- 172.21.15.23:0/3278695341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88181028b0 0x7f8818073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:21.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.091+0000 7f881e52d640 1 --2- 172.21.15.23:0/3278695341 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8818103270 0x7f8818073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:21.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.091+0000 7f881dd2c640 1 -- 172.21.15.23:0/3278695341 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8818103270 msgr2=0x7f8818073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:21.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.091+0000 7f881dd2c640 1 --2- 172.21.15.23:0/3278695341 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8818103270 0x7f8818073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.091+0000 7f881dd2c640 1 -- 172.21.15.23:0/3278695341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88181028b0 msgr2=0x7f8818073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:21.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.091+0000 7f881dd2c640 1 --2- 172.21.15.23:0/3278695341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88181028b0 0x7f8818073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.091+0000 7f881dd2c640 1 -- 172.21.15.23:0/3278695341 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8818074860 con 0x7f88181016c0 2024-03-28T05:21:21.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.092+0000 7f881dd2c640 1 --2- 172.21.15.23:0/3278695341 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88181016c0 0x7f8818075450 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8814000c00 tx=0x7f8814005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:21.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.092+0000 7f880effd640 1 -- 172.21.15.23:0/3278695341 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8814041070 con 0x7f88181016c0 2024-03-28T05:21:21.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.092+0000 7f881ff90640 1 -- 172.21.15.23:0/3278695341 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88181a1a90 con 0x7f88181016c0 2024-03-28T05:21:21.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.092+0000 7f880effd640 1 -- 172.21.15.23:0/3278695341 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88140044d0 con 0x7f88181016c0 2024-03-28T05:21:21.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.092+0000 7f880effd640 1 -- 172.21.15.23:0/3278695341 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8814034440 con 0x7f88181016c0 2024-03-28T05:21:21.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.092+0000 7f881ff90640 1 -- 172.21.15.23:0/3278695341 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88181a2030 con 0x7f88181016c0 2024-03-28T05:21:21.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.093+0000 7f881ff90640 1 -- 172.21.15.23:0/3278695341 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8818108ab0 con 0x7f88181016c0 2024-03-28T05:21:21.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.094+0000 7f880effd640 1 -- 172.21.15.23:0/3278695341 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8814044070 con 0x7f88181016c0 2024-03-28T05:21:21.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.094+0000 7f880effd640 1 --2- 172.21.15.23:0/3278695341 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f88040a2800 0x7f88040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:21.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.095+0000 7f880effd640 1 -- 172.21.15.23:0/3278695341 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f88140f09b0 con 0x7f88181016c0 2024-03-28T05:21:21.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.095+0000 7f881d52b640 1 --2- 172.21.15.23:0/3278695341 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f88040a2800 0x7f88040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:21.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.095+0000 7f881d52b640 1 --2- 172.21.15.23:0/3278695341 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f88040a2800 0x7f88040a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f88080095f0 tx=0x7f8808005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:21.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.098+0000 7f880effd640 1 -- 172.21.15.23:0/3278695341 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f88140bd1c0 con 0x7f88181016c0 2024-03-28T05:21:21.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.280+0000 7f881ff90640 1 -- 172.21.15.23:0/3278695341 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f881802d200 con 0x7f88181016c0 2024-03-28T05:21:21.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.280+0000 7f880effd640 1 -- 172.21.15.23:0/3278695341 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f88140c1260 con 0x7f88181016c0 2024-03-28T05:21:21.278 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T05:21:21.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.282+0000 7f881ff90640 1 -- 172.21.15.23:0/3278695341 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f88040a2800 msgr2=0x7f88040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:21.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.282+0000 7f881ff90640 1 --2- 172.21.15.23:0/3278695341 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f88040a2800 0x7f88040a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f88080095f0 tx=0x7f8808005f90 comp rx=0 tx=0).stop 2024-03-28T05:21:21.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.282+0000 7f881ff90640 1 -- 172.21.15.23:0/3278695341 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88181016c0 msgr2=0x7f8818075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:21.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.282+0000 7f881ff90640 1 --2- 172.21.15.23:0/3278695341 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88181016c0 0x7f8818075450 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8814000c00 tx=0x7f8814005d50 comp rx=0 tx=0).stop 2024-03-28T05:21:21.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.282+0000 7f881ff90640 1 -- 172.21.15.23:0/3278695341 shutdown_connections 2024-03-28T05:21:21.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.282+0000 7f881ff90640 1 --2- 172.21.15.23:0/3278695341 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f88040a2800 0x7f88040a4cc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.282+0000 7f881ff90640 1 --2- 172.21.15.23:0/3278695341 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8818103270 0x7f8818073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.282+0000 7f881ff90640 1 --2- 172.21.15.23:0/3278695341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88181028b0 0x7f8818073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.282+0000 7f881ff90640 1 --2- 172.21.15.23:0/3278695341 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88181016c0 0x7f8818075450 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.282+0000 7f881ff90640 1 -- 172.21.15.23:0/3278695341 >> 172.21.15.23:0/3278695341 conn(0x7f88180fcc30 msgr2=0x7f8818105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:21.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.282+0000 7f881ff90640 1 -- 172.21.15.23:0/3278695341 shutdown_connections 2024-03-28T05:21:21.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.282+0000 7f881ff90640 1 -- 172.21.15.23:0/3278695341 wait complete. 2024-03-28T05:21:21.291 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T05:21:21.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.397+0000 7fc734827640 1 -- 172.21.15.23:0/2350064219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc730103270 msgr2=0x7fc730109b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:21.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.397+0000 7fc734827640 1 --2- 172.21.15.23:0/2350064219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc730103270 0x7fc730109b00 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc728009ca0 tx=0x7fc72802c0f0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.397+0000 7fc734827640 1 -- 172.21.15.23:0/2350064219 shutdown_connections 2024-03-28T05:21:21.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.397+0000 7fc734827640 1 --2- 172.21.15.23:0/2350064219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc730103270 0x7fc730109b00 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.397+0000 7fc734827640 1 --2- 172.21.15.23:0/2350064219 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7301028b0 0x7fc730102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.397+0000 7fc734827640 1 --2- 172.21.15.23:0/2350064219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7301016c0 0x7fc730101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.397+0000 7fc734827640 1 -- 172.21.15.23:0/2350064219 >> 172.21.15.23:0/2350064219 conn(0x7fc7300fcc30 msgr2=0x7fc7300ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:21.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.397+0000 7fc734827640 1 -- 172.21.15.23:0/2350064219 shutdown_connections 2024-03-28T05:21:21.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.397+0000 7fc734827640 1 -- 172.21.15.23:0/2350064219 wait complete. 2024-03-28T05:21:21.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.398+0000 7fc734827640 1 Processor -- start 2024-03-28T05:21:21.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.398+0000 7fc734827640 1 -- start start 2024-03-28T05:21:21.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.398+0000 7fc734827640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7301016c0 0x7fc730197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:21.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.398+0000 7fc734827640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7301028b0 0x7fc730197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:21.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.398+0000 7fc734827640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc730103270 0x7fc73019ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:21.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.398+0000 7fc734827640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73010d340 con 0x7fc730103270 2024-03-28T05:21:21.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.398+0000 7fc734827640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73010d1c0 con 0x7fc7301028b0 2024-03-28T05:21:21.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.398+0000 7fc734827640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73010d4c0 con 0x7fc7301016c0 2024-03-28T05:21:21.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.398+0000 7fc72ed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc730103270 0x7fc73019ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:21.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.398+0000 7fc72e59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7301016c0 0x7fc730197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:21.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.398+0000 7fc72ed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc730103270 0x7fc73019ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57760/0 (socket says 172.21.15.23:57760) 2024-03-28T05:21:21.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.398+0000 7fc72e59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7301016c0 0x7fc730197960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:32942/0 (socket says 172.21.15.23:32942) 2024-03-28T05:21:21.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.398+0000 7fc72ed9d640 1 -- 172.21.15.23:0/3909410030 learned_addr learned my addr 172.21.15.23:0/3909410030 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:21.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.398+0000 7fc725d9b640 1 --2- 172.21.15.23:0/3909410030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7301028b0 0x7fc730197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:21.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.398+0000 7fc72ed9d640 1 -- 172.21.15.23:0/3909410030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7301016c0 msgr2=0x7fc730197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:21.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.398+0000 7fc72ed9d640 1 --2- 172.21.15.23:0/3909410030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7301016c0 0x7fc730197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.398+0000 7fc72ed9d640 1 -- 172.21.15.23:0/3909410030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7301028b0 msgr2=0x7fc730197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:21.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.398+0000 7fc72ed9d640 1 --2- 172.21.15.23:0/3909410030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7301028b0 0x7fc730197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.398+0000 7fc72ed9d640 1 -- 172.21.15.23:0/3909410030 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73019f580 con 0x7fc730103270 2024-03-28T05:21:21.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.398+0000 7fc72e59c640 1 --2- 172.21.15.23:0/3909410030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7301016c0 0x7fc730197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:21:21.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.398+0000 7fc725d9b640 1 --2- 172.21.15.23:0/3909410030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7301028b0 0x7fc730197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:21:21.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.399+0000 7fc72ed9d640 1 --2- 172.21.15.23:0/3909410030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc730103270 0x7fc73019ef10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc728037040 tx=0x7fc728032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:21.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.399+0000 7fc727fff640 1 -- 172.21.15.23:0/3909410030 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc728041020 con 0x7fc730103270 2024-03-28T05:21:21.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.399+0000 7fc727fff640 1 -- 172.21.15.23:0/3909410030 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc728034cd0 con 0x7fc730103270 2024-03-28T05:21:21.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.399+0000 7fc734827640 1 -- 172.21.15.23:0/3909410030 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc73019f810 con 0x7fc730103270 2024-03-28T05:21:21.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.399+0000 7fc727fff640 1 -- 172.21.15.23:0/3909410030 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc728039040 con 0x7fc730103270 2024-03-28T05:21:21.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.399+0000 7fc734827640 1 -- 172.21.15.23:0/3909410030 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc73019fd00 con 0x7fc730103270 2024-03-28T05:21:21.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.400+0000 7fc734827640 1 -- 172.21.15.23:0/3909410030 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc730108980 con 0x7fc730103270 2024-03-28T05:21:21.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.401+0000 7fc727fff640 1 -- 172.21.15.23:0/3909410030 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc728044020 con 0x7fc730103270 2024-03-28T05:21:21.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.401+0000 7fc727fff640 1 --2- 172.21.15.23:0/3909410030 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc71c0a2800 0x7fc71c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:21.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.401+0000 7fc72e59c640 1 --2- 172.21.15.23:0/3909410030 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc71c0a2800 0x7fc71c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:21.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.401+0000 7fc727fff640 1 -- 172.21.15.23:0/3909410030 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc7280f0bc0 con 0x7fc730103270 2024-03-28T05:21:21.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.402+0000 7fc72e59c640 1 --2- 172.21.15.23:0/3909410030 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc71c0a2800 0x7fc71c0a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc730102710 tx=0x7fc720005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:21.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.406+0000 7fc727fff640 1 -- 172.21.15.23:0/3909410030 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc72804b020 con 0x7fc730103270 2024-03-28T05:21:21.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.602+0000 7fc734827640 1 -- 172.21.15.23:0/3909410030 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc7300622a0 con 0x7fc730103270 2024-03-28T05:21:21.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.602+0000 7fc727fff640 1 -- 172.21.15.23:0/3909410030 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+592 (secure 0 0 0) 0x7fc7280dbe20 con 0x7fc730103270 2024-03-28T05:21:21.600 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/1530849123 2024-03-29T05:14:08.640340+0000 2024-03-28T05:21:21.600 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/1530849123 2024-03-29T05:14:08.640340+0000 2024-03-28T05:21:21.600 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1876451381 2024-03-29T05:14:08.640340+0000 2024-03-28T05:21:21.601 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2253559862 2024-03-29T05:14:08.640340+0000 2024-03-28T05:21:21.601 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2653562762 2024-03-29T05:13:42.579607+0000 2024-03-28T05:21:21.601 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/3337463830 2024-03-29T05:13:42.579607+0000 2024-03-28T05:21:21.601 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/1735221490 2024-03-29T05:13:42.579607+0000 2024-03-28T05:21:21.601 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2784771635 2024-03-29T05:14:08.640340+0000 2024-03-28T05:21:21.601 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2389902490 2024-03-29T05:13:42.579607+0000 2024-03-28T05:21:21.601 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/1735221490 2024-03-29T05:13:42.579607+0000 2024-03-28T05:21:21.601 INFO:teuthology.orchestra.run.smithi023.stderr:listed 10 entries 2024-03-28T05:21:21.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.604+0000 7fc734827640 1 -- 172.21.15.23:0/3909410030 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc71c0a2800 msgr2=0x7fc71c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:21.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.604+0000 7fc734827640 1 --2- 172.21.15.23:0/3909410030 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc71c0a2800 0x7fc71c0a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc730102710 tx=0x7fc720005f90 comp rx=0 tx=0).stop 2024-03-28T05:21:21.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.604+0000 7fc734827640 1 -- 172.21.15.23:0/3909410030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc730103270 msgr2=0x7fc73019ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:21.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.604+0000 7fc734827640 1 --2- 172.21.15.23:0/3909410030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc730103270 0x7fc73019ef10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc728037040 tx=0x7fc728032ea0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.604+0000 7fc734827640 1 -- 172.21.15.23:0/3909410030 shutdown_connections 2024-03-28T05:21:21.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.604+0000 7fc734827640 1 --2- 172.21.15.23:0/3909410030 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc71c0a2800 0x7fc71c0a4cc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.604+0000 7fc734827640 1 --2- 172.21.15.23:0/3909410030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc730103270 0x7fc73019ef10 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.604+0000 7fc734827640 1 --2- 172.21.15.23:0/3909410030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7301028b0 0x7fc730197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.604+0000 7fc734827640 1 --2- 172.21.15.23:0/3909410030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc7301016c0 0x7fc730197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.604+0000 7fc734827640 1 -- 172.21.15.23:0/3909410030 >> 172.21.15.23:0/3909410030 conn(0x7fc7300fcc30 msgr2=0x7fc730105b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:21.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.604+0000 7fc734827640 1 -- 172.21.15.23:0/3909410030 shutdown_connections 2024-03-28T05:21:21.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.604+0000 7fc734827640 1 -- 172.21.15.23:0/3909410030 wait complete. 2024-03-28T05:21:21.612 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:21 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1933994673' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:21:21.612 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:21 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3970896721' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:21:21.612 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:21 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3943368445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:21:21.612 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:21 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3278695341' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:21:21.613 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T05:21:21.613 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T05:21:21.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:21 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1933994673' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:21:21.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:21 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3970896721' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:21:21.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:21 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3943368445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:21:21.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:21 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3278695341' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:21:21.628 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T05:21:21.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:21 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1933994673' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:21:21.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:21 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3970896721' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:21:21.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:21 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3943368445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:21:21.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:21 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3278695341' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:21:21.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.772+0000 7f4c4644b640 1 -- 172.21.15.23:0/3064424023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c401005c0 msgr2=0x7f4c401009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:21.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.772+0000 7f4c4644b640 1 --2- 172.21.15.23:0/3064424023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c401005c0 0x7f4c401009e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4c30009900 tx=0x7f4c3002bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:21.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.773+0000 7f4c4644b640 1 -- 172.21.15.23:0/3064424023 shutdown_connections 2024-03-28T05:21:21.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.773+0000 7f4c4644b640 1 --2- 172.21.15.23:0/3064424023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c40102170 0x7f4c40108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.773+0000 7f4c4644b640 1 --2- 172.21.15.23:0/3064424023 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c401017b0 0x7f4c40101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.773+0000 7f4c4644b640 1 --2- 172.21.15.23:0/3064424023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c401005c0 0x7f4c401009e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.773+0000 7f4c4644b640 1 -- 172.21.15.23:0/3064424023 >> 172.21.15.23:0/3064424023 conn(0x7f4c400fbb30 msgr2=0x7f4c400fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:21.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.773+0000 7f4c4644b640 1 -- 172.21.15.23:0/3064424023 shutdown_connections 2024-03-28T05:21:21.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.773+0000 7f4c4644b640 1 -- 172.21.15.23:0/3064424023 wait complete. 2024-03-28T05:21:21.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.773+0000 7f4c4644b640 1 Processor -- start 2024-03-28T05:21:21.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.773+0000 7f4c4644b640 1 -- start start 2024-03-28T05:21:21.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.773+0000 7f4c4644b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c401005c0 0x7f4c401979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:21.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.773+0000 7f4c4644b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c401017b0 0x7f4c40197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:21.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.773+0000 7f4c4644b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c40102170 0x7f4c4019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:21.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.773+0000 7f4c4644b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4010c290 con 0x7f4c40102170 2024-03-28T05:21:21.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.773+0000 7f4c4644b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4010c110 con 0x7f4c401005c0 2024-03-28T05:21:21.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.773+0000 7f4c4644b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4010c410 con 0x7f4c401017b0 2024-03-28T05:21:21.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.774+0000 7f4c449e8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c40102170 0x7f4c4019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:21.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.774+0000 7f4c449e8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c40102170 0x7f4c4019efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57778/0 (socket says 172.21.15.23:57778) 2024-03-28T05:21:21.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.774+0000 7f4c3f7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c401017b0 0x7f4c40197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:21.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.774+0000 7f4c449e8640 1 -- 172.21.15.23:0/4095326660 learned_addr learned my addr 172.21.15.23:0/4095326660 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:21.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.774+0000 7f4c3ffff640 1 --2- 172.21.15.23:0/4095326660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c401005c0 0x7f4c401979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:21.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.774+0000 7f4c449e8640 1 -- 172.21.15.23:0/4095326660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c401017b0 msgr2=0x7f4c40197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:21.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.774+0000 7f4c449e8640 1 --2- 172.21.15.23:0/4095326660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c401017b0 0x7f4c40197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.774+0000 7f4c449e8640 1 -- 172.21.15.23:0/4095326660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c401005c0 msgr2=0x7f4c401979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:21.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.774+0000 7f4c449e8640 1 --2- 172.21.15.23:0/4095326660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c401005c0 0x7f4c401979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.774+0000 7f4c449e8640 1 -- 172.21.15.23:0/4095326660 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4019f610 con 0x7f4c40102170 2024-03-28T05:21:21.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.774+0000 7f4c3f7fe640 1 --2- 172.21.15.23:0/4095326660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c401017b0 0x7f4c40197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:21:21.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.774+0000 7f4c449e8640 1 --2- 172.21.15.23:0/4095326660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c40102170 0x7f4c4019efa0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4c3800cce0 tx=0x7f4c3800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:21.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.774+0000 7f4c3d7fa640 1 -- 172.21.15.23:0/4095326660 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3800f450 con 0x7f4c40102170 2024-03-28T05:21:21.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.774+0000 7f4c3d7fa640 1 -- 172.21.15.23:0/4095326660 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c3800fad0 con 0x7f4c40102170 2024-03-28T05:21:21.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.774+0000 7f4c4644b640 1 -- 172.21.15.23:0/4095326660 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c4019f900 con 0x7f4c40102170 2024-03-28T05:21:21.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.774+0000 7f4c3d7fa640 1 -- 172.21.15.23:0/4095326660 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c38018c10 con 0x7f4c40102170 2024-03-28T05:21:21.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.774+0000 7f4c4644b640 1 -- 172.21.15.23:0/4095326660 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4019fe00 con 0x7f4c40102170 2024-03-28T05:21:21.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.775+0000 7f4c4644b640 1 -- 172.21.15.23:0/4095326660 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c401a0090 con 0x7f4c40102170 2024-03-28T05:21:21.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.776+0000 7f4c3d7fa640 1 -- 172.21.15.23:0/4095326660 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4c3800f5f0 con 0x7f4c40102170 2024-03-28T05:21:21.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.777+0000 7f4c3d7fa640 1 --2- 172.21.15.23:0/4095326660 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f4c280a2800 0x7f4c280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:21.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.777+0000 7f4c3ffff640 1 --2- 172.21.15.23:0/4095326660 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f4c280a2800 0x7f4c280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:21.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.777+0000 7f4c3d7fa640 1 -- 172.21.15.23:0/4095326660 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f4c380cf2e0 con 0x7f4c40102170 2024-03-28T05:21:21.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.777+0000 7f4c3ffff640 1 --2- 172.21.15.23:0/4095326660 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f4c280a2800 0x7f4c280a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4c3002c170 tx=0x7f4c3002df90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:21.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.781+0000 7f4c3d7fa640 1 -- 172.21.15.23:0/4095326660 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4c38014030 con 0x7f4c40102170 2024-03-28T05:21:21.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.967+0000 7f4c4644b640 1 -- 172.21.15.23:0/4095326660 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c401a0260 con 0x7f4c40102170 2024-03-28T05:21:21.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.968+0000 7f4c3d7fa640 1 -- 172.21.15.23:0/4095326660 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24082 (secure 0 0 0) 0x7f4c3809c140 con 0x7f4c40102170 2024-03-28T05:21:21.968 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:21:21.968 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":89,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","created":"2024-03-28T05:13:13.453653+0000","modified":"2024-03-28T05:21:12.641185+0000","last_up_change":"2024-03-28T05:20:20.221527+0000","last_in_change":"2024-03-28T05:20:00.365478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T05:16:32.986974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T05:21:06.103894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T05:21:06.827190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T05:21:10.507418+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"18150942-59a4-4c3b-83d4-c1fc3ccf1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6803","nonce":1219956881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6805","nonce":1219956881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6809","nonce":1219956881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6807","nonce":1219956881}]},"public_addr":"172.21.15.23:6803/1219956881","cluster_addr":"172.21.15.23:6805/1219956881","heartbeat_back_addr":"172.21.15.23:6809/1219956881","heartbeat_front_addr":"172.21.15.23:6807/1219956881","state":["exists","up"]},{"osd":1,"uuid":"ba6d1a95-5135-4670-95f9-7878c0e7bd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6801","nonce":555491234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6803","nonce":555491234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6807","nonce":555491234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6805","nonce":555491234}]},"public_addr":"172.21.15.98:6801/555491234","cluster_addr":"172.21.15.98:6803/555491234","heartbeat_back_addr":"172.21.15.98:6807/555491234","heartbeat_front_addr":"172.21.15.98:6805/555491234","state":["exists","up"]},{"osd":2,"uuid":"873bdd60-2a47-403b-b1f4-ee9226a95067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6801","nonce":1535856914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6803","nonce":1535856914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6807","nonce":1535856914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6805","nonce":1535856914}]},"public_addr":"172.21.15.113:6801/1535856914","cluster_addr":"172.21.15.113:6803/1535856914","heartbeat_back_addr":"172.21.15.113:6807/1535856914","heartbeat_front_addr":"172.21.15.113:6805/1535856914","state":["exists","up"]},{"osd":3,"uuid":"83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6811","nonce":2949415020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6813","nonce":2949415020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6817","nonce":2949415020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6815","nonce":2949415020}]},"public_addr":"172.21.15.23:6811/2949415020","cluster_addr":"172.21.15.23:6813/2949415020","heartbeat_back_addr":"172.21.15.23:6817/2949415020","heartbeat_front_addr":"172.21.15.23:6815/2949415020","state":["exists","up"]},{"osd":4,"uuid":"76a90bac-b221-40a3-8238-40e8049cec36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6809","nonce":2615902819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6811","nonce":2615902819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6815","nonce":2615902819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6813","nonce":2615902819}]},"public_addr":"172.21.15.98:6809/2615902819","cluster_addr":"172.21.15.98:6811/2615902819","heartbeat_back_addr":"172.21.15.98:6815/2615902819","heartbeat_front_addr":"172.21.15.98:6813/2615902819","state":["exists","up"]},{"osd":5,"uuid":"b53fdb55-a8df-40a8-bd1b-8bcef634e26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6809","nonce":625271528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6811","nonce":625271528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6815","nonce":625271528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6813","nonce":625271528}]},"public_addr":"172.21.15.113:6809/625271528","cluster_addr":"172.21.15.113:6811/625271528","heartbeat_back_addr":"172.21.15.113:6815/625271528","heartbeat_front_addr":"172.21.15.113:6813/625271528","state":["exists","up"]},{"osd":6,"uuid":"f00283d2-8d67-4e97-a575-9a9d3b55d5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6819","nonce":4188464176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6821","nonce":4188464176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6825","nonce":4188464176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6823","nonce":4188464176}]},"public_addr":"172.21.15.23:6819/4188464176","cluster_addr":"172.21.15.23:6821/4188464176","heartbeat_back_addr":"172.21.15.23:6825/4188464176","heartbeat_front_addr":"172.21.15.23:6823/4188464176","state":["exists","up"]},{"osd":7,"uuid":"edc008a7-b07b-40a8-88b7-1614ba38a775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6817","nonce":1985546028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6819","nonce":1985546028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6823","nonce":1985546028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6821","nonce":1985546028}]},"public_addr":"172.21.15.98:6817/1985546028","cluster_addr":"172.21.15.98:6819/1985546028","heartbeat_back_addr":"172.21.15.98:6823/1985546028","heartbeat_front_addr":"172.21.15.98:6821/1985546028","state":["exists","up"]},{"osd":8,"uuid":"56e7b2d1-81f0-47b2-8638-f29832bed8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6817","nonce":3534876081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6819","nonce":3534876081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6823","nonce":3534876081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6821","nonce":3534876081}]},"public_addr":"172.21.15.113:6817/3534876081","cluster_addr":"172.21.15.113:6819/3534876081","heartbeat_back_addr":"172.21.15.113:6823/3534876081","heartbeat_front_addr":"172.21.15.113:6821/3534876081","state":["exists","up"]},{"osd":9,"uuid":"a2e90caa-6e9c-42bf-9f18-81f7f47880cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6827","nonce":672264207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6829","nonce":672264207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6833","nonce":672264207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6831","nonce":672264207}]},"public_addr":"172.21.15.23:6827/672264207","cluster_addr":"172.21.15.23:6829/672264207","heartbeat_back_addr":"172.21.15.23:6833/672264207","heartbeat_front_addr":"172.21.15.23:6831/672264207","state":["exists","up"]},{"osd":10,"uuid":"0eda917f-020c-437f-95ae-b26203f6c68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6825","nonce":2699920595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6827","nonce":2699920595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6831","nonce":2699920595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6829","nonce":2699920595}]},"public_addr":"172.21.15.98:6825/2699920595","cluster_addr":"172.21.15.98:6827/2699920595","heartbeat_back_addr":"172.21.15.98:6831/2699920595","heartbeat_front_addr":"172.21.15.98:6829/2699920595","state":["exists","up"]},{"osd":11,"uuid":"aae78501-78a8-469a-be20-3006f5ec5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6825","nonce":2113644020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6827","nonce":2113644020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6831","nonce":2113644020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6829","nonce":2113644020}]},"public_addr":"172.21.15.113:6825/2113644020","cluster_addr":"172.21.15.113:6827/2113644020","heartbeat_back_addr":"172.21.15.113:6831/2113644020","heartbeat_front_addr":"172.21.15.113:6829/2113644020","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:15:43.527475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:07.006538+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:30.594176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:55.810999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:20.595811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:45.474138+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:10.500409+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:35.787638+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:01.001188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:26.761115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:52.952752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:20:18.559832+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:6800/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1876451381":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2253559862":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2653562762":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/3337463830":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6800/1735221490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/2784771635":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2389902490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6801/1735221490":"2024-03-29T05:13:42.579607+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T05:21:21.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.970+0000 7f4c4644b640 1 -- 172.21.15.23:0/4095326660 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f4c280a2800 msgr2=0x7f4c280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:21.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.970+0000 7f4c4644b640 1 --2- 172.21.15.23:0/4095326660 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f4c280a2800 0x7f4c280a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4c3002c170 tx=0x7f4c3002df90 comp rx=0 tx=0).stop 2024-03-28T05:21:21.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.970+0000 7f4c4644b640 1 -- 172.21.15.23:0/4095326660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c40102170 msgr2=0x7f4c4019efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:21.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.970+0000 7f4c4644b640 1 --2- 172.21.15.23:0/4095326660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c40102170 0x7f4c4019efa0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4c3800cce0 tx=0x7f4c3800a620 comp rx=0 tx=0).stop 2024-03-28T05:21:21.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.970+0000 7f4c4644b640 1 -- 172.21.15.23:0/4095326660 shutdown_connections 2024-03-28T05:21:21.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.970+0000 7f4c4644b640 1 --2- 172.21.15.23:0/4095326660 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f4c280a2800 0x7f4c280a4cc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.970+0000 7f4c4644b640 1 --2- 172.21.15.23:0/4095326660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c40102170 0x7f4c4019efa0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.970+0000 7f4c4644b640 1 --2- 172.21.15.23:0/4095326660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c401017b0 0x7f4c40197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.970+0000 7f4c4644b640 1 --2- 172.21.15.23:0/4095326660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c401005c0 0x7f4c401979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:21.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.970+0000 7f4c4644b640 1 -- 172.21.15.23:0/4095326660 >> 172.21.15.23:0/4095326660 conn(0x7f4c400fbb30 msgr2=0x7f4c400fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:21.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.970+0000 7f4c4644b640 1 -- 172.21.15.23:0/4095326660 shutdown_connections 2024-03-28T05:21:21.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:21.970+0000 7f4c4644b640 1 -- 172.21.15.23:0/4095326660 wait complete. 2024-03-28T05:21:21.981 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T05:16:32.986974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T05:21:06.103894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T05:21:06.827190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T05:21:10.507418+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T05:21:21.981 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T05:21:22.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.087+0000 7f3a8a359640 1 -- 172.21.15.23:0/2244499338 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a84107440 msgr2=0x7f3a84109830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:22.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.087+0000 7f3a8a359640 1 --2- 172.21.15.23:0/2244499338 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a84107440 0x7f3a84109830 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c009ca0 tx=0x7f3a7c02c0f0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.087+0000 7f3a8a359640 1 -- 172.21.15.23:0/2244499338 shutdown_connections 2024-03-28T05:21:22.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.087+0000 7f3a8a359640 1 --2- 172.21.15.23:0/2244499338 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a84107440 0x7f3a84109830 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.087+0000 7f3a8a359640 1 --2- 172.21.15.23:0/2244499338 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a84104b10 0x7f3a84106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.087+0000 7f3a8a359640 1 --2- 172.21.15.23:0/2244499338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a841021e0 0x7f3a841045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.087+0000 7f3a8a359640 1 -- 172.21.15.23:0/2244499338 >> 172.21.15.23:0/2244499338 conn(0x7f3a840fbb50 msgr2=0x7f3a840fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:22.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.087+0000 7f3a8a359640 1 -- 172.21.15.23:0/2244499338 shutdown_connections 2024-03-28T05:21:22.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.087+0000 7f3a8a359640 1 -- 172.21.15.23:0/2244499338 wait complete. 2024-03-28T05:21:22.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.088+0000 7f3a8a359640 1 Processor -- start 2024-03-28T05:21:22.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.088+0000 7f3a8a359640 1 -- start start 2024-03-28T05:21:22.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.088+0000 7f3a8a359640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a841021e0 0x7f3a841979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:22.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.088+0000 7f3a8a359640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a84104b10 0x7f3a84197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:22.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.088+0000 7f3a8a359640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a84107440 0x7f3a8419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:22.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.088+0000 7f3a8a359640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8410d360 con 0x7f3a84104b10 2024-03-28T05:21:22.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.088+0000 7f3a8a359640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8410d1e0 con 0x7f3a84107440 2024-03-28T05:21:22.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.088+0000 7f3a8a359640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8410d4e0 con 0x7f3a841021e0 2024-03-28T05:21:22.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.088+0000 7f3a837fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a84104b10 0x7f3a84197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:22.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.088+0000 7f3a83fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a841021e0 0x7f3a841979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:22.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.088+0000 7f3a837fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a84104b10 0x7f3a84197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57788/0 (socket says 172.21.15.23:57788) 2024-03-28T05:21:22.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.088+0000 7f3a837fe640 1 -- 172.21.15.23:0/261883391 learned_addr learned my addr 172.21.15.23:0/261883391 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:22.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.089+0000 7f3a888f6640 1 --2- 172.21.15.23:0/261883391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a84107440 0x7f3a8419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:22.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.089+0000 7f3a837fe640 1 -- 172.21.15.23:0/261883391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a841021e0 msgr2=0x7f3a841979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:22.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.089+0000 7f3a837fe640 1 --2- 172.21.15.23:0/261883391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a841021e0 0x7f3a841979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.089+0000 7f3a837fe640 1 -- 172.21.15.23:0/261883391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a84107440 msgr2=0x7f3a8419ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:22.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.089+0000 7f3a837fe640 1 --2- 172.21.15.23:0/261883391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a84107440 0x7f3a8419ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.089+0000 7f3a837fe640 1 -- 172.21.15.23:0/261883391 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8419f5d0 con 0x7f3a84104b10 2024-03-28T05:21:22.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.089+0000 7f3a83fff640 1 --2- 172.21.15.23:0/261883391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a841021e0 0x7f3a841979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:21:22.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.089+0000 7f3a837fe640 1 --2- 172.21.15.23:0/261883391 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a84104b10 0x7f3a84197ef0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00e9f0 tx=0x7f3a6c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:22.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.089+0000 7f3a817fa640 1 -- 172.21.15.23:0/261883391 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c00ccc0 con 0x7f3a84104b10 2024-03-28T05:21:22.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.089+0000 7f3a8a359640 1 -- 172.21.15.23:0/261883391 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a8419f8c0 con 0x7f3a84104b10 2024-03-28T05:21:22.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.089+0000 7f3a817fa640 1 -- 172.21.15.23:0/261883391 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a6c005500 con 0x7f3a84104b10 2024-03-28T05:21:22.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.089+0000 7f3a817fa640 1 -- 172.21.15.23:0/261883391 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c010690 con 0x7f3a84104b10 2024-03-28T05:21:22.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.089+0000 7f3a8a359640 1 -- 172.21.15.23:0/261883391 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a8419fdc0 con 0x7f3a84104b10 2024-03-28T05:21:22.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.090+0000 7f3a8a359640 1 -- 172.21.15.23:0/261883391 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a840fd640 con 0x7f3a84104b10 2024-03-28T05:21:22.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.091+0000 7f3a817fa640 1 -- 172.21.15.23:0/261883391 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3a6c0046c0 con 0x7f3a84104b10 2024-03-28T05:21:22.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.092+0000 7f3a817fa640 1 --2- 172.21.15.23:0/261883391 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3a700a2800 0x7f3a700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:22.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.092+0000 7f3a83fff640 1 --2- 172.21.15.23:0/261883391 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3a700a2800 0x7f3a700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:22.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.092+0000 7f3a817fa640 1 -- 172.21.15.23:0/261883391 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f3a6c0cf0b0 con 0x7f3a84104b10 2024-03-28T05:21:22.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.092+0000 7f3a83fff640 1 --2- 172.21.15.23:0/261883391 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3a700a2800 0x7f3a700a4cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400bfd0 tx=0x7f3a7400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:22.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.096+0000 7f3a817fa640 1 -- 172.21.15.23:0/261883391 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3a6c09bf10 con 0x7f3a84104b10 2024-03-28T05:21:22.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.279+0000 7f3a8a359640 1 -- 172.21.15.23:0/261883391 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3a840622a0 con 0x7f3a84104b10 2024-03-28T05:21:22.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.279+0000 7f3a817fa640 1 -- 172.21.15.23:0/261883391 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3a6c09b810 con 0x7f3a84104b10 2024-03-28T05:21:22.278 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-28T05:21:22.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.283+0000 7f3a8a359640 1 -- 172.21.15.23:0/261883391 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3a700a2800 msgr2=0x7f3a700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:22.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.283+0000 7f3a8a359640 1 --2- 172.21.15.23:0/261883391 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3a700a2800 0x7f3a700a4cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400bfd0 tx=0x7f3a7400b390 comp rx=0 tx=0).stop 2024-03-28T05:21:22.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.283+0000 7f3a8a359640 1 -- 172.21.15.23:0/261883391 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a84104b10 msgr2=0x7f3a84197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:22.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.283+0000 7f3a8a359640 1 --2- 172.21.15.23:0/261883391 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a84104b10 0x7f3a84197ef0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00e9f0 tx=0x7f3a6c00eeb0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.283+0000 7f3a8a359640 1 -- 172.21.15.23:0/261883391 shutdown_connections 2024-03-28T05:21:22.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.283+0000 7f3a8a359640 1 --2- 172.21.15.23:0/261883391 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3a700a2800 0x7f3a700a4cc0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.283+0000 7f3a8a359640 1 --2- 172.21.15.23:0/261883391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a84107440 0x7f3a8419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.283+0000 7f3a8a359640 1 --2- 172.21.15.23:0/261883391 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a84104b10 0x7f3a84197ef0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.283+0000 7f3a8a359640 1 --2- 172.21.15.23:0/261883391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a841021e0 0x7f3a841979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.283+0000 7f3a8a359640 1 -- 172.21.15.23:0/261883391 >> 172.21.15.23:0/261883391 conn(0x7f3a840fbb50 msgr2=0x7f3a84102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:22.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.283+0000 7f3a8a359640 1 -- 172.21.15.23:0/261883391 shutdown_connections 2024-03-28T05:21:22.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.283+0000 7f3a8a359640 1 -- 172.21.15.23:0/261883391 wait complete. 2024-03-28T05:21:22.293 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T05:21:22.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.400+0000 7f2df6012640 1 -- 172.21.15.23:0/2205934777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2df0102140 msgr2=0x7f2df0104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:22.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.400+0000 7f2df6012640 1 --2- 172.21.15.23:0/2205934777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2df0102140 0x7f2df0104530 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2de0009900 tx=0x7f2de002bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:22.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.400+0000 7f2df6012640 1 -- 172.21.15.23:0/2205934777 shutdown_connections 2024-03-28T05:21:22.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.400+0000 7f2df6012640 1 --2- 172.21.15.23:0/2205934777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2df01073a0 0x7f2df01097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.400+0000 7f2df6012640 1 --2- 172.21.15.23:0/2205934777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df0104a70 0x7f2df0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.400+0000 7f2df6012640 1 --2- 172.21.15.23:0/2205934777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2df0102140 0x7f2df0104530 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.400+0000 7f2df6012640 1 -- 172.21.15.23:0/2205934777 >> 172.21.15.23:0/2205934777 conn(0x7f2df00fbb50 msgr2=0x7f2df00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:22.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.400+0000 7f2df6012640 1 -- 172.21.15.23:0/2205934777 shutdown_connections 2024-03-28T05:21:22.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.400+0000 7f2df6012640 1 -- 172.21.15.23:0/2205934777 wait complete. 2024-03-28T05:21:22.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.400+0000 7f2df6012640 1 Processor -- start 2024-03-28T05:21:22.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.401+0000 7f2df6012640 1 -- start start 2024-03-28T05:21:22.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.401+0000 7f2df6012640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df0102140 0x7f2df0197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:22.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.401+0000 7f2df6012640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2df0104a70 0x7f2df0197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:22.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.401+0000 7f2df6012640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2df01073a0 0x7f2df019ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:22.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.401+0000 7f2df6012640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df010d370 con 0x7f2df0104a70 2024-03-28T05:21:22.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.401+0000 7f2df6012640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df010d1f0 con 0x7f2df0102140 2024-03-28T05:21:22.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.401+0000 7f2df6012640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df010d4f0 con 0x7f2df01073a0 2024-03-28T05:21:22.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.401+0000 7f2deeffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2df0104a70 0x7f2df0197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:22.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.401+0000 7f2deeffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2df0104a70 0x7f2df0197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57802/0 (socket says 172.21.15.23:57802) 2024-03-28T05:21:22.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.401+0000 7f2deeffd640 1 -- 172.21.15.23:0/821803002 learned_addr learned my addr 172.21.15.23:0/821803002 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:22.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.401+0000 7f2def7fe640 1 --2- 172.21.15.23:0/821803002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df0102140 0x7f2df0197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:22.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.401+0000 7f2deffff640 1 --2- 172.21.15.23:0/821803002 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2df01073a0 0x7f2df019ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:22.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.401+0000 7f2deeffd640 1 -- 172.21.15.23:0/821803002 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2df01073a0 msgr2=0x7f2df019ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:22.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.401+0000 7f2deeffd640 1 --2- 172.21.15.23:0/821803002 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2df01073a0 0x7f2df019ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.401+0000 7f2deeffd640 1 -- 172.21.15.23:0/821803002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df0102140 msgr2=0x7f2df0197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:22.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.401+0000 7f2deeffd640 1 --2- 172.21.15.23:0/821803002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df0102140 0x7f2df0197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.401+0000 7f2deeffd640 1 -- 172.21.15.23:0/821803002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df019f5b0 con 0x7f2df0104a70 2024-03-28T05:21:22.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.401+0000 7f2deffff640 1 --2- 172.21.15.23:0/821803002 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2df01073a0 0x7f2df019ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:21:22.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.401+0000 7f2deeffd640 1 --2- 172.21.15.23:0/821803002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2df0104a70 0x7f2df0197ed0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2de400c970 tx=0x7f2de400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:22.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.401+0000 7f2decff9640 1 -- 172.21.15.23:0/821803002 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de4014070 con 0x7f2df0104a70 2024-03-28T05:21:22.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.402+0000 7f2df6012640 1 -- 172.21.15.23:0/821803002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df019f8a0 con 0x7f2df0104a70 2024-03-28T05:21:22.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.402+0000 7f2decff9640 1 -- 172.21.15.23:0/821803002 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2de4004440 con 0x7f2df0104a70 2024-03-28T05:21:22.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.402+0000 7f2decff9640 1 -- 172.21.15.23:0/821803002 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de400f450 con 0x7f2df0104a70 2024-03-28T05:21:22.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.402+0000 7f2df6012640 1 -- 172.21.15.23:0/821803002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df019fdf0 con 0x7f2df0104a70 2024-03-28T05:21:22.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.403+0000 7f2df6012640 1 -- 172.21.15.23:0/821803002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2df00fd640 con 0x7f2df0104a70 2024-03-28T05:21:22.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.404+0000 7f2decff9640 1 -- 172.21.15.23:0/821803002 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2de4017070 con 0x7f2df0104a70 2024-03-28T05:21:22.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.404+0000 7f2decff9640 1 --2- 172.21.15.23:0/821803002 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2dd80a2800 0x7f2dd80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:22.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.404+0000 7f2def7fe640 1 --2- 172.21.15.23:0/821803002 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2dd80a2800 0x7f2dd80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:22.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.404+0000 7f2decff9640 1 -- 172.21.15.23:0/821803002 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f2de40cea40 con 0x7f2df0104a70 2024-03-28T05:21:22.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.405+0000 7f2def7fe640 1 --2- 172.21.15.23:0/821803002 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2dd80a2800 0x7f2dd80a4cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2de002c7b0 tx=0x7f2de0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:22.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.409+0000 7f2decff9640 1 -- 172.21.15.23:0/821803002 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2de401e020 con 0x7f2df0104a70 2024-03-28T05:21:22.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.591+0000 7f2df6012640 1 -- 172.21.15.23:0/821803002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2df00622a0 con 0x7f2df0104a70 2024-03-28T05:21:22.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.591+0000 7f2decff9640 1 -- 172.21.15.23:0/821803002 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2de409b8a0 con 0x7f2df0104a70 2024-03-28T05:21:22.590 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T05:21:22.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.593+0000 7f2df6012640 1 -- 172.21.15.23:0/821803002 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2dd80a2800 msgr2=0x7f2dd80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:22.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.593+0000 7f2df6012640 1 --2- 172.21.15.23:0/821803002 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2dd80a2800 0x7f2dd80a4cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2de002c7b0 tx=0x7f2de0005fd0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.593+0000 7f2df6012640 1 -- 172.21.15.23:0/821803002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2df0104a70 msgr2=0x7f2df0197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:22.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.593+0000 7f2df6012640 1 --2- 172.21.15.23:0/821803002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2df0104a70 0x7f2df0197ed0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2de400c970 tx=0x7f2de400ce30 comp rx=0 tx=0).stop 2024-03-28T05:21:22.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.593+0000 7f2df6012640 1 -- 172.21.15.23:0/821803002 shutdown_connections 2024-03-28T05:21:22.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.593+0000 7f2df6012640 1 --2- 172.21.15.23:0/821803002 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2dd80a2800 0x7f2dd80a4cc0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.593+0000 7f2df6012640 1 --2- 172.21.15.23:0/821803002 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2df01073a0 0x7f2df019ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.593+0000 7f2df6012640 1 --2- 172.21.15.23:0/821803002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2df0104a70 0x7f2df0197ed0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.593+0000 7f2df6012640 1 --2- 172.21.15.23:0/821803002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2df0102140 0x7f2df0197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.593+0000 7f2df6012640 1 -- 172.21.15.23:0/821803002 >> 172.21.15.23:0/821803002 conn(0x7f2df00fbb50 msgr2=0x7f2df0102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:22.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.593+0000 7f2df6012640 1 -- 172.21.15.23:0/821803002 shutdown_connections 2024-03-28T05:21:22.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.593+0000 7f2df6012640 1 -- 172.21.15.23:0/821803002 wait complete. 2024-03-28T05:21:22.602 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:22 smithi023 ceph-mon[31954]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 484 B/s wr, 3 op/s 2024-03-28T05:21:22.602 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:22 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3909410030' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T05:21:22.602 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:22 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/4095326660' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:22.602 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:22 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/261883391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:21:22.603 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T05:21:22.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:22 smithi098 ceph-mon[36087]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 484 B/s wr, 3 op/s 2024-03-28T05:21:22.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:22 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3909410030' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T05:21:22.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:22 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/4095326660' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:22.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:22 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/261883391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:21:22.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.709+0000 7f361390a640 1 -- 172.21.15.23:0/374365040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f360c1016c0 msgr2=0x7f360c101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:22.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.709+0000 7f361390a640 1 --2- 172.21.15.23:0/374365040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f360c1016c0 0x7f360c101ae0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3608009900 tx=0x7f360802bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:22.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.709+0000 7f361390a640 1 -- 172.21.15.23:0/374365040 shutdown_connections 2024-03-28T05:21:22.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.709+0000 7f361390a640 1 --2- 172.21.15.23:0/374365040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f360c103270 0x7f360c109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.709+0000 7f361390a640 1 --2- 172.21.15.23:0/374365040 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f360c1028b0 0x7f360c102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.709+0000 7f361390a640 1 --2- 172.21.15.23:0/374365040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f360c1016c0 0x7f360c101ae0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.709+0000 7f361390a640 1 -- 172.21.15.23:0/374365040 >> 172.21.15.23:0/374365040 conn(0x7f360c0fcc30 msgr2=0x7f360c0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:22.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.709+0000 7f361390a640 1 -- 172.21.15.23:0/374365040 shutdown_connections 2024-03-28T05:21:22.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.709+0000 7f361390a640 1 -- 172.21.15.23:0/374365040 wait complete. 2024-03-28T05:21:22.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.710+0000 7f361390a640 1 Processor -- start 2024-03-28T05:21:22.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.710+0000 7f361390a640 1 -- start start 2024-03-28T05:21:22.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.710+0000 7f361390a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f360c1016c0 0x7f360c1957b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:22.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.710+0000 7f361390a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f360c1028b0 0x7f360c195cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:22.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.710+0000 7f361390a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f360c103270 0x7f360c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:22.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.710+0000 7f361390a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c10b160 con 0x7f360c1016c0 2024-03-28T05:21:22.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.710+0000 7f361390a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c10afe0 con 0x7f360c103270 2024-03-28T05:21:22.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.710+0000 7f361390a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c10b2e0 con 0x7f360c1028b0 2024-03-28T05:21:22.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.710+0000 7f36116a6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f360c1016c0 0x7f360c1957b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:22.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.710+0000 7f36116a6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f360c1016c0 0x7f360c1957b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57830/0 (socket says 172.21.15.23:57830) 2024-03-28T05:21:22.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.710+0000 7f36116a6640 1 -- 172.21.15.23:0/4058164050 learned_addr learned my addr 172.21.15.23:0/4058164050 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:22.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.710+0000 7f3610ea5640 1 --2- 172.21.15.23:0/4058164050 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f360c1028b0 0x7f360c195cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:22.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.710+0000 7f3611ea7640 1 --2- 172.21.15.23:0/4058164050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f360c103270 0x7f360c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:22.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.710+0000 7f36116a6640 1 -- 172.21.15.23:0/4058164050 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f360c1028b0 msgr2=0x7f360c195cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:22.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.710+0000 7f36116a6640 1 --2- 172.21.15.23:0/4058164050 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f360c1028b0 0x7f360c195cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.710+0000 7f36116a6640 1 -- 172.21.15.23:0/4058164050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f360c103270 msgr2=0x7f360c19cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:22.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.710+0000 7f36116a6640 1 --2- 172.21.15.23:0/4058164050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f360c103270 0x7f360c19cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.710+0000 7f36116a6640 1 -- 172.21.15.23:0/4058164050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360c19d3d0 con 0x7f360c1016c0 2024-03-28T05:21:22.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.710+0000 7f3611ea7640 1 --2- 172.21.15.23:0/4058164050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f360c103270 0x7f360c19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:21:22.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.710+0000 7f36116a6640 1 --2- 172.21.15.23:0/4058164050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f360c1016c0 0x7f360c1957b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3608000c00 tx=0x7f3608005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:22.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.711+0000 7f36027fc640 1 -- 172.21.15.23:0/4058164050 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608041070 con 0x7f360c1016c0 2024-03-28T05:21:22.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.711+0000 7f36027fc640 1 -- 172.21.15.23:0/4058164050 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36080044d0 con 0x7f360c1016c0 2024-03-28T05:21:22.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.711+0000 7f361390a640 1 -- 172.21.15.23:0/4058164050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f360c19d660 con 0x7f360c1016c0 2024-03-28T05:21:22.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.711+0000 7f36027fc640 1 -- 172.21.15.23:0/4058164050 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608034440 con 0x7f360c1016c0 2024-03-28T05:21:22.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.711+0000 7f361390a640 1 -- 172.21.15.23:0/4058164050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f360c19dba0 con 0x7f360c1016c0 2024-03-28T05:21:22.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.712+0000 7f361390a640 1 -- 172.21.15.23:0/4058164050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f360c1089e0 con 0x7f360c1016c0 2024-03-28T05:21:22.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.713+0000 7f36027fc640 1 -- 172.21.15.23:0/4058164050 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3608044070 con 0x7f360c1016c0 2024-03-28T05:21:22.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.713+0000 7f36027fc640 1 --2- 172.21.15.23:0/4058164050 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f35f80a6c90 0x7f35f80a9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:22.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.713+0000 7f3610ea5640 1 --2- 172.21.15.23:0/4058164050 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f35f80a6c90 0x7f35f80a9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:22.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.713+0000 7f36027fc640 1 -- 172.21.15.23:0/4058164050 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f36080f09b0 con 0x7f360c1016c0 2024-03-28T05:21:22.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.714+0000 7f3610ea5640 1 --2- 172.21.15.23:0/4058164050 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f35f80a6c90 0x7f35f80a9150 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f35fc0095f0 tx=0x7f35fc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:22.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.717+0000 7f36027fc640 1 -- 172.21.15.23:0/4058164050 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f36080bd810 con 0x7f360c1016c0 2024-03-28T05:21:22.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:22 smithi113 ceph-mon[37164]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 484 B/s wr, 3 op/s 2024-03-28T05:21:22.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:22 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3909410030' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T05:21:22.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:22 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/4095326660' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:22.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:22 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/261883391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:21:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.901+0000 7f361390a640 1 -- 172.21.15.23:0/4058164050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f360c02d200 con 0x7f360c1016c0 2024-03-28T05:21:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.902+0000 7f36027fc640 1 -- 172.21.15.23:0/4058164050 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f36080bd110 con 0x7f360c1016c0 2024-03-28T05:21:22.901 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-28T05:21:22.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.904+0000 7f361390a640 1 -- 172.21.15.23:0/4058164050 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f35f80a6c90 msgr2=0x7f35f80a9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:22.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.904+0000 7f361390a640 1 --2- 172.21.15.23:0/4058164050 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f35f80a6c90 0x7f35f80a9150 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f35fc0095f0 tx=0x7f35fc005f90 comp rx=0 tx=0).stop 2024-03-28T05:21:22.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.904+0000 7f361390a640 1 -- 172.21.15.23:0/4058164050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f360c1016c0 msgr2=0x7f360c1957b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:22.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.904+0000 7f361390a640 1 --2- 172.21.15.23:0/4058164050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f360c1016c0 0x7f360c1957b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3608000c00 tx=0x7f3608005d50 comp rx=0 tx=0).stop 2024-03-28T05:21:22.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.904+0000 7f361390a640 1 -- 172.21.15.23:0/4058164050 shutdown_connections 2024-03-28T05:21:22.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.904+0000 7f361390a640 1 --2- 172.21.15.23:0/4058164050 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f35f80a6c90 0x7f35f80a9150 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.904+0000 7f361390a640 1 --2- 172.21.15.23:0/4058164050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f360c103270 0x7f360c19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.904+0000 7f361390a640 1 --2- 172.21.15.23:0/4058164050 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f360c1028b0 0x7f360c195cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.904+0000 7f361390a640 1 --2- 172.21.15.23:0/4058164050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f360c1016c0 0x7f360c1957b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:22.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.904+0000 7f361390a640 1 -- 172.21.15.23:0/4058164050 >> 172.21.15.23:0/4058164050 conn(0x7f360c0fcc30 msgr2=0x7f360c105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:22.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.904+0000 7f361390a640 1 -- 172.21.15.23:0/4058164050 shutdown_connections 2024-03-28T05:21:22.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:22.904+0000 7f361390a640 1 -- 172.21.15.23:0/4058164050 wait complete. 2024-03-28T05:21:22.914 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T05:21:23.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.020+0000 7fc912ea1640 1 -- 172.21.15.23:0/925022402 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc90c101790 msgr2=0x7fc90c101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:23.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.020+0000 7fc912ea1640 1 --2- 172.21.15.23:0/925022402 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc90c101790 0x7fc90c101c10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8009900 tx=0x7fc8f802bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:23.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.020+0000 7fc912ea1640 1 -- 172.21.15.23:0/925022402 shutdown_connections 2024-03-28T05:21:23.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.020+0000 7fc912ea1640 1 --2- 172.21.15.23:0/925022402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc90c102150 0x7fc90c1089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:23.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.020+0000 7fc912ea1640 1 --2- 172.21.15.23:0/925022402 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc90c101790 0x7fc90c101c10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:23.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.020+0000 7fc912ea1640 1 --2- 172.21.15.23:0/925022402 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc90c1005a0 0x7fc90c1009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:23.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.020+0000 7fc912ea1640 1 -- 172.21.15.23:0/925022402 >> 172.21.15.23:0/925022402 conn(0x7fc90c0fbb30 msgr2=0x7fc90c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:23.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.020+0000 7fc912ea1640 1 -- 172.21.15.23:0/925022402 shutdown_connections 2024-03-28T05:21:23.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.021+0000 7fc912ea1640 1 -- 172.21.15.23:0/925022402 wait complete. 2024-03-28T05:21:23.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.021+0000 7fc912ea1640 1 Processor -- start 2024-03-28T05:21:23.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.021+0000 7fc912ea1640 1 -- start start 2024-03-28T05:21:23.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.021+0000 7fc912ea1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc90c1005a0 0x7fc90c069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:23.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.021+0000 7fc912ea1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc90c101790 0x7fc90c19c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:23.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.021+0000 7fc912ea1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc90c102150 0x7fc90c19ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:23.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.021+0000 7fc912ea1640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c10c230 con 0x7fc90c102150 2024-03-28T05:21:23.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.021+0000 7fc912ea1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c10c0b0 con 0x7fc90c1005a0 2024-03-28T05:21:23.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.021+0000 7fc912ea1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c10c3b0 con 0x7fc90c101790 2024-03-28T05:21:23.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.021+0000 7fc91143e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc90c102150 0x7fc90c19ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:23.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.021+0000 7fc91143e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc90c102150 0x7fc90c19ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44380/0 (socket says 172.21.15.23:44380) 2024-03-28T05:21:23.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.021+0000 7fc91143e640 1 -- 172.21.15.23:0/3413689198 learned_addr learned my addr 172.21.15.23:0/3413689198 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:23.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.021+0000 7fc90bfff640 1 --2- 172.21.15.23:0/3413689198 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc90c101790 0x7fc90c19c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:23.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.021+0000 7fc910c3d640 1 --2- 172.21.15.23:0/3413689198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc90c1005a0 0x7fc90c069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:23.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.022+0000 7fc91143e640 1 -- 172.21.15.23:0/3413689198 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc90c101790 msgr2=0x7fc90c19c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:23.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.022+0000 7fc91143e640 1 --2- 172.21.15.23:0/3413689198 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc90c101790 0x7fc90c19c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:23.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.022+0000 7fc91143e640 1 -- 172.21.15.23:0/3413689198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc90c1005a0 msgr2=0x7fc90c069390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:23.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.022+0000 7fc91143e640 1 --2- 172.21.15.23:0/3413689198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc90c1005a0 0x7fc90c069390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:23.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.022+0000 7fc91143e640 1 -- 172.21.15.23:0/3413689198 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90c19f570 con 0x7fc90c102150 2024-03-28T05:21:23.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.022+0000 7fc91143e640 1 --2- 172.21.15.23:0/3413689198 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc90c102150 0x7fc90c19ecb0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc90400efd0 tx=0x7fc90400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:23.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.022+0000 7fc909ffb640 1 -- 172.21.15.23:0/3413689198 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc904003b70 con 0x7fc90c102150 2024-03-28T05:21:23.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.022+0000 7fc912ea1640 1 -- 172.21.15.23:0/3413689198 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc90c19f860 con 0x7fc90c102150 2024-03-28T05:21:23.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.022+0000 7fc909ffb640 1 -- 172.21.15.23:0/3413689198 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc904003d10 con 0x7fc90c102150 2024-03-28T05:21:23.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.022+0000 7fc909ffb640 1 -- 172.21.15.23:0/3413689198 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9040109f0 con 0x7fc90c102150 2024-03-28T05:21:23.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.022+0000 7fc912ea1640 1 -- 172.21.15.23:0/3413689198 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc90c19fd30 con 0x7fc90c102150 2024-03-28T05:21:23.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.023+0000 7fc912ea1640 1 -- 172.21.15.23:0/3413689198 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc90c1078c0 con 0x7fc90c102150 2024-03-28T05:21:23.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.024+0000 7fc909ffb640 1 -- 172.21.15.23:0/3413689198 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc904004020 con 0x7fc90c102150 2024-03-28T05:21:23.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.025+0000 7fc909ffb640 1 --2- 172.21.15.23:0/3413689198 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc8fc0a2800 0x7fc8fc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:23.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.025+0000 7fc910c3d640 1 --2- 172.21.15.23:0/3413689198 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc8fc0a2800 0x7fc8fc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:23.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.025+0000 7fc909ffb640 1 -- 172.21.15.23:0/3413689198 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc9040cf110 con 0x7fc90c102150 2024-03-28T05:21:23.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.025+0000 7fc910c3d640 1 --2- 172.21.15.23:0/3413689198 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc8fc0a2800 0x7fc8fc0a4cc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc90c1015f0 tx=0x7fc900005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:23.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.029+0000 7fc909ffb640 1 -- 172.21.15.23:0/3413689198 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc90409bf70 con 0x7fc90c102150 2024-03-28T05:21:23.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.213+0000 7fc912ea1640 1 -- 172.21.15.23:0/3413689198 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc90c0622a0 con 0x7fc90c102150 2024-03-28T05:21:23.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.214+0000 7fc909ffb640 1 -- 172.21.15.23:0/3413689198 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc90409b870 con 0x7fc90c102150 2024-03-28T05:21:23.213 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T05:21:23.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.216+0000 7fc912ea1640 1 -- 172.21.15.23:0/3413689198 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc8fc0a2800 msgr2=0x7fc8fc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:23.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.216+0000 7fc912ea1640 1 --2- 172.21.15.23:0/3413689198 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc8fc0a2800 0x7fc8fc0a4cc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc90c1015f0 tx=0x7fc900005f90 comp rx=0 tx=0).stop 2024-03-28T05:21:23.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.216+0000 7fc912ea1640 1 -- 172.21.15.23:0/3413689198 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc90c102150 msgr2=0x7fc90c19ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:23.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.216+0000 7fc912ea1640 1 --2- 172.21.15.23:0/3413689198 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc90c102150 0x7fc90c19ecb0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc90400efd0 tx=0x7fc90400c6a0 comp rx=0 tx=0).stop 2024-03-28T05:21:23.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.216+0000 7fc912ea1640 1 -- 172.21.15.23:0/3413689198 shutdown_connections 2024-03-28T05:21:23.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.216+0000 7fc912ea1640 1 --2- 172.21.15.23:0/3413689198 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc8fc0a2800 0x7fc8fc0a4cc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:23.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.216+0000 7fc912ea1640 1 --2- 172.21.15.23:0/3413689198 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc90c102150 0x7fc90c19ecb0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:23.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.216+0000 7fc912ea1640 1 --2- 172.21.15.23:0/3413689198 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc90c101790 0x7fc90c19c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:23.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.216+0000 7fc912ea1640 1 --2- 172.21.15.23:0/3413689198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc90c1005a0 0x7fc90c069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:23.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.216+0000 7fc912ea1640 1 -- 172.21.15.23:0/3413689198 >> 172.21.15.23:0/3413689198 conn(0x7fc90c0fbb30 msgr2=0x7fc90c104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:23.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.216+0000 7fc912ea1640 1 -- 172.21.15.23:0/3413689198 shutdown_connections 2024-03-28T05:21:23.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.216+0000 7fc912ea1640 1 -- 172.21.15.23:0/3413689198 wait complete. 2024-03-28T05:21:23.226 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T05:21:23.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.332+0000 7f4f7a148640 1 -- 172.21.15.23:0/124956022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f74101290 msgr2=0x7f4f74101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:23.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.332+0000 7f4f7a148640 1 --2- 172.21.15.23:0/124956022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f74101290 0x7f4f74101710 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c009ca0 tx=0x7f4f6c02c180 comp rx=0 tx=0).stop 2024-03-28T05:21:23.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.332+0000 7f4f7a148640 1 -- 172.21.15.23:0/124956022 shutdown_connections 2024-03-28T05:21:23.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.332+0000 7f4f7a148640 1 --2- 172.21.15.23:0/124956022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f74101290 0x7f4f74101710 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:23.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.332+0000 7f4f7a148640 1 --2- 172.21.15.23:0/124956022 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f741008f0 0x7f4f74100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:23.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.332+0000 7f4f7a148640 1 --2- 172.21.15.23:0/124956022 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f740fff90 0x7f4f741003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:23.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.332+0000 7f4f7a148640 1 -- 172.21.15.23:0/124956022 >> 172.21.15.23:0/124956022 conn(0x7f4f740fbb30 msgr2=0x7f4f740fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:23.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.332+0000 7f4f7a148640 1 -- 172.21.15.23:0/124956022 shutdown_connections 2024-03-28T05:21:23.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.332+0000 7f4f7a148640 1 -- 172.21.15.23:0/124956022 wait complete. 2024-03-28T05:21:23.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.333+0000 7f4f7a148640 1 Processor -- start 2024-03-28T05:21:23.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.333+0000 7f4f7a148640 1 -- start start 2024-03-28T05:21:23.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.333+0000 7f4f7a148640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f740fff90 0x7f4f74199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:23.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.333+0000 7f4f7a148640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f741008f0 0x7f4f7419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:23.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.333+0000 7f4f7a148640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f74101290 0x7f4f741a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:23.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.333+0000 7f4f7a148640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7410f520 con 0x7f4f74101290 2024-03-28T05:21:23.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.333+0000 7f4f7a148640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7410f3a0 con 0x7f4f741008f0 2024-03-28T05:21:23.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.333+0000 7f4f7a148640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7410f6a0 con 0x7f4f740fff90 2024-03-28T05:21:23.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.333+0000 7f4f73fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f74101290 0x7f4f741a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:23.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.333+0000 7f4f73fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f74101290 0x7f4f741a1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44402/0 (socket says 172.21.15.23:44402) 2024-03-28T05:21:23.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.333+0000 7f4f73fff640 1 -- 172.21.15.23:0/1615571324 learned_addr learned my addr 172.21.15.23:0/1615571324 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:23.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.333+0000 7f4f737fe640 1 --2- 172.21.15.23:0/1615571324 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f740fff90 0x7f4f74199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:23.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.333+0000 7f4f72ffd640 1 --2- 172.21.15.23:0/1615571324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f741008f0 0x7f4f7419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:23.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.333+0000 7f4f73fff640 1 -- 172.21.15.23:0/1615571324 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f740fff90 msgr2=0x7f4f74199b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:23.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.333+0000 7f4f73fff640 1 --2- 172.21.15.23:0/1615571324 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f740fff90 0x7f4f74199b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:23.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.333+0000 7f4f73fff640 1 -- 172.21.15.23:0/1615571324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f741008f0 msgr2=0x7f4f7419a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:23.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.333+0000 7f4f73fff640 1 --2- 172.21.15.23:0/1615571324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f741008f0 0x7f4f7419a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:23.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.333+0000 7f4f73fff640 1 -- 172.21.15.23:0/1615571324 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f741a17b0 con 0x7f4f74101290 2024-03-28T05:21:23.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.333+0000 7f4f72ffd640 1 --2- 172.21.15.23:0/1615571324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f741008f0 0x7f4f7419a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:21:23.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.334+0000 7f4f73fff640 1 --2- 172.21.15.23:0/1615571324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f74101290 0x7f4f741a1140 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c037040 tx=0x7f4f6c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:23.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.334+0000 7f4f70ff9640 1 -- 172.21.15.23:0/1615571324 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c039040 con 0x7f4f74101290 2024-03-28T05:21:23.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.334+0000 7f4f70ff9640 1 -- 172.21.15.23:0/1615571324 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f6c044070 con 0x7f4f74101290 2024-03-28T05:21:23.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.334+0000 7f4f70ff9640 1 -- 172.21.15.23:0/1615571324 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c004330 con 0x7f4f74101290 2024-03-28T05:21:23.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.334+0000 7f4f7a148640 1 -- 172.21.15.23:0/1615571324 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f741a1a40 con 0x7f4f74101290 2024-03-28T05:21:23.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.334+0000 7f4f7a148640 1 -- 172.21.15.23:0/1615571324 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f741a1ee0 con 0x7f4f74101290 2024-03-28T05:21:23.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.335+0000 7f4f7a148640 1 -- 172.21.15.23:0/1615571324 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f7410ae70 con 0x7f4f74101290 2024-03-28T05:21:23.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.336+0000 7f4f70ff9640 1 -- 172.21.15.23:0/1615571324 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4f6c004510 con 0x7f4f74101290 2024-03-28T05:21:23.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.336+0000 7f4f70ff9640 1 --2- 172.21.15.23:0/1615571324 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f4f5c0a2800 0x7f4f5c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:23.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.337+0000 7f4f737fe640 1 --2- 172.21.15.23:0/1615571324 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f4f5c0a2800 0x7f4f5c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:23.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.337+0000 7f4f70ff9640 1 -- 172.21.15.23:0/1615571324 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f4f6c0f1390 con 0x7f4f74101290 2024-03-28T05:21:23.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.337+0000 7f4f737fe640 1 --2- 172.21.15.23:0/1615571324 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f4f5c0a2800 0x7f4f5c0a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4f68009bd0 tx=0x7f4f68005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:23.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.343+0000 7f4f70ff9640 1 -- 172.21.15.23:0/1615571324 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4f6c0be1f0 con 0x7f4f74101290 2024-03-28T05:21:23.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:23 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/821803002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:21:23.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:23 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/4058164050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:21:23.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:23 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3413689198' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:21:23.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.540+0000 7f4f7a148640 1 -- 172.21.15.23:0/1615571324 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4f7402d070 con 0x7f4f74101290 2024-03-28T05:21:23.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.541+0000 7f4f70ff9640 1 -- 172.21.15.23:0/1615571324 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+592 (secure 0 0 0) 0x7f4f6c0bdaf0 con 0x7f4f74101290 2024-03-28T05:21:23.540 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/1530849123 2024-03-29T05:14:08.640340+0000 2024-03-28T05:21:23.540 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/1530849123 2024-03-29T05:14:08.640340+0000 2024-03-28T05:21:23.540 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1876451381 2024-03-29T05:14:08.640340+0000 2024-03-28T05:21:23.540 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2253559862 2024-03-29T05:14:08.640340+0000 2024-03-28T05:21:23.540 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2653562762 2024-03-29T05:13:42.579607+0000 2024-03-28T05:21:23.540 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/3337463830 2024-03-29T05:13:42.579607+0000 2024-03-28T05:21:23.540 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/1735221490 2024-03-29T05:13:42.579607+0000 2024-03-28T05:21:23.540 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2784771635 2024-03-29T05:14:08.640340+0000 2024-03-28T05:21:23.540 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2389902490 2024-03-29T05:13:42.579607+0000 2024-03-28T05:21:23.540 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/1735221490 2024-03-29T05:13:42.579607+0000 2024-03-28T05:21:23.541 INFO:teuthology.orchestra.run.smithi023.stderr:listed 10 entries 2024-03-28T05:21:23.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.542+0000 7f4f7a148640 1 -- 172.21.15.23:0/1615571324 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f4f5c0a2800 msgr2=0x7f4f5c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:23.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.542+0000 7f4f7a148640 1 --2- 172.21.15.23:0/1615571324 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f4f5c0a2800 0x7f4f5c0a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4f68009bd0 tx=0x7f4f68005f90 comp rx=0 tx=0).stop 2024-03-28T05:21:23.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.543+0000 7f4f7a148640 1 -- 172.21.15.23:0/1615571324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f74101290 msgr2=0x7f4f741a1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:23.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.543+0000 7f4f7a148640 1 --2- 172.21.15.23:0/1615571324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f74101290 0x7f4f741a1140 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c037040 tx=0x7f4f6c003c20 comp rx=0 tx=0).stop 2024-03-28T05:21:23.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.543+0000 7f4f7a148640 1 -- 172.21.15.23:0/1615571324 shutdown_connections 2024-03-28T05:21:23.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.543+0000 7f4f7a148640 1 --2- 172.21.15.23:0/1615571324 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f4f5c0a2800 0x7f4f5c0a4cc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:23.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.543+0000 7f4f7a148640 1 --2- 172.21.15.23:0/1615571324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f74101290 0x7f4f741a1140 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:23.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.543+0000 7f4f7a148640 1 --2- 172.21.15.23:0/1615571324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f741008f0 0x7f4f7419a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:23.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.543+0000 7f4f7a148640 1 --2- 172.21.15.23:0/1615571324 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f740fff90 0x7f4f74199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:23.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.543+0000 7f4f7a148640 1 -- 172.21.15.23:0/1615571324 >> 172.21.15.23:0/1615571324 conn(0x7f4f740fbb30 msgr2=0x7f4f740fd230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:23.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.543+0000 7f4f7a148640 1 -- 172.21.15.23:0/1615571324 shutdown_connections 2024-03-28T05:21:23.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:23.543+0000 7f4f7a148640 1 -- 172.21.15.23:0/1615571324 wait complete. 2024-03-28T05:21:23.552 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi023.front.sepia.ceph.com... 2024-03-28T05:21:23.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:21:23.553 DEBUG:teuthology.orchestra.run.smithi023:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-28T05:21:23.578 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:ceph-fuse[75997]: fuse finished with error 0 and tester_r 0 2024-03-28T05:21:23.580 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T05:21:23.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:23 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/821803002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:21:23.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:23 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/4058164050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:21:23.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:23 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3413689198' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:21:23.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:23 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/821803002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:21:23.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:23 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/4058164050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:21:23.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:23 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3413689198' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:21:24.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:24 smithi098 ceph-mon[36087]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s wr, 0 op/s 2024-03-28T05:21:24.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:24 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1615571324' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T05:21:24.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:24 smithi113 ceph-mon[37164]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s wr, 0 op/s 2024-03-28T05:21:24.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:24 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1615571324' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T05:21:24.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:24 smithi023 ceph-mon[31954]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s wr, 0 op/s 2024-03-28T05:21:24.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:24 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1615571324' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T05:21:26.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:26 smithi098 ceph-mon[36087]: pgmap v297: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 447 B/s wr, 0 op/s 2024-03-28T05:21:26.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:26 smithi113 ceph-mon[37164]: pgmap v297: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 447 B/s wr, 0 op/s 2024-03-28T05:21:26.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:26 smithi023 ceph-mon[31954]: pgmap v297: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 447 B/s wr, 0 op/s 2024-03-28T05:21:28.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:28 smithi023 ceph-mon[31954]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T05:21:28.614 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:28 smithi098 ceph-mon[36087]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T05:21:28.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:28 smithi113 ceph-mon[37164]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T05:21:29.682 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi023.front.sepia.ceph.com 2024-03-28T05:21:29.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:21:29.682 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-28T05:21:29.710 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T05:21:29.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.864+0000 7fcf1cf5b640 1 -- 172.21.15.23:0/236437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf18101830 msgr2=0x7fcf18101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:29.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.864+0000 7fcf1cf5b640 1 --2- 172.21.15.23:0/236437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf18101830 0x7fcf18101cb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fcf00009900 tx=0x7fcf0002bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:29.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.864+0000 7fcf1cf5b640 1 -- 172.21.15.23:0/236437 shutdown_connections 2024-03-28T05:21:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.864+0000 7fcf1cf5b640 1 --2- 172.21.15.23:0/236437 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf181021f0 0x7fcf18108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.864+0000 7fcf1cf5b640 1 --2- 172.21.15.23:0/236437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf18101830 0x7fcf18101cb0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.864+0000 7fcf1cf5b640 1 --2- 172.21.15.23:0/236437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf18100640 0x7fcf18100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.864+0000 7fcf1cf5b640 1 -- 172.21.15.23:0/236437 >> 172.21.15.23:0/236437 conn(0x7fcf180fbbd0 msgr2=0x7fcf180fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.864+0000 7fcf1cf5b640 1 -- 172.21.15.23:0/236437 shutdown_connections 2024-03-28T05:21:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.864+0000 7fcf1cf5b640 1 -- 172.21.15.23:0/236437 wait complete. 2024-03-28T05:21:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.865+0000 7fcf1cf5b640 1 Processor -- start 2024-03-28T05:21:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.865+0000 7fcf1cf5b640 1 -- start start 2024-03-28T05:21:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.865+0000 7fcf1cf5b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf18100640 0x7fcf181979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.865+0000 7fcf1cf5b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf18101830 0x7fcf18197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.865+0000 7fcf1cf5b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf181021f0 0x7fcf1819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.865+0000 7fcf1cf5b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1810c250 con 0x7fcf181021f0 2024-03-28T05:21:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.865+0000 7fcf1cf5b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1810c0d0 con 0x7fcf18100640 2024-03-28T05:21:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.865+0000 7fcf1cf5b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1810c3d0 con 0x7fcf18101830 2024-03-28T05:21:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.865+0000 7fcf16d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf181021f0 0x7fcf1819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.866+0000 7fcf16d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf181021f0 0x7fcf1819ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44416/0 (socket says 172.21.15.23:44416) 2024-03-28T05:21:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.866+0000 7fcf0ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf18101830 0x7fcf18197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.866+0000 7fcf16d9d640 1 -- 172.21.15.23:0/3172025155 learned_addr learned my addr 172.21.15.23:0/3172025155 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.866+0000 7fcf1659c640 1 --2- 172.21.15.23:0/3172025155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf18100640 0x7fcf181979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.866+0000 7fcf16d9d640 1 -- 172.21.15.23:0/3172025155 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf18101830 msgr2=0x7fcf18197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.866+0000 7fcf16d9d640 1 --2- 172.21.15.23:0/3172025155 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf18101830 0x7fcf18197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.866+0000 7fcf16d9d640 1 -- 172.21.15.23:0/3172025155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf18100640 msgr2=0x7fcf181979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.866+0000 7fcf16d9d640 1 --2- 172.21.15.23:0/3172025155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf18100640 0x7fcf181979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:29.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.866+0000 7fcf16d9d640 1 -- 172.21.15.23:0/3172025155 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1819f5d0 con 0x7fcf181021f0 2024-03-28T05:21:29.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.866+0000 7fcf1659c640 1 --2- 172.21.15.23:0/3172025155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf18100640 0x7fcf181979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:21:29.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.866+0000 7fcf0ffff640 1 --2- 172.21.15.23:0/3172025155 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf18101830 0x7fcf18197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:21:29.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.866+0000 7fcf16d9d640 1 --2- 172.21.15.23:0/3172025155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf181021f0 0x7fcf1819ef60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcf1000eb10 tx=0x7fcf1000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:29.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.866+0000 7fcf0f7fe640 1 -- 172.21.15.23:0/3172025155 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf10004b20 con 0x7fcf181021f0 2024-03-28T05:21:29.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.866+0000 7fcf0f7fe640 1 -- 172.21.15.23:0/3172025155 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf10004cc0 con 0x7fcf181021f0 2024-03-28T05:21:29.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.866+0000 7fcf1cf5b640 1 -- 172.21.15.23:0/3172025155 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf1819f8c0 con 0x7fcf181021f0 2024-03-28T05:21:29.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.866+0000 7fcf0f7fe640 1 -- 172.21.15.23:0/3172025155 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf10010760 con 0x7fcf181021f0 2024-03-28T05:21:29.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.866+0000 7fcf1cf5b640 1 -- 172.21.15.23:0/3172025155 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1819fde0 con 0x7fcf181021f0 2024-03-28T05:21:29.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.868+0000 7fcf1cf5b640 1 -- 172.21.15.23:0/3172025155 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf180688e0 con 0x7fcf181021f0 2024-03-28T05:21:29.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.869+0000 7fcf0f7fe640 1 -- 172.21.15.23:0/3172025155 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcf10005020 con 0x7fcf181021f0 2024-03-28T05:21:29.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.869+0000 7fcf0f7fe640 1 --2- 172.21.15.23:0/3172025155 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fcf040a2800 0x7fcf040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:29.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.869+0000 7fcf1659c640 1 --2- 172.21.15.23:0/3172025155 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fcf040a2800 0x7fcf040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:29.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.869+0000 7fcf0f7fe640 1 -- 172.21.15.23:0/3172025155 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fcf100cf460 con 0x7fcf181021f0 2024-03-28T05:21:29.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.870+0000 7fcf1659c640 1 --2- 172.21.15.23:0/3172025155 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fcf040a2800 0x7fcf040a4cc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcf080058c0 tx=0x7fcf08005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:29.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:29.874+0000 7fcf0f7fe640 1 -- 172.21.15.23:0/3172025155 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcf1009c2c0 con 0x7fcf181021f0 2024-03-28T05:21:30.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.057+0000 7fcf1cf5b640 1 -- 172.21.15.23:0/3172025155 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf181a0760 con 0x7fcf181021f0 2024-03-28T05:21:30.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.058+0000 7fcf0f7fe640 1 -- 172.21.15.23:0/3172025155 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24082 (secure 0 0 0) 0x7fcf1009bbc0 con 0x7fcf181021f0 2024-03-28T05:21:30.058 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:21:30.058 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":89,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","created":"2024-03-28T05:13:13.453653+0000","modified":"2024-03-28T05:21:12.641185+0000","last_up_change":"2024-03-28T05:20:20.221527+0000","last_in_change":"2024-03-28T05:20:00.365478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T05:16:32.986974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T05:21:06.103894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T05:21:06.827190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T05:21:10.507418+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"18150942-59a4-4c3b-83d4-c1fc3ccf1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6803","nonce":1219956881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6805","nonce":1219956881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6809","nonce":1219956881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6807","nonce":1219956881}]},"public_addr":"172.21.15.23:6803/1219956881","cluster_addr":"172.21.15.23:6805/1219956881","heartbeat_back_addr":"172.21.15.23:6809/1219956881","heartbeat_front_addr":"172.21.15.23:6807/1219956881","state":["exists","up"]},{"osd":1,"uuid":"ba6d1a95-5135-4670-95f9-7878c0e7bd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6801","nonce":555491234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6803","nonce":555491234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6807","nonce":555491234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6805","nonce":555491234}]},"public_addr":"172.21.15.98:6801/555491234","cluster_addr":"172.21.15.98:6803/555491234","heartbeat_back_addr":"172.21.15.98:6807/555491234","heartbeat_front_addr":"172.21.15.98:6805/555491234","state":["exists","up"]},{"osd":2,"uuid":"873bdd60-2a47-403b-b1f4-ee9226a95067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6801","nonce":1535856914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6803","nonce":1535856914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6807","nonce":1535856914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6805","nonce":1535856914}]},"public_addr":"172.21.15.113:6801/1535856914","cluster_addr":"172.21.15.113:6803/1535856914","heartbeat_back_addr":"172.21.15.113:6807/1535856914","heartbeat_front_addr":"172.21.15.113:6805/1535856914","state":["exists","up"]},{"osd":3,"uuid":"83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6811","nonce":2949415020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6813","nonce":2949415020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6817","nonce":2949415020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6815","nonce":2949415020}]},"public_addr":"172.21.15.23:6811/2949415020","cluster_addr":"172.21.15.23:6813/2949415020","heartbeat_back_addr":"172.21.15.23:6817/2949415020","heartbeat_front_addr":"172.21.15.23:6815/2949415020","state":["exists","up"]},{"osd":4,"uuid":"76a90bac-b221-40a3-8238-40e8049cec36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6809","nonce":2615902819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6811","nonce":2615902819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6815","nonce":2615902819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6813","nonce":2615902819}]},"public_addr":"172.21.15.98:6809/2615902819","cluster_addr":"172.21.15.98:6811/2615902819","heartbeat_back_addr":"172.21.15.98:6815/2615902819","heartbeat_front_addr":"172.21.15.98:6813/2615902819","state":["exists","up"]},{"osd":5,"uuid":"b53fdb55-a8df-40a8-bd1b-8bcef634e26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6809","nonce":625271528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6811","nonce":625271528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6815","nonce":625271528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6813","nonce":625271528}]},"public_addr":"172.21.15.113:6809/625271528","cluster_addr":"172.21.15.113:6811/625271528","heartbeat_back_addr":"172.21.15.113:6815/625271528","heartbeat_front_addr":"172.21.15.113:6813/625271528","state":["exists","up"]},{"osd":6,"uuid":"f00283d2-8d67-4e97-a575-9a9d3b55d5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6819","nonce":4188464176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6821","nonce":4188464176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6825","nonce":4188464176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6823","nonce":4188464176}]},"public_addr":"172.21.15.23:6819/4188464176","cluster_addr":"172.21.15.23:6821/4188464176","heartbeat_back_addr":"172.21.15.23:6825/4188464176","heartbeat_front_addr":"172.21.15.23:6823/4188464176","state":["exists","up"]},{"osd":7,"uuid":"edc008a7-b07b-40a8-88b7-1614ba38a775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6817","nonce":1985546028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6819","nonce":1985546028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6823","nonce":1985546028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6821","nonce":1985546028}]},"public_addr":"172.21.15.98:6817/1985546028","cluster_addr":"172.21.15.98:6819/1985546028","heartbeat_back_addr":"172.21.15.98:6823/1985546028","heartbeat_front_addr":"172.21.15.98:6821/1985546028","state":["exists","up"]},{"osd":8,"uuid":"56e7b2d1-81f0-47b2-8638-f29832bed8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6817","nonce":3534876081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6819","nonce":3534876081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6823","nonce":3534876081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6821","nonce":3534876081}]},"public_addr":"172.21.15.113:6817/3534876081","cluster_addr":"172.21.15.113:6819/3534876081","heartbeat_back_addr":"172.21.15.113:6823/3534876081","heartbeat_front_addr":"172.21.15.113:6821/3534876081","state":["exists","up"]},{"osd":9,"uuid":"a2e90caa-6e9c-42bf-9f18-81f7f47880cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6827","nonce":672264207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6829","nonce":672264207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6833","nonce":672264207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6831","nonce":672264207}]},"public_addr":"172.21.15.23:6827/672264207","cluster_addr":"172.21.15.23:6829/672264207","heartbeat_back_addr":"172.21.15.23:6833/672264207","heartbeat_front_addr":"172.21.15.23:6831/672264207","state":["exists","up"]},{"osd":10,"uuid":"0eda917f-020c-437f-95ae-b26203f6c68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6825","nonce":2699920595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6827","nonce":2699920595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6831","nonce":2699920595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6829","nonce":2699920595}]},"public_addr":"172.21.15.98:6825/2699920595","cluster_addr":"172.21.15.98:6827/2699920595","heartbeat_back_addr":"172.21.15.98:6831/2699920595","heartbeat_front_addr":"172.21.15.98:6829/2699920595","state":["exists","up"]},{"osd":11,"uuid":"aae78501-78a8-469a-be20-3006f5ec5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6825","nonce":2113644020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6827","nonce":2113644020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6831","nonce":2113644020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6829","nonce":2113644020}]},"public_addr":"172.21.15.113:6825/2113644020","cluster_addr":"172.21.15.113:6827/2113644020","heartbeat_back_addr":"172.21.15.113:6831/2113644020","heartbeat_front_addr":"172.21.15.113:6829/2113644020","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:15:43.527475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:07.006538+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:30.594176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:55.810999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:20.595811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:45.474138+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:10.500409+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:35.787638+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:01.001188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:26.761115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:52.952752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:20:18.559832+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:6800/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1876451381":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2253559862":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2653562762":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/3337463830":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6800/1735221490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/2784771635":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2389902490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6801/1735221490":"2024-03-29T05:13:42.579607+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T05:21:30.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.060+0000 7fcf1cf5b640 1 -- 172.21.15.23:0/3172025155 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fcf040a2800 msgr2=0x7fcf040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:30.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.060+0000 7fcf1cf5b640 1 --2- 172.21.15.23:0/3172025155 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fcf040a2800 0x7fcf040a4cc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcf080058c0 tx=0x7fcf08005850 comp rx=0 tx=0).stop 2024-03-28T05:21:30.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.060+0000 7fcf1cf5b640 1 -- 172.21.15.23:0/3172025155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf181021f0 msgr2=0x7fcf1819ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:30.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.060+0000 7fcf1cf5b640 1 --2- 172.21.15.23:0/3172025155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf181021f0 0x7fcf1819ef60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcf1000eb10 tx=0x7fcf1000c2d0 comp rx=0 tx=0).stop 2024-03-28T05:21:30.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.060+0000 7fcf1cf5b640 1 -- 172.21.15.23:0/3172025155 shutdown_connections 2024-03-28T05:21:30.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.060+0000 7fcf1cf5b640 1 --2- 172.21.15.23:0/3172025155 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fcf040a2800 0x7fcf040a4cc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:30.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.060+0000 7fcf1cf5b640 1 --2- 172.21.15.23:0/3172025155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf181021f0 0x7fcf1819ef60 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:30.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.060+0000 7fcf1cf5b640 1 --2- 172.21.15.23:0/3172025155 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf18101830 0x7fcf18197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:30.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.060+0000 7fcf1cf5b640 1 --2- 172.21.15.23:0/3172025155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf18100640 0x7fcf181979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:30.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.060+0000 7fcf1cf5b640 1 -- 172.21.15.23:0/3172025155 >> 172.21.15.23:0/3172025155 conn(0x7fcf180fbbd0 msgr2=0x7fcf180fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:30.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.060+0000 7fcf1cf5b640 1 -- 172.21.15.23:0/3172025155 shutdown_connections 2024-03-28T05:21:30.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.060+0000 7fcf1cf5b640 1 -- 172.21.15.23:0/3172025155 wait complete. 2024-03-28T05:21:30.071 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T05:21:30.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.175+0000 7f604ad59640 1 -- 172.21.15.23:0/3314022941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6044100640 msgr2=0x7f6044100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:30.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.175+0000 7f604ad59640 1 --2- 172.21.15.23:0/3314022941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6044100640 0x7f6044100a60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6038009900 tx=0x7f603802bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:30.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.175+0000 7f604ad59640 1 -- 172.21.15.23:0/3314022941 shutdown_connections 2024-03-28T05:21:30.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.175+0000 7f604ad59640 1 --2- 172.21.15.23:0/3314022941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60441021f0 0x7f6044108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:30.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.175+0000 7f604ad59640 1 --2- 172.21.15.23:0/3314022941 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6044101830 0x7f6044101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:30.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.175+0000 7f604ad59640 1 --2- 172.21.15.23:0/3314022941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6044100640 0x7f6044100a60 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:30.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.175+0000 7f604ad59640 1 -- 172.21.15.23:0/3314022941 >> 172.21.15.23:0/3314022941 conn(0x7f60440fbbd0 msgr2=0x7f60440fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:30.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.175+0000 7f604ad59640 1 -- 172.21.15.23:0/3314022941 shutdown_connections 2024-03-28T05:21:30.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.175+0000 7f604ad59640 1 -- 172.21.15.23:0/3314022941 wait complete. 2024-03-28T05:21:30.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.176+0000 7f604ad59640 1 Processor -- start 2024-03-28T05:21:30.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.176+0000 7f604ad59640 1 -- start start 2024-03-28T05:21:30.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.176+0000 7f604ad59640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6044100640 0x7f604410afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:30.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.176+0000 7f604ad59640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6044101830 0x7f6044109600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:30.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.176+0000 7f604ad59640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60441021f0 0x7f6044109b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:30.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.176+0000 7f604ad59640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604410d370 con 0x7f6044100640 2024-03-28T05:21:30.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.176+0000 7f604ad59640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604410d1f0 con 0x7f6044101830 2024-03-28T05:21:30.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.176+0000 7f604ad59640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604410d4f0 con 0x7f60441021f0 2024-03-28T05:21:30.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.176+0000 7f6048af5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6044100640 0x7f604410afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:30.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.176+0000 7f6048af5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6044100640 0x7f604410afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44438/0 (socket says 172.21.15.23:44438) 2024-03-28T05:21:30.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.176+0000 7f6048af5640 1 -- 172.21.15.23:0/1780737745 learned_addr learned my addr 172.21.15.23:0/1780737745 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:30.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.176+0000 7f60492f6640 1 --2- 172.21.15.23:0/1780737745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60441021f0 0x7f6044109b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:30.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.176+0000 7f6048af5640 1 -- 172.21.15.23:0/1780737745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60441021f0 msgr2=0x7f6044109b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:30.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.176+0000 7f6048af5640 1 --2- 172.21.15.23:0/1780737745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60441021f0 0x7f6044109b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:30.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.176+0000 7f6048af5640 1 -- 172.21.15.23:0/1780737745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6044101830 msgr2=0x7f6044109600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:30.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.176+0000 7f6048af5640 1 --2- 172.21.15.23:0/1780737745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6044101830 0x7f6044109600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:30.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.176+0000 7f6048af5640 1 -- 172.21.15.23:0/1780737745 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604410a3d0 con 0x7f6044100640 2024-03-28T05:21:30.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.176+0000 7f60492f6640 1 --2- 172.21.15.23:0/1780737745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60441021f0 0x7f6044109b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:21:30.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.176+0000 7f6048af5640 1 --2- 172.21.15.23:0/1780737745 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6044100640 0x7f604410afc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6038005ab0 tx=0x7f6038005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:30.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.177+0000 7f6041ffb640 1 -- 172.21.15.23:0/1780737745 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6038041070 con 0x7f6044100640 2024-03-28T05:21:30.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.177+0000 7f604ad59640 1 -- 172.21.15.23:0/1780737745 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60441a1aa0 con 0x7f6044100640 2024-03-28T05:21:30.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.177+0000 7f6041ffb640 1 -- 172.21.15.23:0/1780737745 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60380054e0 con 0x7f6044100640 2024-03-28T05:21:30.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.177+0000 7f6041ffb640 1 -- 172.21.15.23:0/1780737745 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6038034440 con 0x7f6044100640 2024-03-28T05:21:30.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.177+0000 7f604ad59640 1 -- 172.21.15.23:0/1780737745 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60441a1ff0 con 0x7f6044100640 2024-03-28T05:21:30.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.178+0000 7f604ad59640 1 -- 172.21.15.23:0/1780737745 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6044107960 con 0x7f6044100640 2024-03-28T05:21:30.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.179+0000 7f6041ffb640 1 -- 172.21.15.23:0/1780737745 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6038044070 con 0x7f6044100640 2024-03-28T05:21:30.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.179+0000 7f6041ffb640 1 --2- 172.21.15.23:0/1780737745 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f60340a2800 0x7f60340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:30.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.179+0000 7f6043fff640 1 --2- 172.21.15.23:0/1780737745 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f60340a2800 0x7f60340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:30.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.179+0000 7f6041ffb640 1 -- 172.21.15.23:0/1780737745 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f60380f09a0 con 0x7f6044100640 2024-03-28T05:21:30.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.180+0000 7f6043fff640 1 --2- 172.21.15.23:0/1780737745 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f60340a2800 0x7f60340a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f604410ad30 tx=0x7f6030005ed0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:30.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.184+0000 7f6041ffb640 1 -- 172.21.15.23:0/1780737745 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f60380bd800 con 0x7f6044100640 2024-03-28T05:21:30.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.368+0000 7f604ad59640 1 -- 172.21.15.23:0/1780737745 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f604402d240 con 0x7f6044100640 2024-03-28T05:21:30.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.370+0000 7f6041ffb640 1 -- 172.21.15.23:0/1780737745 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24082 (secure 0 0 0) 0x7f60380bd100 con 0x7f6044100640 2024-03-28T05:21:30.369 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:21:30.370 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":89,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","created":"2024-03-28T05:13:13.453653+0000","modified":"2024-03-28T05:21:12.641185+0000","last_up_change":"2024-03-28T05:20:20.221527+0000","last_in_change":"2024-03-28T05:20:00.365478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T05:16:32.986974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T05:21:06.103894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T05:21:06.827190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T05:21:10.507418+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"18150942-59a4-4c3b-83d4-c1fc3ccf1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6803","nonce":1219956881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6805","nonce":1219956881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6809","nonce":1219956881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6807","nonce":1219956881}]},"public_addr":"172.21.15.23:6803/1219956881","cluster_addr":"172.21.15.23:6805/1219956881","heartbeat_back_addr":"172.21.15.23:6809/1219956881","heartbeat_front_addr":"172.21.15.23:6807/1219956881","state":["exists","up"]},{"osd":1,"uuid":"ba6d1a95-5135-4670-95f9-7878c0e7bd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6801","nonce":555491234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6803","nonce":555491234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6807","nonce":555491234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6805","nonce":555491234}]},"public_addr":"172.21.15.98:6801/555491234","cluster_addr":"172.21.15.98:6803/555491234","heartbeat_back_addr":"172.21.15.98:6807/555491234","heartbeat_front_addr":"172.21.15.98:6805/555491234","state":["exists","up"]},{"osd":2,"uuid":"873bdd60-2a47-403b-b1f4-ee9226a95067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6801","nonce":1535856914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6803","nonce":1535856914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6807","nonce":1535856914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6805","nonce":1535856914}]},"public_addr":"172.21.15.113:6801/1535856914","cluster_addr":"172.21.15.113:6803/1535856914","heartbeat_back_addr":"172.21.15.113:6807/1535856914","heartbeat_front_addr":"172.21.15.113:6805/1535856914","state":["exists","up"]},{"osd":3,"uuid":"83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6811","nonce":2949415020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6813","nonce":2949415020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6817","nonce":2949415020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6815","nonce":2949415020}]},"public_addr":"172.21.15.23:6811/2949415020","cluster_addr":"172.21.15.23:6813/2949415020","heartbeat_back_addr":"172.21.15.23:6817/2949415020","heartbeat_front_addr":"172.21.15.23:6815/2949415020","state":["exists","up"]},{"osd":4,"uuid":"76a90bac-b221-40a3-8238-40e8049cec36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6809","nonce":2615902819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6811","nonce":2615902819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6815","nonce":2615902819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6813","nonce":2615902819}]},"public_addr":"172.21.15.98:6809/2615902819","cluster_addr":"172.21.15.98:6811/2615902819","heartbeat_back_addr":"172.21.15.98:6815/2615902819","heartbeat_front_addr":"172.21.15.98:6813/2615902819","state":["exists","up"]},{"osd":5,"uuid":"b53fdb55-a8df-40a8-bd1b-8bcef634e26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6809","nonce":625271528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6811","nonce":625271528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6815","nonce":625271528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6813","nonce":625271528}]},"public_addr":"172.21.15.113:6809/625271528","cluster_addr":"172.21.15.113:6811/625271528","heartbeat_back_addr":"172.21.15.113:6815/625271528","heartbeat_front_addr":"172.21.15.113:6813/625271528","state":["exists","up"]},{"osd":6,"uuid":"f00283d2-8d67-4e97-a575-9a9d3b55d5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6819","nonce":4188464176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6821","nonce":4188464176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6825","nonce":4188464176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6823","nonce":4188464176}]},"public_addr":"172.21.15.23:6819/4188464176","cluster_addr":"172.21.15.23:6821/4188464176","heartbeat_back_addr":"172.21.15.23:6825/4188464176","heartbeat_front_addr":"172.21.15.23:6823/4188464176","state":["exists","up"]},{"osd":7,"uuid":"edc008a7-b07b-40a8-88b7-1614ba38a775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6817","nonce":1985546028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6819","nonce":1985546028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6823","nonce":1985546028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6821","nonce":1985546028}]},"public_addr":"172.21.15.98:6817/1985546028","cluster_addr":"172.21.15.98:6819/1985546028","heartbeat_back_addr":"172.21.15.98:6823/1985546028","heartbeat_front_addr":"172.21.15.98:6821/1985546028","state":["exists","up"]},{"osd":8,"uuid":"56e7b2d1-81f0-47b2-8638-f29832bed8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6817","nonce":3534876081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6819","nonce":3534876081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6823","nonce":3534876081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6821","nonce":3534876081}]},"public_addr":"172.21.15.113:6817/3534876081","cluster_addr":"172.21.15.113:6819/3534876081","heartbeat_back_addr":"172.21.15.113:6823/3534876081","heartbeat_front_addr":"172.21.15.113:6821/3534876081","state":["exists","up"]},{"osd":9,"uuid":"a2e90caa-6e9c-42bf-9f18-81f7f47880cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6827","nonce":672264207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6829","nonce":672264207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6833","nonce":672264207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6831","nonce":672264207}]},"public_addr":"172.21.15.23:6827/672264207","cluster_addr":"172.21.15.23:6829/672264207","heartbeat_back_addr":"172.21.15.23:6833/672264207","heartbeat_front_addr":"172.21.15.23:6831/672264207","state":["exists","up"]},{"osd":10,"uuid":"0eda917f-020c-437f-95ae-b26203f6c68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6825","nonce":2699920595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6827","nonce":2699920595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6831","nonce":2699920595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6829","nonce":2699920595}]},"public_addr":"172.21.15.98:6825/2699920595","cluster_addr":"172.21.15.98:6827/2699920595","heartbeat_back_addr":"172.21.15.98:6831/2699920595","heartbeat_front_addr":"172.21.15.98:6829/2699920595","state":["exists","up"]},{"osd":11,"uuid":"aae78501-78a8-469a-be20-3006f5ec5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6825","nonce":2113644020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6827","nonce":2113644020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6831","nonce":2113644020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6829","nonce":2113644020}]},"public_addr":"172.21.15.113:6825/2113644020","cluster_addr":"172.21.15.113:6827/2113644020","heartbeat_back_addr":"172.21.15.113:6831/2113644020","heartbeat_front_addr":"172.21.15.113:6829/2113644020","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:15:43.527475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:07.006538+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:30.594176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:55.810999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:20.595811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:45.474138+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:10.500409+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:35.787638+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:01.001188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:26.761115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:52.952752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:20:18.559832+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:6800/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1876451381":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2253559862":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2653562762":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/3337463830":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6800/1735221490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/2784771635":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2389902490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6801/1735221490":"2024-03-29T05:13:42.579607+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T05:21:30.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.372+0000 7f604ad59640 1 -- 172.21.15.23:0/1780737745 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f60340a2800 msgr2=0x7f60340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:30.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.372+0000 7f604ad59640 1 --2- 172.21.15.23:0/1780737745 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f60340a2800 0x7f60340a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f604410ad30 tx=0x7f6030005ed0 comp rx=0 tx=0).stop 2024-03-28T05:21:30.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.372+0000 7f604ad59640 1 -- 172.21.15.23:0/1780737745 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6044100640 msgr2=0x7f604410afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:30.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.372+0000 7f604ad59640 1 --2- 172.21.15.23:0/1780737745 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6044100640 0x7f604410afc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6038005ab0 tx=0x7f6038005dc0 comp rx=0 tx=0).stop 2024-03-28T05:21:30.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.372+0000 7f604ad59640 1 -- 172.21.15.23:0/1780737745 shutdown_connections 2024-03-28T05:21:30.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.372+0000 7f604ad59640 1 --2- 172.21.15.23:0/1780737745 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f60340a2800 0x7f60340a4cc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:30.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.372+0000 7f604ad59640 1 --2- 172.21.15.23:0/1780737745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60441021f0 0x7f6044109b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:30.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.372+0000 7f604ad59640 1 --2- 172.21.15.23:0/1780737745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6044101830 0x7f6044109600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:30.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.372+0000 7f604ad59640 1 --2- 172.21.15.23:0/1780737745 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6044100640 0x7f604410afc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:30.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.372+0000 7f604ad59640 1 -- 172.21.15.23:0/1780737745 >> 172.21.15.23:0/1780737745 conn(0x7f60440fbbd0 msgr2=0x7f6044104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:30.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.372+0000 7f604ad59640 1 -- 172.21.15.23:0/1780737745 shutdown_connections 2024-03-28T05:21:30.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.372+0000 7f604ad59640 1 -- 172.21.15.23:0/1780737745 wait complete. 2024-03-28T05:21:30.383 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T05:21:30.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.489+0000 7f94e3db3640 1 -- 172.21.15.23:0/3269687408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94dc104b30 msgr2=0x7f94dc106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:30.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.489+0000 7f94e3db3640 1 --2- 172.21.15.23:0/3269687408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94dc104b30 0x7f94dc106f20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f94d0009900 tx=0x7f94d002bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:30.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.490+0000 7f94e3db3640 1 -- 172.21.15.23:0/3269687408 shutdown_connections 2024-03-28T05:21:30.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.490+0000 7f94e3db3640 1 --2- 172.21.15.23:0/3269687408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94dc107460 0x7f94dc109850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:30.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.490+0000 7f94e3db3640 1 --2- 172.21.15.23:0/3269687408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94dc104b30 0x7f94dc106f20 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:30.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.490+0000 7f94e3db3640 1 --2- 172.21.15.23:0/3269687408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94dc102200 0x7f94dc1045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:30.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.490+0000 7f94e3db3640 1 -- 172.21.15.23:0/3269687408 >> 172.21.15.23:0/3269687408 conn(0x7f94dc0fbb70 msgr2=0x7f94dc0fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:30.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.490+0000 7f94e3db3640 1 -- 172.21.15.23:0/3269687408 shutdown_connections 2024-03-28T05:21:30.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.490+0000 7f94e3db3640 1 -- 172.21.15.23:0/3269687408 wait complete. 2024-03-28T05:21:30.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.490+0000 7f94e3db3640 1 Processor -- start 2024-03-28T05:21:30.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.490+0000 7f94e3db3640 1 -- start start 2024-03-28T05:21:30.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.490+0000 7f94e3db3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94dc102200 0x7f94dc1979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:30.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.490+0000 7f94e3db3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94dc104b30 0x7f94dc197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:30.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.490+0000 7f94e3db3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94dc107460 0x7f94dc19ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:30.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.490+0000 7f94e3db3640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94dc10d340 con 0x7f94dc102200 2024-03-28T05:21:30.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.490+0000 7f94e3db3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94dc10d1c0 con 0x7f94dc104b30 2024-03-28T05:21:30.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.490+0000 7f94e3db3640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94dc10d4c0 con 0x7f94dc107460 2024-03-28T05:21:30.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.490+0000 7f94e1b4f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94dc102200 0x7f94dc1979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:30.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.490+0000 7f94e1b4f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94dc102200 0x7f94dc1979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44446/0 (socket says 172.21.15.23:44446) 2024-03-28T05:21:30.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.490+0000 7f94e1b4f640 1 -- 172.21.15.23:0/2431095579 learned_addr learned my addr 172.21.15.23:0/2431095579 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:30.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.490+0000 7f94e2350640 1 --2- 172.21.15.23:0/2431095579 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94dc107460 0x7f94dc19ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:30.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.491+0000 7f94e134e640 1 --2- 172.21.15.23:0/2431095579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94dc104b30 0x7f94dc197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:30.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.491+0000 7f94e1b4f640 1 -- 172.21.15.23:0/2431095579 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94dc107460 msgr2=0x7f94dc19ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:30.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.491+0000 7f94e1b4f640 1 --2- 172.21.15.23:0/2431095579 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94dc107460 0x7f94dc19ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:30.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.491+0000 7f94e1b4f640 1 -- 172.21.15.23:0/2431095579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94dc104b30 msgr2=0x7f94dc197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:30.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.491+0000 7f94e1b4f640 1 --2- 172.21.15.23:0/2431095579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94dc104b30 0x7f94dc197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:30.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.491+0000 7f94e1b4f640 1 -- 172.21.15.23:0/2431095579 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94dc19f600 con 0x7f94dc102200 2024-03-28T05:21:30.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.491+0000 7f94e134e640 1 --2- 172.21.15.23:0/2431095579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94dc104b30 0x7f94dc197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:21:30.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.491+0000 7f94e1b4f640 1 --2- 172.21.15.23:0/2431095579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94dc102200 0x7f94dc1979e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f94d400e9f0 tx=0x7f94d400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:30.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.491+0000 7f94daffd640 1 -- 172.21.15.23:0/2431095579 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d400cd80 con 0x7f94dc102200 2024-03-28T05:21:30.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.491+0000 7f94e3db3640 1 -- 172.21.15.23:0/2431095579 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94dc19f8f0 con 0x7f94dc102200 2024-03-28T05:21:30.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.491+0000 7f94daffd640 1 -- 172.21.15.23:0/2431095579 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94d4005500 con 0x7f94dc102200 2024-03-28T05:21:30.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.491+0000 7f94daffd640 1 -- 172.21.15.23:0/2431095579 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d4010690 con 0x7f94dc102200 2024-03-28T05:21:30.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.491+0000 7f94e3db3640 1 -- 172.21.15.23:0/2431095579 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94dc19fdf0 con 0x7f94dc102200 2024-03-28T05:21:30.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.492+0000 7f94e3db3640 1 -- 172.21.15.23:0/2431095579 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94dc0fd660 con 0x7f94dc102200 2024-03-28T05:21:30.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.493+0000 7f94daffd640 1 -- 172.21.15.23:0/2431095579 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f94d40046c0 con 0x7f94dc102200 2024-03-28T05:21:30.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.494+0000 7f94daffd640 1 --2- 172.21.15.23:0/2431095579 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f94c40a2800 0x7f94c40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:30.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.494+0000 7f94daffd640 1 -- 172.21.15.23:0/2431095579 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f94d40cf320 con 0x7f94dc102200 2024-03-28T05:21:30.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.494+0000 7f94e134e640 1 --2- 172.21.15.23:0/2431095579 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f94c40a2800 0x7f94c40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:30.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.494+0000 7f94e134e640 1 --2- 172.21.15.23:0/2431095579 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f94c40a2800 0x7f94c40a4cc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f94d0006010 tx=0x7f94d0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:30.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.497+0000 7f94daffd640 1 -- 172.21.15.23:0/2431095579 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f94d4053740 con 0x7f94dc102200 2024-03-28T05:21:30.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:30 smithi098 ceph-mon[36087]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T05:21:30.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:30 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3172025155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:30.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:30.720+0000 7f94e3db3640 1 -- 172.21.15.23:0/2431095579 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f94dc02d070 con 0x7f94dc102200 2024-03-28T05:21:30.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:30 smithi113 ceph-mon[37164]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T05:21:30.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:30 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3172025155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:30.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:30 smithi023 ceph-mon[31954]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T05:21:30.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:30 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3172025155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:31.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.361+0000 7f94daffd640 1 -- 172.21.15.23:0/2431095579 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f94d409c180 con 0x7f94dc102200 2024-03-28T05:21:31.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.363+0000 7f94e3db3640 1 -- 172.21.15.23:0/2431095579 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f94c40a2800 msgr2=0x7f94c40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:31.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.363+0000 7f94e3db3640 1 --2- 172.21.15.23:0/2431095579 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f94c40a2800 0x7f94c40a4cc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f94d0006010 tx=0x7f94d0000bc0 comp rx=0 tx=0).stop 2024-03-28T05:21:31.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.364+0000 7f94e3db3640 1 -- 172.21.15.23:0/2431095579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94dc102200 msgr2=0x7f94dc1979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:31.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.364+0000 7f94e3db3640 1 --2- 172.21.15.23:0/2431095579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94dc102200 0x7f94dc1979e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f94d400e9f0 tx=0x7f94d400edf0 comp rx=0 tx=0).stop 2024-03-28T05:21:31.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.364+0000 7f94e3db3640 1 -- 172.21.15.23:0/2431095579 shutdown_connections 2024-03-28T05:21:31.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.364+0000 7f94e3db3640 1 --2- 172.21.15.23:0/2431095579 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f94c40a2800 0x7f94c40a4cc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:31.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.364+0000 7f94e3db3640 1 --2- 172.21.15.23:0/2431095579 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94dc107460 0x7f94dc19ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:31.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.364+0000 7f94e3db3640 1 --2- 172.21.15.23:0/2431095579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94dc104b30 0x7f94dc197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:31.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.364+0000 7f94e3db3640 1 --2- 172.21.15.23:0/2431095579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94dc102200 0x7f94dc1979e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:31.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.364+0000 7f94e3db3640 1 -- 172.21.15.23:0/2431095579 >> 172.21.15.23:0/2431095579 conn(0x7f94dc0fbb70 msgr2=0x7f94dc102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:31.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.364+0000 7f94e3db3640 1 -- 172.21.15.23:0/2431095579 shutdown_connections 2024-03-28T05:21:31.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.364+0000 7f94e3db3640 1 -- 172.21.15.23:0/2431095579 wait complete. 2024-03-28T05:21:31.376 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 1, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-03-28T05:21:31.376 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-28T05:21:31.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.487+0000 7fd3c8f48640 1 -- 172.21.15.23:0/348236686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3c4104ab0 msgr2=0x7fd3c4106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:31.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.487+0000 7fd3c8f48640 1 --2- 172.21.15.23:0/348236686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3c4104ab0 0x7fd3c4106ea0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac009900 tx=0x7fd3ac02bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:31.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.488+0000 7fd3c8f48640 1 -- 172.21.15.23:0/348236686 shutdown_connections 2024-03-28T05:21:31.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.488+0000 7fd3c8f48640 1 --2- 172.21.15.23:0/348236686 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3c41073e0 0x7fd3c41097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:31.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.488+0000 7fd3c8f48640 1 --2- 172.21.15.23:0/348236686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3c4104ab0 0x7fd3c4106ea0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:31.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.488+0000 7fd3c8f48640 1 --2- 172.21.15.23:0/348236686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c4102180 0x7fd3c4104570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:31.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.488+0000 7fd3c8f48640 1 -- 172.21.15.23:0/348236686 >> 172.21.15.23:0/348236686 conn(0x7fd3c40fbb10 msgr2=0x7fd3c40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:31.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.488+0000 7fd3c8f48640 1 -- 172.21.15.23:0/348236686 shutdown_connections 2024-03-28T05:21:31.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.488+0000 7fd3c8f48640 1 -- 172.21.15.23:0/348236686 wait complete. 2024-03-28T05:21:31.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.488+0000 7fd3c8f48640 1 Processor -- start 2024-03-28T05:21:31.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.488+0000 7fd3c8f48640 1 -- start start 2024-03-28T05:21:31.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.488+0000 7fd3c8f48640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c4102180 0x7fd3c4101af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:31.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.488+0000 7fd3c8f48640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3c4104ab0 0x7fd3c4100130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:31.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.488+0000 7fd3c8f48640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3c41073e0 0x7fd3c4100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:31.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.488+0000 7fd3c8f48640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c410d300 con 0x7fd3c41073e0 2024-03-28T05:21:31.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.488+0000 7fd3c8f48640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c410d180 con 0x7fd3c4102180 2024-03-28T05:21:31.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.488+0000 7fd3c8f48640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c410d480 con 0x7fd3c4104ab0 2024-03-28T05:21:31.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.489+0000 7fd3c2d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3c41073e0 0x7fd3c4100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:31.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.489+0000 7fd3c2d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3c41073e0 0x7fd3c4100670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44462/0 (socket says 172.21.15.23:44462) 2024-03-28T05:21:31.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.489+0000 7fd3c2d9d640 1 -- 172.21.15.23:0/3322444979 learned_addr learned my addr 172.21.15.23:0/3322444979 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:31.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.489+0000 7fd3c1d9b640 1 --2- 172.21.15.23:0/3322444979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3c4104ab0 0x7fd3c4100130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:31.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.489+0000 7fd3c259c640 1 --2- 172.21.15.23:0/3322444979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c4102180 0x7fd3c4101af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:31.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.489+0000 7fd3c2d9d640 1 -- 172.21.15.23:0/3322444979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3c4104ab0 msgr2=0x7fd3c4100130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:31.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.489+0000 7fd3c2d9d640 1 --2- 172.21.15.23:0/3322444979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3c4104ab0 0x7fd3c4100130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:31.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.489+0000 7fd3c2d9d640 1 -- 172.21.15.23:0/3322444979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c4102180 msgr2=0x7fd3c4101af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:31.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.489+0000 7fd3c2d9d640 1 --2- 172.21.15.23:0/3322444979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c4102180 0x7fd3c4101af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:31.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.489+0000 7fd3c2d9d640 1 -- 172.21.15.23:0/3322444979 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c4100f00 con 0x7fd3c41073e0 2024-03-28T05:21:31.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.489+0000 7fd3c259c640 1 --2- 172.21.15.23:0/3322444979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c4102180 0x7fd3c4101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:21:31.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.489+0000 7fd3c2d9d640 1 --2- 172.21.15.23:0/3322444979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3c41073e0 0x7fd3c4100670 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400eb10 tx=0x7fd3b400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:31.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.489+0000 7fd3bb7fe640 1 -- 172.21.15.23:0/3322444979 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4004b20 con 0x7fd3c41073e0 2024-03-28T05:21:31.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.489+0000 7fd3c8f48640 1 -- 172.21.15.23:0/3322444979 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3c41a1a90 con 0x7fd3c41073e0 2024-03-28T05:21:31.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.489+0000 7fd3bb7fe640 1 -- 172.21.15.23:0/3322444979 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3b4004cc0 con 0x7fd3c41073e0 2024-03-28T05:21:31.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.489+0000 7fd3bb7fe640 1 -- 172.21.15.23:0/3322444979 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4010760 con 0x7fd3c41073e0 2024-03-28T05:21:31.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.489+0000 7fd3c8f48640 1 -- 172.21.15.23:0/3322444979 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3c41a1fb0 con 0x7fd3c41073e0 2024-03-28T05:21:31.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.490+0000 7fd3c8f48640 1 -- 172.21.15.23:0/3322444979 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3c4068490 con 0x7fd3c41073e0 2024-03-28T05:21:31.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.491+0000 7fd3bb7fe640 1 -- 172.21.15.23:0/3322444979 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd3b4005000 con 0x7fd3c41073e0 2024-03-28T05:21:31.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.492+0000 7fd3bb7fe640 1 --2- 172.21.15.23:0/3322444979 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd3b00a2800 0x7fd3b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:31.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.492+0000 7fd3c259c640 1 --2- 172.21.15.23:0/3322444979 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd3b00a2800 0x7fd3b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:31.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.492+0000 7fd3bb7fe640 1 -- 172.21.15.23:0/3322444979 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd3b40cf570 con 0x7fd3c41073e0 2024-03-28T05:21:31.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.492+0000 7fd3c259c640 1 --2- 172.21.15.23:0/3322444979 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd3b00a2800 0x7fd3b00a4cc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc00abe0 tx=0x7fd3bc00a5c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:31.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.495+0000 7fd3bb7fe640 1 -- 172.21.15.23:0/3322444979 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd3b4053740 con 0x7fd3c41073e0 2024-03-28T05:21:31.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:31 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1780737745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:31.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:31 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2431095579' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T05:21:31.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:31.720+0000 7fd3c8f48640 1 -- 172.21.15.23:0/3322444979 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fd3c402d200 con 0x7fd3c41073e0 2024-03-28T05:21:31.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:31 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1780737745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:31.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:31 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2431095579' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T05:21:31.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:31 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1780737745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:31.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:31 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2431095579' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T05:21:32.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.370+0000 7fd3bb7fe640 1 -- 172.21.15.23:0/3322444979 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v18) v1 ==== 121+0+0 (secure 0 0 0) 0x7fd3b409c3d0 con 0x7fd3c41073e0 2024-03-28T05:21:32.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.373+0000 7fd3c8f48640 1 -- 172.21.15.23:0/3322444979 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd3b00a2800 msgr2=0x7fd3b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:32.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.373+0000 7fd3c8f48640 1 --2- 172.21.15.23:0/3322444979 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd3b00a2800 0x7fd3b00a4cc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc00abe0 tx=0x7fd3bc00a5c0 comp rx=0 tx=0).stop 2024-03-28T05:21:32.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.373+0000 7fd3c8f48640 1 -- 172.21.15.23:0/3322444979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3c41073e0 msgr2=0x7fd3c4100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:32.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.373+0000 7fd3c8f48640 1 --2- 172.21.15.23:0/3322444979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3c41073e0 0x7fd3c4100670 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400eb10 tx=0x7fd3b400c2d0 comp rx=0 tx=0).stop 2024-03-28T05:21:32.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.373+0000 7fd3c8f48640 1 -- 172.21.15.23:0/3322444979 shutdown_connections 2024-03-28T05:21:32.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.373+0000 7fd3c8f48640 1 --2- 172.21.15.23:0/3322444979 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd3b00a2800 0x7fd3b00a4cc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:32.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.373+0000 7fd3c8f48640 1 --2- 172.21.15.23:0/3322444979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3c41073e0 0x7fd3c4100670 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:32.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.373+0000 7fd3c8f48640 1 --2- 172.21.15.23:0/3322444979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3c4104ab0 0x7fd3c4100130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:32.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.373+0000 7fd3c8f48640 1 --2- 172.21.15.23:0/3322444979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3c4102180 0x7fd3c4101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:32.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.373+0000 7fd3c8f48640 1 -- 172.21.15.23:0/3322444979 >> 172.21.15.23:0/3322444979 conn(0x7fd3c40fbb10 msgr2=0x7fd3c4102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:32.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.373+0000 7fd3c8f48640 1 -- 172.21.15.23:0/3322444979 shutdown_connections 2024-03-28T05:21:32.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.373+0000 7fd3c8f48640 1 -- 172.21.15.23:0/3322444979 wait complete. 2024-03-28T05:21:32.386 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-28T05:21:32.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.497+0000 7ff4e8cf8640 1 -- 172.21.15.23:0/2444566201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4e4102150 msgr2=0x7ff4e41089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:32.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.497+0000 7ff4e8cf8640 1 --2- 172.21.15.23:0/2444566201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4e4102150 0x7ff4e41089e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc009ca0 tx=0x7ff4dc02c0f0 comp rx=0 tx=0).stop 2024-03-28T05:21:32.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.497+0000 7ff4e8cf8640 1 -- 172.21.15.23:0/2444566201 shutdown_connections 2024-03-28T05:21:32.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.497+0000 7ff4e8cf8640 1 --2- 172.21.15.23:0/2444566201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4e4102150 0x7ff4e41089e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:32.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.497+0000 7ff4e8cf8640 1 --2- 172.21.15.23:0/2444566201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4e4101790 0x7ff4e4101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:32.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.497+0000 7ff4e8cf8640 1 --2- 172.21.15.23:0/2444566201 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4e41005a0 0x7ff4e41009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:32.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.497+0000 7ff4e8cf8640 1 -- 172.21.15.23:0/2444566201 >> 172.21.15.23:0/2444566201 conn(0x7ff4e40fbb30 msgr2=0x7ff4e40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:32.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.498+0000 7ff4e8cf8640 1 -- 172.21.15.23:0/2444566201 shutdown_connections 2024-03-28T05:21:32.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.498+0000 7ff4e8cf8640 1 -- 172.21.15.23:0/2444566201 wait complete. 2024-03-28T05:21:32.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.498+0000 7ff4e8cf8640 1 Processor -- start 2024-03-28T05:21:32.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.498+0000 7ff4e8cf8640 1 -- start start 2024-03-28T05:21:32.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.498+0000 7ff4e8cf8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4e41005a0 0x7ff4e4196850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:32.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.498+0000 7ff4e8cf8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4e4101790 0x7ff4e4196d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:32.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.498+0000 7ff4e8cf8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4e4102150 0x7ff4e419de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:32.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.498+0000 7ff4e8cf8640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e410c200 con 0x7ff4e41005a0 2024-03-28T05:21:32.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.498+0000 7ff4e8cf8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e410c080 con 0x7ff4e4102150 2024-03-28T05:21:32.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.498+0000 7ff4e8cf8640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e410c380 con 0x7ff4e4101790 2024-03-28T05:21:32.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.499+0000 7ff4e259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4e41005a0 0x7ff4e4196850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:32.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.499+0000 7ff4e259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4e41005a0 0x7ff4e4196850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44476/0 (socket says 172.21.15.23:44476) 2024-03-28T05:21:32.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.499+0000 7ff4e259c640 1 -- 172.21.15.23:0/907274730 learned_addr learned my addr 172.21.15.23:0/907274730 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:32.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.499+0000 7ff4e1d9b640 1 --2- 172.21.15.23:0/907274730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4e4101790 0x7ff4e4196d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:32.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.499+0000 7ff4e2d9d640 1 --2- 172.21.15.23:0/907274730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4e4102150 0x7ff4e419de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:32.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.499+0000 7ff4e259c640 1 -- 172.21.15.23:0/907274730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4e4101790 msgr2=0x7ff4e4196d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:32.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.499+0000 7ff4e259c640 1 --2- 172.21.15.23:0/907274730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4e4101790 0x7ff4e4196d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:32.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.499+0000 7ff4e259c640 1 -- 172.21.15.23:0/907274730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4e4102150 msgr2=0x7ff4e419de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:32.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.499+0000 7ff4e259c640 1 --2- 172.21.15.23:0/907274730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4e4102150 0x7ff4e419de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:32.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.499+0000 7ff4e259c640 1 -- 172.21.15.23:0/907274730 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e419e470 con 0x7ff4e41005a0 2024-03-28T05:21:32.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.499+0000 7ff4e259c640 1 --2- 172.21.15.23:0/907274730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4e41005a0 0x7ff4e4196850 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc00e9c0 tx=0x7ff4cc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:32.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.499+0000 7ff4db7fe640 1 -- 172.21.15.23:0/907274730 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc00cd80 con 0x7ff4e41005a0 2024-03-28T05:21:32.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.499+0000 7ff4db7fe640 1 -- 172.21.15.23:0/907274730 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4cc0035f0 con 0x7ff4e41005a0 2024-03-28T05:21:32.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.499+0000 7ff4e8cf8640 1 -- 172.21.15.23:0/907274730 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e419e760 con 0x7ff4e41005a0 2024-03-28T05:21:32.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.499+0000 7ff4db7fe640 1 -- 172.21.15.23:0/907274730 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc010690 con 0x7ff4e41005a0 2024-03-28T05:21:32.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.499+0000 7ff4e8cf8640 1 -- 172.21.15.23:0/907274730 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e419ed00 con 0x7ff4e41005a0 2024-03-28T05:21:32.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.500+0000 7ff4e8cf8640 1 -- 172.21.15.23:0/907274730 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e41078c0 con 0x7ff4e41005a0 2024-03-28T05:21:32.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.501+0000 7ff4db7fe640 1 -- 172.21.15.23:0/907274730 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff4cc0037a0 con 0x7ff4e41005a0 2024-03-28T05:21:32.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.502+0000 7ff4db7fe640 1 --2- 172.21.15.23:0/907274730 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff4d00a2880 0x7ff4d00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:32.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.502+0000 7ff4db7fe640 1 -- 172.21.15.23:0/907274730 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7ff4cc0cefd0 con 0x7ff4e41005a0 2024-03-28T05:21:32.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.502+0000 7ff4e1d9b640 1 --2- 172.21.15.23:0/907274730 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff4d00a2880 0x7ff4d00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:32.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.502+0000 7ff4e1d9b640 1 --2- 172.21.15.23:0/907274730 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff4d00a2880 0x7ff4d00a4d40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4197d60 tx=0x7ff4d400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:32.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.506+0000 7ff4db7fe640 1 -- 172.21.15.23:0/907274730 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff4cc09be30 con 0x7ff4e41005a0 2024-03-28T05:21:32.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:32 smithi098 ceph-mon[36087]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T05:21:32.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:32 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2431095579' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T05:21:32.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:32 smithi098 ceph-mon[36087]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T05:21:32.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:32 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3322444979' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T05:21:32.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:32.730+0000 7ff4e8cf8640 1 -- 172.21.15.23:0/907274730 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7ff4e402d070 con 0x7ff4e41005a0 2024-03-28T05:21:32.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:32 smithi113 ceph-mon[37164]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T05:21:32.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:32 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2431095579' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T05:21:32.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:32 smithi113 ceph-mon[37164]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T05:21:32.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:32 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3322444979' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T05:21:32.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:32 smithi023 ceph-mon[31954]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T05:21:32.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:32 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2431095579' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T05:21:32.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:32 smithi023 ceph-mon[31954]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T05:21:32.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:32 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3322444979' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T05:21:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.381+0000 7ff4db7fe640 1 -- 172.21.15.23:0/907274730 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v20) v1 ==== 115+0+0 (secure 0 0 0) 0x7ff4cc09b730 con 0x7ff4e41005a0 2024-03-28T05:21:33.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.384+0000 7ff4e8cf8640 1 -- 172.21.15.23:0/907274730 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff4d00a2880 msgr2=0x7ff4d00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:33.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.384+0000 7ff4e8cf8640 1 --2- 172.21.15.23:0/907274730 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff4d00a2880 0x7ff4d00a4d40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4197d60 tx=0x7ff4d400b390 comp rx=0 tx=0).stop 2024-03-28T05:21:33.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.384+0000 7ff4e8cf8640 1 -- 172.21.15.23:0/907274730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4e41005a0 msgr2=0x7ff4e4196850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:33.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.384+0000 7ff4e8cf8640 1 --2- 172.21.15.23:0/907274730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4e41005a0 0x7ff4e4196850 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc00e9c0 tx=0x7ff4cc00ee80 comp rx=0 tx=0).stop 2024-03-28T05:21:33.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.384+0000 7ff4e8cf8640 1 -- 172.21.15.23:0/907274730 shutdown_connections 2024-03-28T05:21:33.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.384+0000 7ff4e8cf8640 1 --2- 172.21.15.23:0/907274730 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff4d00a2880 0x7ff4d00a4d40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:33.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.384+0000 7ff4e8cf8640 1 --2- 172.21.15.23:0/907274730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4e4102150 0x7ff4e419de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:33.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.384+0000 7ff4e8cf8640 1 --2- 172.21.15.23:0/907274730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4e4101790 0x7ff4e4196d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:33.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.384+0000 7ff4e8cf8640 1 --2- 172.21.15.23:0/907274730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4e41005a0 0x7ff4e4196850 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:33.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.384+0000 7ff4e8cf8640 1 -- 172.21.15.23:0/907274730 >> 172.21.15.23:0/907274730 conn(0x7ff4e40fbb30 msgr2=0x7ff4e4104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:33.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.384+0000 7ff4e8cf8640 1 -- 172.21.15.23:0/907274730 shutdown_connections 2024-03-28T05:21:33.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.384+0000 7ff4e8cf8640 1 -- 172.21.15.23:0/907274730 wait complete. 2024-03-28T05:21:33.397 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-03-28T05:21:33.397 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-28T05:21:33.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.509+0000 7f6c0ff5b640 1 -- 172.21.15.23:0/3867803281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c081005a0 msgr2=0x7f6c081009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:33.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.509+0000 7f6c0ff5b640 1 --2- 172.21.15.23:0/3867803281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c081005a0 0x7f6c081009c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009900 tx=0x7f6c0402bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:33.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.509+0000 7f6c0ff5b640 1 -- 172.21.15.23:0/3867803281 shutdown_connections 2024-03-28T05:21:33.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.509+0000 7f6c0ff5b640 1 --2- 172.21.15.23:0/3867803281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c08102150 0x7f6c081089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:33.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.510+0000 7f6c0ff5b640 1 --2- 172.21.15.23:0/3867803281 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c08101790 0x7f6c08101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:33.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.510+0000 7f6c0ff5b640 1 --2- 172.21.15.23:0/3867803281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c081005a0 0x7f6c081009c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:33.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.510+0000 7f6c0ff5b640 1 -- 172.21.15.23:0/3867803281 >> 172.21.15.23:0/3867803281 conn(0x7f6c080fbb30 msgr2=0x7f6c080fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:33.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.510+0000 7f6c0ff5b640 1 -- 172.21.15.23:0/3867803281 shutdown_connections 2024-03-28T05:21:33.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.510+0000 7f6c0ff5b640 1 -- 172.21.15.23:0/3867803281 wait complete. 2024-03-28T05:21:33.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.510+0000 7f6c0ff5b640 1 Processor -- start 2024-03-28T05:21:33.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.510+0000 7f6c0ff5b640 1 -- start start 2024-03-28T05:21:33.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.510+0000 7f6c0ff5b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c081005a0 0x7f6c08197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:33.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.510+0000 7f6c0ff5b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c08101790 0x7f6c08197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:33.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.510+0000 7f6c0ff5b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c08102150 0x7f6c0819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:33.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.510+0000 7f6c0ff5b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0810c260 con 0x7f6c081005a0 2024-03-28T05:21:33.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.510+0000 7f6c0ff5b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0810c0e0 con 0x7f6c08102150 2024-03-28T05:21:33.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.510+0000 7f6c0ff5b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0810c3e0 con 0x7f6c08101790 2024-03-28T05:21:33.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.510+0000 7f6c0dcf7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c081005a0 0x7f6c08197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:33.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.510+0000 7f6c0dcf7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c081005a0 0x7f6c08197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53208/0 (socket says 172.21.15.23:53208) 2024-03-28T05:21:33.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.510+0000 7f6c0dcf7640 1 -- 172.21.15.23:0/129269226 learned_addr learned my addr 172.21.15.23:0/129269226 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:33.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.511+0000 7f6c0d4f6640 1 --2- 172.21.15.23:0/129269226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c08101790 0x7f6c08197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:33.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.511+0000 7f6c0e4f8640 1 --2- 172.21.15.23:0/129269226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c08102150 0x7f6c0819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:33.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.511+0000 7f6c0dcf7640 1 -- 172.21.15.23:0/129269226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c08101790 msgr2=0x7f6c08197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:33.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.511+0000 7f6c0dcf7640 1 --2- 172.21.15.23:0/129269226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c08101790 0x7f6c08197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:33.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.511+0000 7f6c0dcf7640 1 -- 172.21.15.23:0/129269226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c08102150 msgr2=0x7f6c0819efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:33.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.511+0000 7f6c0dcf7640 1 --2- 172.21.15.23:0/129269226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c08102150 0x7f6c0819efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:33.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.511+0000 7f6c0dcf7640 1 -- 172.21.15.23:0/129269226 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0819f650 con 0x7f6c081005a0 2024-03-28T05:21:33.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.511+0000 7f6c0d4f6640 1 --2- 172.21.15.23:0/129269226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c08101790 0x7f6c08197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:21:33.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.511+0000 7f6c0dcf7640 1 --2- 172.21.15.23:0/129269226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c081005a0 0x7f6c08197a30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6c04000c00 tx=0x7f6c0402c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:33.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.511+0000 7f6bfeffd640 1 -- 172.21.15.23:0/129269226 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c04043020 con 0x7f6c081005a0 2024-03-28T05:21:33.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.511+0000 7f6bfeffd640 1 -- 172.21.15.23:0/129269226 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6c040054e0 con 0x7f6c081005a0 2024-03-28T05:21:33.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.511+0000 7f6c0ff5b640 1 -- 172.21.15.23:0/129269226 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c0819f8e0 con 0x7f6c081005a0 2024-03-28T05:21:33.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.511+0000 7f6bfeffd640 1 -- 172.21.15.23:0/129269226 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c04003ce0 con 0x7f6c081005a0 2024-03-28T05:21:33.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.511+0000 7f6c0ff5b640 1 -- 172.21.15.23:0/129269226 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c0819fe50 con 0x7f6c081005a0 2024-03-28T05:21:33.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.512+0000 7f6c0ff5b640 1 -- 172.21.15.23:0/129269226 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c081a0020 con 0x7f6c081005a0 2024-03-28T05:21:33.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.514+0000 7f6bfeffd640 1 -- 172.21.15.23:0/129269226 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6c04046020 con 0x7f6c081005a0 2024-03-28T05:21:33.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.514+0000 7f6bfeffd640 1 --2- 172.21.15.23:0/129269226 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f6bf40a2880 0x7f6bf40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:33.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.515+0000 7f6c0d4f6640 1 --2- 172.21.15.23:0/129269226 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f6bf40a2880 0x7f6bf40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:33.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.515+0000 7f6bfeffd640 1 -- 172.21.15.23:0/129269226 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f6c040f1270 con 0x7f6c081005a0 2024-03-28T05:21:33.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.515+0000 7f6c0d4f6640 1 --2- 172.21.15.23:0/129269226 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f6bf40a2880 0x7f6bf40a4d40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6c08198f40 tx=0x7f6bf8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:33.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.518+0000 7f6bfeffd640 1 -- 172.21.15.23:0/129269226 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6c0404d020 con 0x7f6c081005a0 2024-03-28T05:21:33.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.729+0000 7f6c0ff5b640 1 -- 172.21.15.23:0/129269226 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c080fdd00 con 0x7f6bf40a2880 2024-03-28T05:21:33.733 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:33 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3322444979' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T05:21:33.733 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:33 smithi023 ceph-mon[31954]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T05:21:33.733 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:33 smithi023 ceph-mon[31954]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-03-28T05:21:33.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:33 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/907274730' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T05:21:33.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:33 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3322444979' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T05:21:33.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:33 smithi113 ceph-mon[37164]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T05:21:33.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:33 smithi113 ceph-mon[37164]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-03-28T05:21:33.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:33 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/907274730' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T05:21:33.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.746+0000 7f6bfeffd640 1 -- 172.21.15.23:0/129269226 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6c080fdd00 con 0x7f6bf40a2880 2024-03-28T05:21:33.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.748+0000 7f6c0ff5b640 1 -- 172.21.15.23:0/129269226 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f6bf40a2880 msgr2=0x7f6bf40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:33.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.748+0000 7f6c0ff5b640 1 --2- 172.21.15.23:0/129269226 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f6bf40a2880 0x7f6bf40a4d40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6c08198f40 tx=0x7f6bf8005f90 comp rx=0 tx=0).stop 2024-03-28T05:21:33.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.748+0000 7f6c0ff5b640 1 -- 172.21.15.23:0/129269226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c081005a0 msgr2=0x7f6c08197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:33.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.748+0000 7f6c0ff5b640 1 --2- 172.21.15.23:0/129269226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c081005a0 0x7f6c08197a30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6c04000c00 tx=0x7f6c0402c6c0 comp rx=0 tx=0).stop 2024-03-28T05:21:33.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.748+0000 7f6c0ff5b640 1 -- 172.21.15.23:0/129269226 shutdown_connections 2024-03-28T05:21:33.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.748+0000 7f6c0ff5b640 1 --2- 172.21.15.23:0/129269226 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f6bf40a2880 0x7f6bf40a4d40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:33.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.748+0000 7f6c0ff5b640 1 --2- 172.21.15.23:0/129269226 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c08102150 0x7f6c0819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:33.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.748+0000 7f6c0ff5b640 1 --2- 172.21.15.23:0/129269226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c08101790 0x7f6c08197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:33.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.748+0000 7f6c0ff5b640 1 --2- 172.21.15.23:0/129269226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c081005a0 0x7f6c08197a30 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:33.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.748+0000 7f6c0ff5b640 1 -- 172.21.15.23:0/129269226 >> 172.21.15.23:0/129269226 conn(0x7f6c080fbb30 msgr2=0x7f6c080fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:33.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.749+0000 7f6c0ff5b640 1 -- 172.21.15.23:0/129269226 shutdown_connections 2024-03-28T05:21:33.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.749+0000 7f6c0ff5b640 1 -- 172.21.15.23:0/129269226 wait complete. 2024-03-28T05:21:33.759 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-28T05:21:33.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:33 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3322444979' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T05:21:33.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:33 smithi098 ceph-mon[36087]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-03-28T05:21:33.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:33 smithi098 ceph-mon[36087]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-03-28T05:21:33.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:33 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/907274730' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T05:21:33.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.867+0000 7fdf2bfff640 1 -- 172.21.15.23:0/2253229496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf2c1004a0 msgr2=0x7fdf2c1008c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:33.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.867+0000 7fdf2bfff640 1 --2- 172.21.15.23:0/2253229496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf2c1004a0 0x7fdf2c1008c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c009900 tx=0x7fdf1c02bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:33.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.867+0000 7fdf2bfff640 1 -- 172.21.15.23:0/2253229496 shutdown_connections 2024-03-28T05:21:33.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.867+0000 7fdf2bfff640 1 --2- 172.21.15.23:0/2253229496 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf2c102050 0x7fdf2c1088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:33.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.867+0000 7fdf2bfff640 1 --2- 172.21.15.23:0/2253229496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf2c101690 0x7fdf2c101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:33.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.867+0000 7fdf2bfff640 1 --2- 172.21.15.23:0/2253229496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf2c1004a0 0x7fdf2c1008c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:33.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.867+0000 7fdf2bfff640 1 -- 172.21.15.23:0/2253229496 >> 172.21.15.23:0/2253229496 conn(0x7fdf2c0fba10 msgr2=0x7fdf2c0fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:33.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.867+0000 7fdf2bfff640 1 -- 172.21.15.23:0/2253229496 shutdown_connections 2024-03-28T05:21:33.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.867+0000 7fdf2bfff640 1 -- 172.21.15.23:0/2253229496 wait complete. 2024-03-28T05:21:33.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.868+0000 7fdf2bfff640 1 Processor -- start 2024-03-28T05:21:33.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.868+0000 7fdf2bfff640 1 -- start start 2024-03-28T05:21:33.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.868+0000 7fdf2bfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf2c1004a0 0x7fdf2c06a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:33.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.868+0000 7fdf2bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf2c101690 0x7fdf2c19c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:33.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.868+0000 7fdf2bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf2c102050 0x7fdf2c19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:33.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.868+0000 7fdf2bfff640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c10c130 con 0x7fdf2c102050 2024-03-28T05:21:33.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.868+0000 7fdf2bfff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c10bfb0 con 0x7fdf2c101690 2024-03-28T05:21:33.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.868+0000 7fdf2bfff640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c10c2b0 con 0x7fdf2c1004a0 2024-03-28T05:21:33.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.868+0000 7fdf2b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf2c102050 0x7fdf2c19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:33.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.868+0000 7fdf2b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf2c102050 0x7fdf2c19ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53252/0 (socket says 172.21.15.23:53252) 2024-03-28T05:21:33.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.868+0000 7fdf2b7fe640 1 -- 172.21.15.23:0/2083816131 learned_addr learned my addr 172.21.15.23:0/2083816131 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:33.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.868+0000 7fdf2affd640 1 --2- 172.21.15.23:0/2083816131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf2c1004a0 0x7fdf2c06a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:33.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.868+0000 7fdf2b7fe640 1 -- 172.21.15.23:0/2083816131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf2c1004a0 msgr2=0x7fdf2c06a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:33.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.868+0000 7fdf2b7fe640 1 --2- 172.21.15.23:0/2083816131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf2c1004a0 0x7fdf2c06a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:33.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.868+0000 7fdf2b7fe640 1 -- 172.21.15.23:0/2083816131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf2c101690 msgr2=0x7fdf2c19c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:33.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.868+0000 7fdf2b7fe640 1 --2- 172.21.15.23:0/2083816131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf2c101690 0x7fdf2c19c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:33.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.868+0000 7fdf2b7fe640 1 -- 172.21.15.23:0/2083816131 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2c19f500 con 0x7fdf2c102050 2024-03-28T05:21:33.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.868+0000 7fdf2b7fe640 1 --2- 172.21.15.23:0/2083816131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf2c102050 0x7fdf2c19ec40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdf2400eb10 tx=0x7fdf2400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:33.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.868+0000 7fdf28ff9640 1 -- 172.21.15.23:0/2083816131 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf24004b20 con 0x7fdf2c102050 2024-03-28T05:21:33.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.869+0000 7fdf28ff9640 1 -- 172.21.15.23:0/2083816131 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdf24004cc0 con 0x7fdf2c102050 2024-03-28T05:21:33.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.869+0000 7fdf2bfff640 1 -- 172.21.15.23:0/2083816131 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf2c19f7f0 con 0x7fdf2c102050 2024-03-28T05:21:33.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.869+0000 7fdf28ff9640 1 -- 172.21.15.23:0/2083816131 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf24010760 con 0x7fdf2c102050 2024-03-28T05:21:33.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.869+0000 7fdf2bfff640 1 -- 172.21.15.23:0/2083816131 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf2c19fd90 con 0x7fdf2c102050 2024-03-28T05:21:33.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.869+0000 7fdf2bfff640 1 -- 172.21.15.23:0/2083816131 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf2c107840 con 0x7fdf2c102050 2024-03-28T05:21:33.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.871+0000 7fdf28ff9640 1 -- 172.21.15.23:0/2083816131 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdf24005000 con 0x7fdf2c102050 2024-03-28T05:21:33.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.871+0000 7fdf28ff9640 1 --2- 172.21.15.23:0/2083816131 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fdf180a2880 0x7fdf180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:33.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.871+0000 7fdf2affd640 1 --2- 172.21.15.23:0/2083816131 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fdf180a2880 0x7fdf180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:33.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.871+0000 7fdf28ff9640 1 -- 172.21.15.23:0/2083816131 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fdf240cf720 con 0x7fdf2c102050 2024-03-28T05:21:33.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.872+0000 7fdf2affd640 1 --2- 172.21.15.23:0/2083816131 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fdf180a2880 0x7fdf180a4d40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c000c00 tx=0x7fdf1c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:33.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:33.875+0000 7fdf28ff9640 1 -- 172.21.15.23:0/2083816131 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdf2409c580 con 0x7fdf2c102050 2024-03-28T05:21:34.030 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:21:33 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:21:33.733+0000 7ffb02689700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T05:21:34.030 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:21:33 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:21:33.733+0000 7ffb02689700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T05:21:34.030 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:21:33 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:21:33.733+0000 7ffb02689700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T05:21:34.030 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:21:33 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:21:33.733+0000 7ffb02689700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T05:21:34.030 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:21:33 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:21:33.733+0000 7ffb02689700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T05:21:34.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.086+0000 7fdf2bfff640 1 -- 172.21.15.23:0/2083816131 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf2c061ae0 con 0x7fdf180a2880 2024-03-28T05:21:34.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.107+0000 7fdf28ff9640 1 -- 172.21.15.23:0/2083816131 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdf2c061ae0 con 0x7fdf180a2880 2024-03-28T05:21:34.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.109+0000 7fdf2bfff640 1 -- 172.21.15.23:0/2083816131 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fdf180a2880 msgr2=0x7fdf180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:34.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.109+0000 7fdf2bfff640 1 --2- 172.21.15.23:0/2083816131 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fdf180a2880 0x7fdf180a4d40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c000c00 tx=0x7fdf1c005fd0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.109+0000 7fdf2bfff640 1 -- 172.21.15.23:0/2083816131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf2c102050 msgr2=0x7fdf2c19ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:34.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.109+0000 7fdf2bfff640 1 --2- 172.21.15.23:0/2083816131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf2c102050 0x7fdf2c19ec40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdf2400eb10 tx=0x7fdf2400c2d0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.109+0000 7fdf2bfff640 1 -- 172.21.15.23:0/2083816131 shutdown_connections 2024-03-28T05:21:34.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.109+0000 7fdf2bfff640 1 --2- 172.21.15.23:0/2083816131 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fdf180a2880 0x7fdf180a4d40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.109+0000 7fdf2bfff640 1 --2- 172.21.15.23:0/2083816131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf2c102050 0x7fdf2c19ec40 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.109+0000 7fdf2bfff640 1 --2- 172.21.15.23:0/2083816131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf2c101690 0x7fdf2c19c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.109+0000 7fdf2bfff640 1 --2- 172.21.15.23:0/2083816131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf2c1004a0 0x7fdf2c06a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.109+0000 7fdf2bfff640 1 -- 172.21.15.23:0/2083816131 >> 172.21.15.23:0/2083816131 conn(0x7fdf2c0fba10 msgr2=0x7fdf2c104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:34.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.109+0000 7fdf2bfff640 1 -- 172.21.15.23:0/2083816131 shutdown_connections 2024-03-28T05:21:34.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.109+0000 7fdf2bfff640 1 -- 172.21.15.23:0/2083816131 wait complete. 2024-03-28T05:21:34.119 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-28T05:21:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.227+0000 7fb923509640 1 -- 172.21.15.23:0/3641258062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb91c1000d0 msgr2=0x7fb91c109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.227+0000 7fb923509640 1 --2- 172.21.15.23:0/3641258062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb91c1000d0 0x7fb91c109c80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb914009ca0 tx=0x7fb91402c0f0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.227+0000 7fb923509640 1 -- 172.21.15.23:0/3641258062 shutdown_connections 2024-03-28T05:21:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.227+0000 7fb923509640 1 --2- 172.21.15.23:0/3641258062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb91c1000d0 0x7fb91c109c80 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.227+0000 7fb923509640 1 --2- 172.21.15.23:0/3641258062 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb91c0ff710 0x7fb91c0ffb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.227+0000 7fb923509640 1 --2- 172.21.15.23:0/3641258062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb91c102ca0 0x7fb91c1030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.227+0000 7fb923509640 1 -- 172.21.15.23:0/3641258062 >> 172.21.15.23:0/3641258062 conn(0x7fb91c0fb2b0 msgr2=0x7fb91c0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.227+0000 7fb923509640 1 -- 172.21.15.23:0/3641258062 shutdown_connections 2024-03-28T05:21:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.227+0000 7fb923509640 1 -- 172.21.15.23:0/3641258062 wait complete. 2024-03-28T05:21:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.228+0000 7fb923509640 1 Processor -- start 2024-03-28T05:21:34.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.228+0000 7fb923509640 1 -- start start 2024-03-28T05:21:34.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.228+0000 7fb923509640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb91c0ff710 0x7fb91c197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:34.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.228+0000 7fb923509640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb91c1000d0 0x7fb91c197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:34.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.228+0000 7fb923509640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb91c102ca0 0x7fb91c19ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:34.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.228+0000 7fb923509640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91c10d350 con 0x7fb91c0ff710 2024-03-28T05:21:34.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.228+0000 7fb923509640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91c10d1d0 con 0x7fb91c1000d0 2024-03-28T05:21:34.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.228+0000 7fb923509640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91c10d4d0 con 0x7fb91c102ca0 2024-03-28T05:21:34.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.228+0000 7fb9212a5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb91c0ff710 0x7fb91c197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:34.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.228+0000 7fb9212a5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb91c0ff710 0x7fb91c197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53278/0 (socket says 172.21.15.23:53278) 2024-03-28T05:21:34.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.228+0000 7fb9212a5640 1 -- 172.21.15.23:0/3482277090 learned_addr learned my addr 172.21.15.23:0/3482277090 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:34.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.228+0000 7fb921aa6640 1 --2- 172.21.15.23:0/3482277090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb91c102ca0 0x7fb91c19ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:34.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.228+0000 7fb920aa4640 1 --2- 172.21.15.23:0/3482277090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb91c1000d0 0x7fb91c197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:34.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.228+0000 7fb9212a5640 1 -- 172.21.15.23:0/3482277090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb91c102ca0 msgr2=0x7fb91c19ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:34.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.228+0000 7fb9212a5640 1 --2- 172.21.15.23:0/3482277090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb91c102ca0 0x7fb91c19ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.228+0000 7fb9212a5640 1 -- 172.21.15.23:0/3482277090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb91c1000d0 msgr2=0x7fb91c197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:34.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.228+0000 7fb9212a5640 1 --2- 172.21.15.23:0/3482277090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb91c1000d0 0x7fb91c197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.228+0000 7fb9212a5640 1 -- 172.21.15.23:0/3482277090 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb91c19f5b0 con 0x7fb91c0ff710 2024-03-28T05:21:34.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.228+0000 7fb920aa4640 1 --2- 172.21.15.23:0/3482277090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb91c1000d0 0x7fb91c197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:21:34.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.228+0000 7fb9212a5640 1 --2- 172.21.15.23:0/3482277090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb91c0ff710 0x7fb91c197990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb90c00d950 tx=0x7fb90c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:34.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.229+0000 7fb9127fc640 1 -- 172.21.15.23:0/3482277090 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c017020 con 0x7fb91c0ff710 2024-03-28T05:21:34.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.229+0000 7fb9127fc640 1 -- 172.21.15.23:0/3482277090 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb90c009c30 con 0x7fb91c0ff710 2024-03-28T05:21:34.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.229+0000 7fb9127fc640 1 -- 172.21.15.23:0/3482277090 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c00f720 con 0x7fb91c0ff710 2024-03-28T05:21:34.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.229+0000 7fb923509640 1 -- 172.21.15.23:0/3482277090 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb91c19f8a0 con 0x7fb91c0ff710 2024-03-28T05:21:34.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.229+0000 7fb923509640 1 -- 172.21.15.23:0/3482277090 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb91c19fdf0 con 0x7fb91c0ff710 2024-03-28T05:21:34.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.230+0000 7fb923509640 1 -- 172.21.15.23:0/3482277090 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb91c108b60 con 0x7fb91c0ff710 2024-03-28T05:21:34.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.231+0000 7fb9127fc640 1 -- 172.21.15.23:0/3482277090 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb90c01a020 con 0x7fb91c0ff710 2024-03-28T05:21:34.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.231+0000 7fb9127fc640 1 --2- 172.21.15.23:0/3482277090 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb9080a2800 0x7fb9080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:34.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.231+0000 7fb920aa4640 1 --2- 172.21.15.23:0/3482277090 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb9080a2800 0x7fb9080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:34.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.231+0000 7fb9127fc640 1 -- 172.21.15.23:0/3482277090 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb90c0ceba0 con 0x7fb91c0ff710 2024-03-28T05:21:34.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.232+0000 7fb920aa4640 1 --2- 172.21.15.23:0/3482277090 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb9080a2800 0x7fb9080a4cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb91c198ea0 tx=0x7fb918005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:34.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.236+0000 7fb9127fc640 1 -- 172.21.15.23:0/3482277090 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb90c021020 con 0x7fb91c0ff710 2024-03-28T05:21:34.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.449+0000 7fb923509640 1 -- 172.21.15.23:0/3482277090 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7fb91c061ae0 con 0x7fb9080a2800 2024-03-28T05:21:34.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.456+0000 7fb9127fc640 1 -- 172.21.15.23:0/3482277090 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fb91c061ae0 con 0x7fb9080a2800 2024-03-28T05:21:34.455 INFO:teuthology.orchestra.run.smithi023.stdout:/volumes/qa/sv_0/b626edc4-b1ad-4390-8e17-5d1c1f456de1 2024-03-28T05:21:34.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.458+0000 7fb923509640 1 -- 172.21.15.23:0/3482277090 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb9080a2800 msgr2=0x7fb9080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:34.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.458+0000 7fb923509640 1 --2- 172.21.15.23:0/3482277090 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb9080a2800 0x7fb9080a4cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb91c198ea0 tx=0x7fb918005f90 comp rx=0 tx=0).stop 2024-03-28T05:21:34.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.458+0000 7fb923509640 1 -- 172.21.15.23:0/3482277090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb91c0ff710 msgr2=0x7fb91c197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:34.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.458+0000 7fb923509640 1 --2- 172.21.15.23:0/3482277090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb91c0ff710 0x7fb91c197990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb90c00d950 tx=0x7fb90c00de10 comp rx=0 tx=0).stop 2024-03-28T05:21:34.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.458+0000 7fb923509640 1 -- 172.21.15.23:0/3482277090 shutdown_connections 2024-03-28T05:21:34.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.458+0000 7fb923509640 1 --2- 172.21.15.23:0/3482277090 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb9080a2800 0x7fb9080a4cc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.458+0000 7fb923509640 1 --2- 172.21.15.23:0/3482277090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb91c102ca0 0x7fb91c19ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.458+0000 7fb923509640 1 --2- 172.21.15.23:0/3482277090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb91c1000d0 0x7fb91c197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.458+0000 7fb923509640 1 --2- 172.21.15.23:0/3482277090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb91c0ff710 0x7fb91c197990 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.458+0000 7fb923509640 1 -- 172.21.15.23:0/3482277090 >> 172.21.15.23:0/3482277090 conn(0x7fb91c0fb2b0 msgr2=0x7fb91c105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:34.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.458+0000 7fb923509640 1 -- 172.21.15.23:0/3482277090 shutdown_connections 2024-03-28T05:21:34.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.458+0000 7fb923509640 1 -- 172.21.15.23:0/3482277090 wait complete. 2024-03-28T05:21:34.467 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-28T05:21:34.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.576+0000 7fb961ccc640 1 -- 172.21.15.23:0/3326913850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb95c102160 msgr2=0x7fb95c1089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:34.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.576+0000 7fb961ccc640 1 --2- 172.21.15.23:0/3326913850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb95c102160 0x7fb95c1089f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb954009ca0 tx=0x7fb95402c0f0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.576+0000 7fb961ccc640 1 -- 172.21.15.23:0/3326913850 shutdown_connections 2024-03-28T05:21:34.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.576+0000 7fb961ccc640 1 --2- 172.21.15.23:0/3326913850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb95c102160 0x7fb95c1089f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.576+0000 7fb961ccc640 1 --2- 172.21.15.23:0/3326913850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb95c1017a0 0x7fb95c101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.576+0000 7fb961ccc640 1 --2- 172.21.15.23:0/3326913850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb95c1005b0 0x7fb95c1009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.576+0000 7fb961ccc640 1 -- 172.21.15.23:0/3326913850 >> 172.21.15.23:0/3326913850 conn(0x7fb95c0fbb40 msgr2=0x7fb95c0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:34.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.576+0000 7fb961ccc640 1 -- 172.21.15.23:0/3326913850 shutdown_connections 2024-03-28T05:21:34.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.576+0000 7fb961ccc640 1 -- 172.21.15.23:0/3326913850 wait complete. 2024-03-28T05:21:34.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.576+0000 7fb961ccc640 1 Processor -- start 2024-03-28T05:21:34.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.576+0000 7fb961ccc640 1 -- start start 2024-03-28T05:21:34.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb961ccc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb95c1005b0 0x7fb95c197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:34.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb961ccc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb95c1017a0 0x7fb95c197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:34.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb961ccc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb95c102160 0x7fb95c19ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:34.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb961ccc640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c10d350 con 0x7fb95c102160 2024-03-28T05:21:34.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb961ccc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c10d1d0 con 0x7fb95c1017a0 2024-03-28T05:21:34.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb961ccc640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c10d4d0 con 0x7fb95c1005b0 2024-03-28T05:21:34.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb95bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb95c102160 0x7fb95c19ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:34.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb95bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb95c102160 0x7fb95c19ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53282/0 (socket says 172.21.15.23:53282) 2024-03-28T05:21:34.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb95bfff640 1 -- 172.21.15.23:0/3268356643 learned_addr learned my addr 172.21.15.23:0/3268356643 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:34.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb95b7fe640 1 --2- 172.21.15.23:0/3268356643 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb95c1005b0 0x7fb95c197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:34.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb95affd640 1 --2- 172.21.15.23:0/3268356643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb95c1017a0 0x7fb95c197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:34.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb95bfff640 1 -- 172.21.15.23:0/3268356643 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb95c1005b0 msgr2=0x7fb95c197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:34.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb95bfff640 1 --2- 172.21.15.23:0/3268356643 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb95c1005b0 0x7fb95c197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb95bfff640 1 -- 172.21.15.23:0/3268356643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb95c1017a0 msgr2=0x7fb95c197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:34.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb95bfff640 1 --2- 172.21.15.23:0/3268356643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb95c1017a0 0x7fb95c197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb95bfff640 1 -- 172.21.15.23:0/3268356643 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95c19f570 con 0x7fb95c102160 2024-03-28T05:21:34.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb95affd640 1 --2- 172.21.15.23:0/3268356643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb95c1017a0 0x7fb95c197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:21:34.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb95bfff640 1 --2- 172.21.15.23:0/3268356643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb95c102160 0x7fb95c19ef00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb954037040 tx=0x7fb954032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:34.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb958ff9640 1 -- 172.21.15.23:0/3268356643 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb954041020 con 0x7fb95c102160 2024-03-28T05:21:34.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb958ff9640 1 -- 172.21.15.23:0/3268356643 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb954034d90 con 0x7fb95c102160 2024-03-28T05:21:34.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb961ccc640 1 -- 172.21.15.23:0/3268356643 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb95c19f800 con 0x7fb95c102160 2024-03-28T05:21:34.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb958ff9640 1 -- 172.21.15.23:0/3268356643 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb954039040 con 0x7fb95c102160 2024-03-28T05:21:34.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.577+0000 7fb961ccc640 1 -- 172.21.15.23:0/3268356643 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb95c19fcf0 con 0x7fb95c102160 2024-03-28T05:21:34.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.578+0000 7fb961ccc640 1 -- 172.21.15.23:0/3268356643 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb95c1078d0 con 0x7fb95c102160 2024-03-28T05:21:34.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.579+0000 7fb958ff9640 1 -- 172.21.15.23:0/3268356643 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb954044020 con 0x7fb95c102160 2024-03-28T05:21:34.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.580+0000 7fb958ff9640 1 --2- 172.21.15.23:0/3268356643 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb9480a2800 0x7fb9480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:34.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.580+0000 7fb95b7fe640 1 --2- 172.21.15.23:0/3268356643 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb9480a2800 0x7fb9480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:34.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.580+0000 7fb958ff9640 1 -- 172.21.15.23:0/3268356643 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb9540f0c50 con 0x7fb95c102160 2024-03-28T05:21:34.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.581+0000 7fb95b7fe640 1 --2- 172.21.15.23:0/3268356643 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb9480a2800 0x7fb9480a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb95c101600 tx=0x7fb944005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:34.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.585+0000 7fb958ff9640 1 -- 172.21.15.23:0/3268356643 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb9540f6020 con 0x7fb95c102160 2024-03-28T05:21:34.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:34 smithi113 ceph-mon[37164]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 341 B/s wr, 0 op/s 2024-03-28T05:21:34.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:34 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/907274730' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T05:21:34.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:34 smithi113 ceph-mon[37164]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-03-28T05:21:34.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:34 smithi023 ceph-mon[31954]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 341 B/s wr, 0 op/s 2024-03-28T05:21:34.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:34 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/907274730' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T05:21:34.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:34 smithi023 ceph-mon[31954]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-03-28T05:21:34.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.795+0000 7fb961ccc640 1 -- 172.21.15.23:0/3268356643 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fb95c061ae0 con 0x7fb9480a2800 2024-03-28T05:21:34.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.818+0000 7fb958ff9640 1 -- 172.21.15.23:0/3268356643 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb95c061ae0 con 0x7fb9480a2800 2024-03-28T05:21:34.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.820+0000 7fb961ccc640 1 -- 172.21.15.23:0/3268356643 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb9480a2800 msgr2=0x7fb9480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:34.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.820+0000 7fb961ccc640 1 --2- 172.21.15.23:0/3268356643 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb9480a2800 0x7fb9480a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb95c101600 tx=0x7fb944005f90 comp rx=0 tx=0).stop 2024-03-28T05:21:34.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.820+0000 7fb961ccc640 1 -- 172.21.15.23:0/3268356643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb95c102160 msgr2=0x7fb95c19ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:34.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.820+0000 7fb961ccc640 1 --2- 172.21.15.23:0/3268356643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb95c102160 0x7fb95c19ef00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb954037040 tx=0x7fb954032f20 comp rx=0 tx=0).stop 2024-03-28T05:21:34.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.820+0000 7fb961ccc640 1 -- 172.21.15.23:0/3268356643 shutdown_connections 2024-03-28T05:21:34.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.820+0000 7fb961ccc640 1 --2- 172.21.15.23:0/3268356643 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb9480a2800 0x7fb9480a4cc0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.820+0000 7fb961ccc640 1 --2- 172.21.15.23:0/3268356643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb95c102160 0x7fb95c19ef00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.820+0000 7fb961ccc640 1 --2- 172.21.15.23:0/3268356643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb95c1017a0 0x7fb95c197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.820+0000 7fb961ccc640 1 --2- 172.21.15.23:0/3268356643 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb95c1005b0 0x7fb95c197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.820+0000 7fb961ccc640 1 -- 172.21.15.23:0/3268356643 >> 172.21.15.23:0/3268356643 conn(0x7fb95c0fbb40 msgr2=0x7fb95c104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:34.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.820+0000 7fb961ccc640 1 -- 172.21.15.23:0/3268356643 shutdown_connections 2024-03-28T05:21:34.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.820+0000 7fb961ccc640 1 -- 172.21.15.23:0/3268356643 wait complete. 2024-03-28T05:21:34.830 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-28T05:21:34.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:34 smithi098 ceph-mon[36087]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 341 B/s wr, 0 op/s 2024-03-28T05:21:34.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:34 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/907274730' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T05:21:34.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:34 smithi098 ceph-mon[36087]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-03-28T05:21:34.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.939+0000 7f34c2659640 1 -- 172.21.15.23:0/705916264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34bc1073e0 msgr2=0x7f34bc1097d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:34.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.939+0000 7f34c2659640 1 --2- 172.21.15.23:0/705916264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34bc1073e0 0x7f34bc1097d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f34b4009ca0 tx=0x7f34b402c0f0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.939+0000 7f34c2659640 1 -- 172.21.15.23:0/705916264 shutdown_connections 2024-03-28T05:21:34.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.939+0000 7f34c2659640 1 --2- 172.21.15.23:0/705916264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34bc1073e0 0x7f34bc1097d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.939+0000 7f34c2659640 1 --2- 172.21.15.23:0/705916264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34bc104ab0 0x7f34bc106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.939+0000 7f34c2659640 1 --2- 172.21.15.23:0/705916264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34bc102180 0x7f34bc104570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.939+0000 7f34c2659640 1 -- 172.21.15.23:0/705916264 >> 172.21.15.23:0/705916264 conn(0x7f34bc0fbb30 msgr2=0x7f34bc0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:34.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.939+0000 7f34c2659640 1 -- 172.21.15.23:0/705916264 shutdown_connections 2024-03-28T05:21:34.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.939+0000 7f34c2659640 1 -- 172.21.15.23:0/705916264 wait complete. 2024-03-28T05:21:34.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.939+0000 7f34c2659640 1 Processor -- start 2024-03-28T05:21:34.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.939+0000 7f34c2659640 1 -- start start 2024-03-28T05:21:34.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.940+0000 7f34c2659640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34bc102180 0x7f34bc1979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:34.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.940+0000 7f34c2659640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34bc104ab0 0x7f34bc197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:34.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.940+0000 7f34c2659640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34bc1073e0 0x7f34bc19ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:34.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.940+0000 7f34c2659640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc10d340 con 0x7f34bc1073e0 2024-03-28T05:21:34.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.940+0000 7f34c2659640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc10d1c0 con 0x7f34bc104ab0 2024-03-28T05:21:34.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.940+0000 7f34c2659640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc10d4c0 con 0x7f34bc102180 2024-03-28T05:21:34.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.940+0000 7f34c0bf6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34bc1073e0 0x7f34bc19ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:34.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.940+0000 7f34c0bf6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34bc1073e0 0x7f34bc19ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53306/0 (socket says 172.21.15.23:53306) 2024-03-28T05:21:34.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.940+0000 7f34c0bf6640 1 -- 172.21.15.23:0/1125194727 learned_addr learned my addr 172.21.15.23:0/1125194727 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:34.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.940+0000 7f34bbfff640 1 --2- 172.21.15.23:0/1125194727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34bc102180 0x7f34bc1979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:34.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.940+0000 7f34bb7fe640 1 --2- 172.21.15.23:0/1125194727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34bc104ab0 0x7f34bc197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:34.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.940+0000 7f34c0bf6640 1 -- 172.21.15.23:0/1125194727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34bc102180 msgr2=0x7f34bc1979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:34.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.940+0000 7f34c0bf6640 1 --2- 172.21.15.23:0/1125194727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34bc102180 0x7f34bc1979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.940+0000 7f34c0bf6640 1 -- 172.21.15.23:0/1125194727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34bc104ab0 msgr2=0x7f34bc197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:34.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.940+0000 7f34c0bf6640 1 --2- 172.21.15.23:0/1125194727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34bc104ab0 0x7f34bc197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:34.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.940+0000 7f34c0bf6640 1 -- 172.21.15.23:0/1125194727 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34bc19f5e0 con 0x7f34bc1073e0 2024-03-28T05:21:34.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.940+0000 7f34c0bf6640 1 --2- 172.21.15.23:0/1125194727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34bc1073e0 0x7f34bc19ef70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f34b402c630 tx=0x7f34b4032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:34.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.940+0000 7f34b97fa640 1 -- 172.21.15.23:0/1125194727 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b4041020 con 0x7f34bc1073e0 2024-03-28T05:21:34.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.940+0000 7f34b97fa640 1 -- 172.21.15.23:0/1125194727 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f34b4034cd0 con 0x7f34bc1073e0 2024-03-28T05:21:34.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.940+0000 7f34b97fa640 1 -- 172.21.15.23:0/1125194727 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b4039040 con 0x7f34bc1073e0 2024-03-28T05:21:34.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.940+0000 7f34c2659640 1 -- 172.21.15.23:0/1125194727 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34bc19f870 con 0x7f34bc1073e0 2024-03-28T05:21:34.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.940+0000 7f34c2659640 1 -- 172.21.15.23:0/1125194727 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34bc19fd60 con 0x7f34bc1073e0 2024-03-28T05:21:34.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.941+0000 7f34c2659640 1 -- 172.21.15.23:0/1125194727 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34bc0fd620 con 0x7f34bc1073e0 2024-03-28T05:21:34.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.942+0000 7f34b97fa640 1 -- 172.21.15.23:0/1125194727 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f34b4044020 con 0x7f34bc1073e0 2024-03-28T05:21:34.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.943+0000 7f34b97fa640 1 --2- 172.21.15.23:0/1125194727 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f34a40a2800 0x7f34a40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:34.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.943+0000 7f34bbfff640 1 --2- 172.21.15.23:0/1125194727 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f34a40a2800 0x7f34a40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:34.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.943+0000 7f34b97fa640 1 -- 172.21.15.23:0/1125194727 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f34b40f1080 con 0x7f34bc1073e0 2024-03-28T05:21:34.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.944+0000 7f34bbfff640 1 --2- 172.21.15.23:0/1125194727 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f34a40a2800 0x7f34a40a4cc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009bd0 tx=0x7f34b0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:34.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:34.947+0000 7f34b97fa640 1 -- 172.21.15.23:0/1125194727 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f34b404b020 con 0x7f34bc1073e0 2024-03-28T05:21:35.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.157+0000 7f34c2659640 1 -- 172.21.15.23:0/1125194727 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f34bc061ae0 con 0x7f34a40a2800 2024-03-28T05:21:35.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.163+0000 7f34b97fa640 1 -- 172.21.15.23:0/1125194727 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f34bc061ae0 con 0x7f34a40a2800 2024-03-28T05:21:35.163 INFO:teuthology.orchestra.run.smithi023.stdout:/volumes/qa/sv_1/30671fb5-9586-458a-b54e-fbeb280f6667 2024-03-28T05:21:35.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.165+0000 7f34c2659640 1 -- 172.21.15.23:0/1125194727 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f34a40a2800 msgr2=0x7f34a40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:35.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.165+0000 7f34c2659640 1 --2- 172.21.15.23:0/1125194727 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f34a40a2800 0x7f34a40a4cc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009bd0 tx=0x7f34b0005f90 comp rx=0 tx=0).stop 2024-03-28T05:21:35.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.165+0000 7f34c2659640 1 -- 172.21.15.23:0/1125194727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34bc1073e0 msgr2=0x7f34bc19ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:35.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.165+0000 7f34c2659640 1 --2- 172.21.15.23:0/1125194727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34bc1073e0 0x7f34bc19ef70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f34b402c630 tx=0x7f34b4032ea0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.165+0000 7f34c2659640 1 -- 172.21.15.23:0/1125194727 shutdown_connections 2024-03-28T05:21:35.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.165+0000 7f34c2659640 1 --2- 172.21.15.23:0/1125194727 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f34a40a2800 0x7f34a40a4cc0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.165+0000 7f34c2659640 1 --2- 172.21.15.23:0/1125194727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34bc1073e0 0x7f34bc19ef70 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.165+0000 7f34c2659640 1 --2- 172.21.15.23:0/1125194727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34bc104ab0 0x7f34bc197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.165+0000 7f34c2659640 1 --2- 172.21.15.23:0/1125194727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34bc102180 0x7f34bc1979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.165+0000 7f34c2659640 1 -- 172.21.15.23:0/1125194727 >> 172.21.15.23:0/1125194727 conn(0x7f34bc0fbb30 msgr2=0x7f34bc102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:35.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.166+0000 7f34c2659640 1 -- 172.21.15.23:0/1125194727 shutdown_connections 2024-03-28T05:21:35.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.166+0000 7f34c2659640 1 -- 172.21.15.23:0/1125194727 wait complete. 2024-03-28T05:21:35.176 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:21:35.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.280+0000 7f762b8ff640 1 -- 172.21.15.23:0/3498985399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7624101830 msgr2=0x7f7624101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:35.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.280+0000 7f762b8ff640 1 --2- 172.21.15.23:0/3498985399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7624101830 0x7f7624101cb0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7614009900 tx=0x7f761402bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:35.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.280+0000 7f762b8ff640 1 -- 172.21.15.23:0/3498985399 shutdown_connections 2024-03-28T05:21:35.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.280+0000 7f762b8ff640 1 --2- 172.21.15.23:0/3498985399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76241021f0 0x7f7624108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.280+0000 7f762b8ff640 1 --2- 172.21.15.23:0/3498985399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7624101830 0x7f7624101cb0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.280+0000 7f762b8ff640 1 --2- 172.21.15.23:0/3498985399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7624100640 0x7f7624100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.280+0000 7f762b8ff640 1 -- 172.21.15.23:0/3498985399 >> 172.21.15.23:0/3498985399 conn(0x7f76240fbbd0 msgr2=0x7f76240fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:35.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.280+0000 7f762b8ff640 1 -- 172.21.15.23:0/3498985399 shutdown_connections 2024-03-28T05:21:35.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.280+0000 7f762b8ff640 1 -- 172.21.15.23:0/3498985399 wait complete. 2024-03-28T05:21:35.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.280+0000 7f762b8ff640 1 Processor -- start 2024-03-28T05:21:35.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.280+0000 7f762b8ff640 1 -- start start 2024-03-28T05:21:35.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.281+0000 7f762b8ff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7624100640 0x7f7624197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:35.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.281+0000 7f762b8ff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7624101830 0x7f7624197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:35.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.281+0000 7f762b8ff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76241021f0 0x7f762419f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:35.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.281+0000 7f762b8ff640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762410c230 con 0x7f7624100640 2024-03-28T05:21:35.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.281+0000 7f762b8ff640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762410c0b0 con 0x7f76241021f0 2024-03-28T05:21:35.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.281+0000 7f762b8ff640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762410c3b0 con 0x7f7624101830 2024-03-28T05:21:35.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.281+0000 7f762969b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7624100640 0x7f7624197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:35.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.281+0000 7f762969b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7624100640 0x7f7624197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53328/0 (socket says 172.21.15.23:53328) 2024-03-28T05:21:35.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.281+0000 7f762969b640 1 -- 172.21.15.23:0/1551061551 learned_addr learned my addr 172.21.15.23:0/1551061551 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:35.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.281+0000 7f7628e9a640 1 --2- 172.21.15.23:0/1551061551 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7624101830 0x7f7624197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:35.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.281+0000 7f762969b640 1 -- 172.21.15.23:0/1551061551 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7624101830 msgr2=0x7f7624197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:35.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.281+0000 7f762969b640 1 --2- 172.21.15.23:0/1551061551 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7624101830 0x7f7624197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.281+0000 7f762969b640 1 -- 172.21.15.23:0/1551061551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76241021f0 msgr2=0x7f762419f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:35.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.281+0000 7f762969b640 1 --2- 172.21.15.23:0/1551061551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76241021f0 0x7f762419f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.281+0000 7f762969b640 1 -- 172.21.15.23:0/1551061551 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762419f6a0 con 0x7f7624100640 2024-03-28T05:21:35.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.281+0000 7f762969b640 1 --2- 172.21.15.23:0/1551061551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7624100640 0x7f7624197a80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f762000e9c0 tx=0x7f762000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:35.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.281+0000 7f761a7fc640 1 -- 172.21.15.23:0/1551061551 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762000cd80 con 0x7f7624100640 2024-03-28T05:21:35.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.281+0000 7f761a7fc640 1 -- 172.21.15.23:0/1551061551 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76200035f0 con 0x7f7624100640 2024-03-28T05:21:35.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.281+0000 7f762b8ff640 1 -- 172.21.15.23:0/1551061551 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762419f990 con 0x7f7624100640 2024-03-28T05:21:35.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.281+0000 7f761a7fc640 1 -- 172.21.15.23:0/1551061551 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7620010690 con 0x7f7624100640 2024-03-28T05:21:35.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.281+0000 7f762b8ff640 1 -- 172.21.15.23:0/1551061551 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762419ff00 con 0x7f7624100640 2024-03-28T05:21:35.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.282+0000 7f762b8ff640 1 -- 172.21.15.23:0/1551061551 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76241a0190 con 0x7f7624100640 2024-03-28T05:21:35.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.284+0000 7f761a7fc640 1 -- 172.21.15.23:0/1551061551 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f76200037a0 con 0x7f7624100640 2024-03-28T05:21:35.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.284+0000 7f761a7fc640 1 --2- 172.21.15.23:0/1551061551 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f76100a2880 0x7f76100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:35.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.284+0000 7f761a7fc640 1 -- 172.21.15.23:0/1551061551 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f76200cf110 con 0x7f7624100640 2024-03-28T05:21:35.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.284+0000 7f7628e9a640 1 --2- 172.21.15.23:0/1551061551 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f76100a2880 0x7f76100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:35.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.285+0000 7f7628e9a640 1 --2- 172.21.15.23:0/1551061551 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f76100a2880 0x7f76100a4d40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7614009230 tx=0x7f7614005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:35.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.288+0000 7f761a7fc640 1 -- 172.21.15.23:0/1551061551 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f762009c000 con 0x7f7624100640 2024-03-28T05:21:35.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.402+0000 7f761a7fc640 1 -- 172.21.15.23:0/1551061551 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f7620097f70 con 0x7f7624100640 2024-03-28T05:21:35.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.517+0000 7f762b8ff640 1 -- 172.21.15.23:0/1551061551 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76241a0360 con 0x7f7624100640 2024-03-28T05:21:35.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.518+0000 7f761a7fc640 1 -- 172.21.15.23:0/1551061551 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11324 (secure 0 0 0) 0x7f76200c5810 con 0x7f7624100640 2024-03-28T05:21:35.517 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:21:35.517 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/733913400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":733913400},{"type":"v1","addr":"172.21.15.23:6835","nonce":733913400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1972901387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1972901387},{"type":"v1","addr":"172.21.15.23:6837","nonce":1972901387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/2390673788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2390673788},{"type":"v1","addr":"172.21.15.23:6839","nonce":2390673788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/289263156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":289263156},{"type":"v1","addr":"172.21.15.23:6841","nonce":289263156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3313932732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3313932732},{"type":"v1","addr":"172.21.15.98:6835","nonce":3313932732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2695340024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2695340024},{"type":"v1","addr":"172.21.15.113:6835","nonce":2695340024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2750367469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2750367469},{"type":"v1","addr":"172.21.15.98:6837","nonce":2750367469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/3222726274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":3222726274},{"type":"v1","addr":"172.21.15.113:6837","nonce":3222726274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T05:21:07.616553+0000","modified":"2024-03-28T05:21:32.732132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/144113007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":144113007},{"type":"v1","addr":"172.21.15.98:6839","nonce":144113007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.113:6839/212060489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":212060489},{"type":"v1","addr":"172.21.15.113:6839","nonce":212060489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24487,"qdb_cluster":[24487]},"id":1}]} 2024-03-28T05:21:35.517 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 20 2024-03-28T05:21:35.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.520+0000 7f762b8ff640 1 -- 172.21.15.23:0/1551061551 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f76100a2880 msgr2=0x7f76100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:35.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.520+0000 7f762b8ff640 1 --2- 172.21.15.23:0/1551061551 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f76100a2880 0x7f76100a4d40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7614009230 tx=0x7f7614005fd0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.520+0000 7f762b8ff640 1 -- 172.21.15.23:0/1551061551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7624100640 msgr2=0x7f7624197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:35.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.520+0000 7f762b8ff640 1 --2- 172.21.15.23:0/1551061551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7624100640 0x7f7624197a80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f762000e9c0 tx=0x7f762000ee80 comp rx=0 tx=0).stop 2024-03-28T05:21:35.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.520+0000 7f762b8ff640 1 -- 172.21.15.23:0/1551061551 shutdown_connections 2024-03-28T05:21:35.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.520+0000 7f762b8ff640 1 --2- 172.21.15.23:0/1551061551 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f76100a2880 0x7f76100a4d40 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.520+0000 7f762b8ff640 1 --2- 172.21.15.23:0/1551061551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76241021f0 0x7f762419f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.520+0000 7f762b8ff640 1 --2- 172.21.15.23:0/1551061551 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7624101830 0x7f7624197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.520+0000 7f762b8ff640 1 --2- 172.21.15.23:0/1551061551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7624100640 0x7f7624197a80 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.520+0000 7f762b8ff640 1 -- 172.21.15.23:0/1551061551 >> 172.21.15.23:0/1551061551 conn(0x7f76240fbbd0 msgr2=0x7f76240fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:35.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.520+0000 7f762b8ff640 1 -- 172.21.15.23:0/1551061551 shutdown_connections 2024-03-28T05:21:35.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.520+0000 7f762b8ff640 1 -- 172.21.15.23:0/1551061551 wait complete. 2024-03-28T05:21:35.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:35 smithi023 ceph-mon[31954]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:21:35.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:35 smithi023 ceph-mon[31954]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:21:35.529 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:35 smithi023 ceph-mon[31954]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:21:35.530 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T05:21:35.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.634+0000 7f5b1cd7e640 1 -- 172.21.15.23:0/1583132255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b180ffd50 msgr2=0x7f5b18100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:35.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.634+0000 7f5b1cd7e640 1 --2- 172.21.15.23:0/1583132255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b180ffd50 0x7f5b18100170 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c009900 tx=0x7f5b0c02bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:35.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.634+0000 7f5b1cd7e640 1 -- 172.21.15.23:0/1583132255 shutdown_connections 2024-03-28T05:21:35.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.634+0000 7f5b1cd7e640 1 --2- 172.21.15.23:0/1583132255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b18101900 0x7f5b18108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.634+0000 7f5b1cd7e640 1 --2- 172.21.15.23:0/1583132255 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b18100f40 0x7f5b181013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.634+0000 7f5b1cd7e640 1 --2- 172.21.15.23:0/1583132255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b180ffd50 0x7f5b18100170 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.634+0000 7f5b1cd7e640 1 -- 172.21.15.23:0/1583132255 >> 172.21.15.23:0/1583132255 conn(0x7f5b180fb2c0 msgr2=0x7f5b180fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:35.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.634+0000 7f5b1cd7e640 1 -- 172.21.15.23:0/1583132255 shutdown_connections 2024-03-28T05:21:35.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.634+0000 7f5b1cd7e640 1 -- 172.21.15.23:0/1583132255 wait complete. 2024-03-28T05:21:35.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.634+0000 7f5b1cd7e640 1 Processor -- start 2024-03-28T05:21:35.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.634+0000 7f5b1cd7e640 1 -- start start 2024-03-28T05:21:35.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.635+0000 7f5b1cd7e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b180ffd50 0x7f5b18109690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:35.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.635+0000 7f5b1cd7e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b18100f40 0x7f5b1819c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:35.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.635+0000 7f5b1cd7e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b18101900 0x7f5b1819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:35.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.635+0000 7f5b1cd7e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1810c240 con 0x7f5b18100f40 2024-03-28T05:21:35.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.635+0000 7f5b1cd7e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1810c0c0 con 0x7f5b180ffd50 2024-03-28T05:21:35.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.635+0000 7f5b1cd7e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1810c3c0 con 0x7f5b18101900 2024-03-28T05:21:35.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.635+0000 7f5b15d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b18100f40 0x7f5b1819c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:35.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.635+0000 7f5b15d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b18100f40 0x7f5b1819c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53342/0 (socket says 172.21.15.23:53342) 2024-03-28T05:21:35.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.635+0000 7f5b15d9b640 1 -- 172.21.15.23:0/1704576560 learned_addr learned my addr 172.21.15.23:0/1704576560 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:35.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.635+0000 7f5b16d9d640 1 --2- 172.21.15.23:0/1704576560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b18101900 0x7f5b1819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:35.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.635+0000 7f5b1659c640 1 --2- 172.21.15.23:0/1704576560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b180ffd50 0x7f5b18109690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:35.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.635+0000 7f5b15d9b640 1 -- 172.21.15.23:0/1704576560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b18101900 msgr2=0x7f5b1819edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:35.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.635+0000 7f5b15d9b640 1 --2- 172.21.15.23:0/1704576560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b18101900 0x7f5b1819edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.635+0000 7f5b15d9b640 1 -- 172.21.15.23:0/1704576560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b180ffd50 msgr2=0x7f5b18109690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:35.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.635+0000 7f5b15d9b640 1 --2- 172.21.15.23:0/1704576560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b180ffd50 0x7f5b18109690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.635+0000 7f5b15d9b640 1 -- 172.21.15.23:0/1704576560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1819f670 con 0x7f5b18100f40 2024-03-28T05:21:35.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.635+0000 7f5b15d9b640 1 --2- 172.21.15.23:0/1704576560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b18100f40 0x7f5b1819c7c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5b0800e9f0 tx=0x7f5b0800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:35.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.635+0000 7f5b077fe640 1 -- 172.21.15.23:0/1704576560 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0800cd80 con 0x7f5b18100f40 2024-03-28T05:21:35.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.635+0000 7f5b1cd7e640 1 -- 172.21.15.23:0/1704576560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1819f960 con 0x7f5b18100f40 2024-03-28T05:21:35.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.635+0000 7f5b077fe640 1 -- 172.21.15.23:0/1704576560 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b08005500 con 0x7f5b18100f40 2024-03-28T05:21:35.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.635+0000 7f5b077fe640 1 -- 172.21.15.23:0/1704576560 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b08010690 con 0x7f5b18100f40 2024-03-28T05:21:35.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.635+0000 7f5b1cd7e640 1 -- 172.21.15.23:0/1704576560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1819fe60 con 0x7f5b18100f40 2024-03-28T05:21:35.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.636+0000 7f5b1cd7e640 1 -- 172.21.15.23:0/1704576560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b18107130 con 0x7f5b18100f40 2024-03-28T05:21:35.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.637+0000 7f5b077fe640 1 -- 172.21.15.23:0/1704576560 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f5b080046c0 con 0x7f5b18100f40 2024-03-28T05:21:35.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.638+0000 7f5b077fe640 1 --2- 172.21.15.23:0/1704576560 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5b000a2890 0x7f5b000a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:35.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.638+0000 7f5b1659c640 1 --2- 172.21.15.23:0/1704576560 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5b000a2890 0x7f5b000a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:35.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.638+0000 7f5b077fe640 1 -- 172.21.15.23:0/1704576560 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f5b080cf220 con 0x7f5b18100f40 2024-03-28T05:21:35.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.638+0000 7f5b1659c640 1 --2- 172.21.15.23:0/1704576560 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5b000a2890 0x7f5b000a4d50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c000c00 tx=0x7f5b0c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:35.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.642+0000 7f5b077fe640 1 -- 172.21.15.23:0/1704576560 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5b0809bf20 con 0x7f5b18100f40 2024-03-28T05:21:35.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:35 smithi113 ceph-mon[37164]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:21:35.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:35 smithi113 ceph-mon[37164]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:21:35.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:35 smithi113 ceph-mon[37164]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:21:35.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.828+0000 7f5b1cd7e640 1 -- 172.21.15.23:0/1704576560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b1802d280 con 0x7f5b18100f40 2024-03-28T05:21:35.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.829+0000 7f5b077fe640 1 -- 172.21.15.23:0/1704576560 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24082 (secure 0 0 0) 0x7f5b0809b820 con 0x7f5b18100f40 2024-03-28T05:21:35.829 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:21:35.829 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":89,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","created":"2024-03-28T05:13:13.453653+0000","modified":"2024-03-28T05:21:12.641185+0000","last_up_change":"2024-03-28T05:20:20.221527+0000","last_in_change":"2024-03-28T05:20:00.365478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T05:16:32.986974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T05:21:06.103894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T05:21:06.827190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T05:21:10.507418+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"18150942-59a4-4c3b-83d4-c1fc3ccf1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6803","nonce":1219956881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6805","nonce":1219956881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6809","nonce":1219956881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6807","nonce":1219956881}]},"public_addr":"172.21.15.23:6803/1219956881","cluster_addr":"172.21.15.23:6805/1219956881","heartbeat_back_addr":"172.21.15.23:6809/1219956881","heartbeat_front_addr":"172.21.15.23:6807/1219956881","state":["exists","up"]},{"osd":1,"uuid":"ba6d1a95-5135-4670-95f9-7878c0e7bd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6801","nonce":555491234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6803","nonce":555491234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6807","nonce":555491234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6805","nonce":555491234}]},"public_addr":"172.21.15.98:6801/555491234","cluster_addr":"172.21.15.98:6803/555491234","heartbeat_back_addr":"172.21.15.98:6807/555491234","heartbeat_front_addr":"172.21.15.98:6805/555491234","state":["exists","up"]},{"osd":2,"uuid":"873bdd60-2a47-403b-b1f4-ee9226a95067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6801","nonce":1535856914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6803","nonce":1535856914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6807","nonce":1535856914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6805","nonce":1535856914}]},"public_addr":"172.21.15.113:6801/1535856914","cluster_addr":"172.21.15.113:6803/1535856914","heartbeat_back_addr":"172.21.15.113:6807/1535856914","heartbeat_front_addr":"172.21.15.113:6805/1535856914","state":["exists","up"]},{"osd":3,"uuid":"83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6811","nonce":2949415020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6813","nonce":2949415020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6817","nonce":2949415020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6815","nonce":2949415020}]},"public_addr":"172.21.15.23:6811/2949415020","cluster_addr":"172.21.15.23:6813/2949415020","heartbeat_back_addr":"172.21.15.23:6817/2949415020","heartbeat_front_addr":"172.21.15.23:6815/2949415020","state":["exists","up"]},{"osd":4,"uuid":"76a90bac-b221-40a3-8238-40e8049cec36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6809","nonce":2615902819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6811","nonce":2615902819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6815","nonce":2615902819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6813","nonce":2615902819}]},"public_addr":"172.21.15.98:6809/2615902819","cluster_addr":"172.21.15.98:6811/2615902819","heartbeat_back_addr":"172.21.15.98:6815/2615902819","heartbeat_front_addr":"172.21.15.98:6813/2615902819","state":["exists","up"]},{"osd":5,"uuid":"b53fdb55-a8df-40a8-bd1b-8bcef634e26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6809","nonce":625271528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6811","nonce":625271528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6815","nonce":625271528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6813","nonce":625271528}]},"public_addr":"172.21.15.113:6809/625271528","cluster_addr":"172.21.15.113:6811/625271528","heartbeat_back_addr":"172.21.15.113:6815/625271528","heartbeat_front_addr":"172.21.15.113:6813/625271528","state":["exists","up"]},{"osd":6,"uuid":"f00283d2-8d67-4e97-a575-9a9d3b55d5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6819","nonce":4188464176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6821","nonce":4188464176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6825","nonce":4188464176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6823","nonce":4188464176}]},"public_addr":"172.21.15.23:6819/4188464176","cluster_addr":"172.21.15.23:6821/4188464176","heartbeat_back_addr":"172.21.15.23:6825/4188464176","heartbeat_front_addr":"172.21.15.23:6823/4188464176","state":["exists","up"]},{"osd":7,"uuid":"edc008a7-b07b-40a8-88b7-1614ba38a775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6817","nonce":1985546028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6819","nonce":1985546028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6823","nonce":1985546028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6821","nonce":1985546028}]},"public_addr":"172.21.15.98:6817/1985546028","cluster_addr":"172.21.15.98:6819/1985546028","heartbeat_back_addr":"172.21.15.98:6823/1985546028","heartbeat_front_addr":"172.21.15.98:6821/1985546028","state":["exists","up"]},{"osd":8,"uuid":"56e7b2d1-81f0-47b2-8638-f29832bed8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6817","nonce":3534876081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6819","nonce":3534876081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6823","nonce":3534876081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6821","nonce":3534876081}]},"public_addr":"172.21.15.113:6817/3534876081","cluster_addr":"172.21.15.113:6819/3534876081","heartbeat_back_addr":"172.21.15.113:6823/3534876081","heartbeat_front_addr":"172.21.15.113:6821/3534876081","state":["exists","up"]},{"osd":9,"uuid":"a2e90caa-6e9c-42bf-9f18-81f7f47880cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6827","nonce":672264207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6829","nonce":672264207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6833","nonce":672264207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6831","nonce":672264207}]},"public_addr":"172.21.15.23:6827/672264207","cluster_addr":"172.21.15.23:6829/672264207","heartbeat_back_addr":"172.21.15.23:6833/672264207","heartbeat_front_addr":"172.21.15.23:6831/672264207","state":["exists","up"]},{"osd":10,"uuid":"0eda917f-020c-437f-95ae-b26203f6c68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6825","nonce":2699920595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6827","nonce":2699920595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6831","nonce":2699920595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6829","nonce":2699920595}]},"public_addr":"172.21.15.98:6825/2699920595","cluster_addr":"172.21.15.98:6827/2699920595","heartbeat_back_addr":"172.21.15.98:6831/2699920595","heartbeat_front_addr":"172.21.15.98:6829/2699920595","state":["exists","up"]},{"osd":11,"uuid":"aae78501-78a8-469a-be20-3006f5ec5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6825","nonce":2113644020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6827","nonce":2113644020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6831","nonce":2113644020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6829","nonce":2113644020}]},"public_addr":"172.21.15.113:6825/2113644020","cluster_addr":"172.21.15.113:6827/2113644020","heartbeat_back_addr":"172.21.15.113:6831/2113644020","heartbeat_front_addr":"172.21.15.113:6829/2113644020","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:15:43.527475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:07.006538+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:30.594176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:55.810999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:20.595811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:45.474138+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:10.500409+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:35.787638+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:01.001188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:26.761115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:52.952752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:20:18.559832+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:6800/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1876451381":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2253559862":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2653562762":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/3337463830":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6800/1735221490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/2784771635":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2389902490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6801/1735221490":"2024-03-29T05:13:42.579607+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T05:21:35.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.831+0000 7f5b1cd7e640 1 -- 172.21.15.23:0/1704576560 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5b000a2890 msgr2=0x7f5b000a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:35.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.831+0000 7f5b1cd7e640 1 --2- 172.21.15.23:0/1704576560 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5b000a2890 0x7f5b000a4d50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c000c00 tx=0x7f5b0c005fd0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.831+0000 7f5b1cd7e640 1 -- 172.21.15.23:0/1704576560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b18100f40 msgr2=0x7f5b1819c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:35.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.831+0000 7f5b1cd7e640 1 --2- 172.21.15.23:0/1704576560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b18100f40 0x7f5b1819c7c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5b0800e9f0 tx=0x7f5b0800edf0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.831+0000 7f5b1cd7e640 1 -- 172.21.15.23:0/1704576560 shutdown_connections 2024-03-28T05:21:35.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.831+0000 7f5b1cd7e640 1 --2- 172.21.15.23:0/1704576560 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5b000a2890 0x7f5b000a4d50 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.831+0000 7f5b1cd7e640 1 --2- 172.21.15.23:0/1704576560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b18101900 0x7f5b1819edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.831+0000 7f5b1cd7e640 1 --2- 172.21.15.23:0/1704576560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b18100f40 0x7f5b1819c7c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.831+0000 7f5b1cd7e640 1 --2- 172.21.15.23:0/1704576560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b180ffd50 0x7f5b18109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.831+0000 7f5b1cd7e640 1 -- 172.21.15.23:0/1704576560 >> 172.21.15.23:0/1704576560 conn(0x7f5b180fb2c0 msgr2=0x7f5b18104300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:35.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.831+0000 7f5b1cd7e640 1 -- 172.21.15.23:0/1704576560 shutdown_connections 2024-03-28T05:21:35.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.831+0000 7f5b1cd7e640 1 -- 172.21.15.23:0/1704576560 wait complete. 2024-03-28T05:21:35.842 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T05:21:35.842 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T05:21:35.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:35 smithi098 ceph-mon[36087]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:21:35.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:35 smithi098 ceph-mon[36087]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:21:35.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:35 smithi098 ceph-mon[36087]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:21:35.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.945+0000 7f59e0177640 1 -- 172.21.15.23:0/1087720006 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59d8100f40 msgr2=0x7f59d81013c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:35.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.945+0000 7f59e0177640 1 --2- 172.21.15.23:0/1087720006 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59d8100f40 0x7f59d81013c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f59cc009860 tx=0x7f59cc02bb70 comp rx=0 tx=0).stop 2024-03-28T05:21:35.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.945+0000 7f59e0177640 1 -- 172.21.15.23:0/1087720006 shutdown_connections 2024-03-28T05:21:35.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.945+0000 7f59e0177640 1 --2- 172.21.15.23:0/1087720006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59d8101900 0x7f59d8108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.945+0000 7f59e0177640 1 --2- 172.21.15.23:0/1087720006 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59d8100f40 0x7f59d81013c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.945+0000 7f59e0177640 1 --2- 172.21.15.23:0/1087720006 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59d80ffd50 0x7f59d8100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.945+0000 7f59e0177640 1 -- 172.21.15.23:0/1087720006 >> 172.21.15.23:0/1087720006 conn(0x7f59d80fb2c0 msgr2=0x7f59d80fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:35.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.945+0000 7f59e0177640 1 -- 172.21.15.23:0/1087720006 shutdown_connections 2024-03-28T05:21:35.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.945+0000 7f59e0177640 1 -- 172.21.15.23:0/1087720006 wait complete. 2024-03-28T05:21:35.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.946+0000 7f59e0177640 1 Processor -- start 2024-03-28T05:21:35.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.946+0000 7f59e0177640 1 -- start start 2024-03-28T05:21:35.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.946+0000 7f59e0177640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59d80ffd50 0x7f59d80fd430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:35.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.946+0000 7f59e0177640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59d8100f40 0x7f59d819c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:35.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.946+0000 7f59e0177640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59d8101900 0x7f59d819ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:35.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.946+0000 7f59e0177640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d810b9b0 con 0x7f59d80ffd50 2024-03-28T05:21:35.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.946+0000 7f59e0177640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d810b830 con 0x7f59d8100f40 2024-03-28T05:21:35.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.946+0000 7f59e0177640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d810bb30 con 0x7f59d8101900 2024-03-28T05:21:35.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.946+0000 7f59ddf13640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59d80ffd50 0x7f59d80fd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:35.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.946+0000 7f59ddf13640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59d80ffd50 0x7f59d80fd430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53356/0 (socket says 172.21.15.23:53356) 2024-03-28T05:21:35.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.946+0000 7f59ddf13640 1 -- 172.21.15.23:0/76365064 learned_addr learned my addr 172.21.15.23:0/76365064 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:35.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.946+0000 7f59dd712640 1 --2- 172.21.15.23:0/76365064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59d8100f40 0x7f59d819c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:35.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.946+0000 7f59de714640 1 --2- 172.21.15.23:0/76365064 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59d8101900 0x7f59d819ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:35.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.947+0000 7f59ddf13640 1 -- 172.21.15.23:0/76365064 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59d8101900 msgr2=0x7f59d819ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:35.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.947+0000 7f59ddf13640 1 --2- 172.21.15.23:0/76365064 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59d8101900 0x7f59d819ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.947+0000 7f59ddf13640 1 -- 172.21.15.23:0/76365064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59d8100f40 msgr2=0x7f59d819c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:35.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.947+0000 7f59ddf13640 1 --2- 172.21.15.23:0/76365064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59d8100f40 0x7f59d819c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:35.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.947+0000 7f59ddf13640 1 -- 172.21.15.23:0/76365064 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d819f600 con 0x7f59d80ffd50 2024-03-28T05:21:35.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.947+0000 7f59ddf13640 1 --2- 172.21.15.23:0/76365064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59d80ffd50 0x7f59d80fd430 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f59d000e9f0 tx=0x7f59d000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:35.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.947+0000 7f59d6ffd640 1 -- 172.21.15.23:0/76365064 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d000ccc0 con 0x7f59d80ffd50 2024-03-28T05:21:35.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.947+0000 7f59d6ffd640 1 -- 172.21.15.23:0/76365064 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59d00035f0 con 0x7f59d80ffd50 2024-03-28T05:21:35.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.947+0000 7f59e0177640 1 -- 172.21.15.23:0/76365064 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d819f890 con 0x7f59d80ffd50 2024-03-28T05:21:35.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.947+0000 7f59d6ffd640 1 -- 172.21.15.23:0/76365064 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d000ce60 con 0x7f59d80ffd50 2024-03-28T05:21:35.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.947+0000 7f59e0177640 1 -- 172.21.15.23:0/76365064 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d819fdb0 con 0x7f59d80ffd50 2024-03-28T05:21:35.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.948+0000 7f59e0177640 1 -- 172.21.15.23:0/76365064 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d8107830 con 0x7f59d80ffd50 2024-03-28T05:21:35.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.949+0000 7f59d6ffd640 1 -- 172.21.15.23:0/76365064 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f59d00037a0 con 0x7f59d80ffd50 2024-03-28T05:21:35.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.949+0000 7f59d6ffd640 1 --2- 172.21.15.23:0/76365064 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f59c00a2890 0x7f59c00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:35.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.950+0000 7f59dd712640 1 --2- 172.21.15.23:0/76365064 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f59c00a2890 0x7f59c00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:35.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.950+0000 7f59d6ffd640 1 -- 172.21.15.23:0/76365064 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f59d00cf140 con 0x7f59d80ffd50 2024-03-28T05:21:35.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.950+0000 7f59dd712640 1 --2- 172.21.15.23:0/76365064 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f59c00a2890 0x7f59c00a4d50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f59cc02c180 tx=0x7f59cc034ad0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:35.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:35.953+0000 7f59d6ffd640 1 -- 172.21.15.23:0/76365064 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f59d009be40 con 0x7f59d80ffd50 2024-03-28T05:21:36.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.133+0000 7f59e0177640 1 -- 172.21.15.23:0/76365064 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f59d8061ae0 con 0x7f59c00a2890 2024-03-28T05:21:36.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.144+0000 7f59d6ffd640 1 -- 172.21.15.23:0/76365064 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340209 (secure 0 0 0) 0x7f59d8061ae0 con 0x7f59c00a2890 2024-03-28T05:21:36.149 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:21:36.152 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-28T05:21:36.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.147+0000 7f59e0177640 1 -- 172.21.15.23:0/76365064 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f59c00a2890 msgr2=0x7f59c00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:36.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.147+0000 7f59e0177640 1 --2- 172.21.15.23:0/76365064 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f59c00a2890 0x7f59c00a4d50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f59cc02c180 tx=0x7f59cc034ad0 comp rx=0 tx=0).stop 2024-03-28T05:21:36.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.147+0000 7f59e0177640 1 -- 172.21.15.23:0/76365064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59d80ffd50 msgr2=0x7f59d80fd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:36.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.147+0000 7f59e0177640 1 --2- 172.21.15.23:0/76365064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59d80ffd50 0x7f59d80fd430 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f59d000e9f0 tx=0x7f59d000eeb0 comp rx=0 tx=0).stop 2024-03-28T05:21:36.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.147+0000 7f59e0177640 1 -- 172.21.15.23:0/76365064 shutdown_connections 2024-03-28T05:21:36.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.147+0000 7f59e0177640 1 --2- 172.21.15.23:0/76365064 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f59c00a2890 0x7f59c00a4d50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:36.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.147+0000 7f59e0177640 1 --2- 172.21.15.23:0/76365064 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59d8101900 0x7f59d819ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:36.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.147+0000 7f59e0177640 1 --2- 172.21.15.23:0/76365064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59d8100f40 0x7f59d819c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:36.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.147+0000 7f59e0177640 1 --2- 172.21.15.23:0/76365064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59d80ffd50 0x7f59d80fd430 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:36.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.147+0000 7f59e0177640 1 -- 172.21.15.23:0/76365064 >> 172.21.15.23:0/76365064 conn(0x7f59d80fb2c0 msgr2=0x7f59d8104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:36.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.147+0000 7f59e0177640 1 -- 172.21.15.23:0/76365064 shutdown_connections 2024-03-28T05:21:36.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.147+0000 7f59e0177640 1 -- 172.21.15.23:0/76365064 wait complete. 2024-03-28T05:21:36.158 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-03-28T05:21:34.726040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":614307,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":150,"num_read_kb":151,"num_write":237,"num_write_kb":2979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":187,"ondisk_log_size":187,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":358428,"kb_used_data":13388,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124583396,"statfs":{"total":1151940427776,"available":1151573397504,"internally_reserved":0,"allocated":13709312,"data_stored":7541599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21080,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":32,"num_write":10,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003506"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876886+0000","last_change":"2024-03-28T05:21:08.043720+0000","last_active":"2024-03-28T05:21:13.876886+0000","last_peered":"2024-03-28T05:21:13.876886+0000","last_clean":"2024-03-28T05:21:13.876886+0000","last_became_active":"2024-03-28T05:21:08.043250+0000","last_became_peered":"2024-03-28T05:21:08.043250+0000","last_unstale":"2024-03-28T05:21:13.876886+0000","last_undegraded":"2024-03-28T05:21:13.876886+0000","last_fullsized":"2024-03-28T05:21:13.876886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:29:10.144086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:14.592555+0000","last_change":"2024-03-28T05:21:08.053247+0000","last_active":"2024-03-28T05:21:14.592555+0000","last_peered":"2024-03-28T05:21:14.592555+0000","last_clean":"2024-03-28T05:21:14.592555+0000","last_became_active":"2024-03-28T05:21:08.052941+0000","last_became_peered":"2024-03-28T05:21:08.052941+0000","last_unstale":"2024-03-28T05:21:14.592555+0000","last_undegraded":"2024-03-28T05:21:14.592555+0000","last_fullsized":"2024-03-28T05:21:14.592555+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:41:22.223858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:14.592448+0000","last_change":"2024-03-28T05:21:08.051938+0000","last_active":"2024-03-28T05:21:14.592448+0000","last_peered":"2024-03-28T05:21:14.592448+0000","last_clean":"2024-03-28T05:21:14.592448+0000","last_became_active":"2024-03-28T05:21:08.050366+0000","last_became_peered":"2024-03-28T05:21:08.050366+0000","last_unstale":"2024-03-28T05:21:14.592448+0000","last_undegraded":"2024-03-28T05:21:14.592448+0000","last_fullsized":"2024-03-28T05:21:14.592448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:27:29.325009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877382+0000","last_change":"2024-03-28T05:21:08.043371+0000","last_active":"2024-03-28T05:21:13.877382+0000","last_peered":"2024-03-28T05:21:13.877382+0000","last_clean":"2024-03-28T05:21:13.877382+0000","last_became_active":"2024-03-28T05:21:08.043038+0000","last_became_peered":"2024-03-28T05:21:08.043038+0000","last_unstale":"2024-03-28T05:21:13.877382+0000","last_undegraded":"2024-03-28T05:21:13.877382+0000","last_fullsized":"2024-03-28T05:21:13.877382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:16:14.137248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876032+0000","last_change":"2024-03-28T05:21:08.048678+0000","last_active":"2024-03-28T05:21:13.876032+0000","last_peered":"2024-03-28T05:21:13.876032+0000","last_clean":"2024-03-28T05:21:13.876032+0000","last_became_active":"2024-03-28T05:21:08.048272+0000","last_became_peered":"2024-03-28T05:21:08.048272+0000","last_unstale":"2024-03-28T05:21:13.876032+0000","last_undegraded":"2024-03-28T05:21:13.876032+0000","last_fullsized":"2024-03-28T05:21:13.876032+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:34:29.029249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877106+0000","last_change":"2024-03-28T05:21:08.041764+0000","last_active":"2024-03-28T05:21:13.877106+0000","last_peered":"2024-03-28T05:21:13.877106+0000","last_clean":"2024-03-28T05:21:13.877106+0000","last_became_active":"2024-03-28T05:21:08.041337+0000","last_became_peered":"2024-03-28T05:21:08.041337+0000","last_unstale":"2024-03-28T05:21:13.877106+0000","last_undegraded":"2024-03-28T05:21:13.877106+0000","last_fullsized":"2024-03-28T05:21:13.877106+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:42:29.136829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.275455+0000","last_change":"2024-03-28T05:21:08.046554+0000","last_active":"2024-03-28T05:21:13.275455+0000","last_peered":"2024-03-28T05:21:13.275455+0000","last_clean":"2024-03-28T05:21:13.275455+0000","last_became_active":"2024-03-28T05:21:08.046245+0000","last_became_peered":"2024-03-28T05:21:08.046245+0000","last_unstale":"2024-03-28T05:21:13.275455+0000","last_undegraded":"2024-03-28T05:21:13.275455+0000","last_fullsized":"2024-03-28T05:21:13.275455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:40:05.549254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.875924+0000","last_change":"2024-03-28T05:21:08.048605+0000","last_active":"2024-03-28T05:21:13.875924+0000","last_peered":"2024-03-28T05:21:13.875924+0000","last_clean":"2024-03-28T05:21:13.875924+0000","last_became_active":"2024-03-28T05:21:08.048279+0000","last_became_peered":"2024-03-28T05:21:08.048279+0000","last_unstale":"2024-03-28T05:21:13.875924+0000","last_undegraded":"2024-03-28T05:21:13.875924+0000","last_fullsized":"2024-03-28T05:21:13.875924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:34:00.935133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.878182+0000","last_change":"2024-03-28T05:21:08.051357+0000","last_active":"2024-03-28T05:21:13.878182+0000","last_peered":"2024-03-28T05:21:13.878182+0000","last_clean":"2024-03-28T05:21:13.878182+0000","last_became_active":"2024-03-28T05:21:08.051012+0000","last_became_peered":"2024-03-28T05:21:08.051012+0000","last_unstale":"2024-03-28T05:21:13.878182+0000","last_undegraded":"2024-03-28T05:21:13.878182+0000","last_fullsized":"2024-03-28T05:21:13.878182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:43:42.090619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877368+0000","last_change":"2024-03-28T05:21:08.038727+0000","last_active":"2024-03-28T05:21:13.877368+0000","last_peered":"2024-03-28T05:21:13.877368+0000","last_clean":"2024-03-28T05:21:13.877368+0000","last_became_active":"2024-03-28T05:21:08.038425+0000","last_became_peered":"2024-03-28T05:21:08.038425+0000","last_unstale":"2024-03-28T05:21:13.877368+0000","last_undegraded":"2024-03-28T05:21:13.877368+0000","last_fullsized":"2024-03-28T05:21:13.877368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:06:53.786136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.276082+0000","last_change":"2024-03-28T05:21:08.054698+0000","last_active":"2024-03-28T05:21:13.276082+0000","last_peered":"2024-03-28T05:21:13.276082+0000","last_clean":"2024-03-28T05:21:13.276082+0000","last_became_active":"2024-03-28T05:21:08.054310+0000","last_became_peered":"2024-03-28T05:21:08.054310+0000","last_unstale":"2024-03-28T05:21:13.276082+0000","last_undegraded":"2024-03-28T05:21:13.276082+0000","last_fullsized":"2024-03-28T05:21:13.276082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:09:47.996381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877139+0000","last_change":"2024-03-28T05:21:08.042826+0000","last_active":"2024-03-28T05:21:13.877139+0000","last_peered":"2024-03-28T05:21:13.877139+0000","last_clean":"2024-03-28T05:21:13.877139+0000","last_became_active":"2024-03-28T05:21:08.042487+0000","last_became_peered":"2024-03-28T05:21:08.042487+0000","last_unstale":"2024-03-28T05:21:13.877139+0000","last_undegraded":"2024-03-28T05:21:13.877139+0000","last_fullsized":"2024-03-28T05:21:13.877139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:47:06.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.828392+0000","last_change":"2024-03-28T05:21:08.030791+0000","last_active":"2024-03-28T05:21:13.828392+0000","last_peered":"2024-03-28T05:21:13.828392+0000","last_clean":"2024-03-28T05:21:13.828392+0000","last_became_active":"2024-03-28T05:21:08.030447+0000","last_became_peered":"2024-03-28T05:21:08.030447+0000","last_unstale":"2024-03-28T05:21:13.828392+0000","last_undegraded":"2024-03-28T05:21:13.828392+0000","last_fullsized":"2024-03-28T05:21:13.828392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:54:53.454037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:14.591851+0000","last_change":"2024-03-28T05:21:08.042477+0000","last_active":"2024-03-28T05:21:14.591851+0000","last_peered":"2024-03-28T05:21:14.591851+0000","last_clean":"2024-03-28T05:21:14.591851+0000","last_became_active":"2024-03-28T05:21:08.042164+0000","last_became_peered":"2024-03-28T05:21:08.042164+0000","last_unstale":"2024-03-28T05:21:14.591851+0000","last_undegraded":"2024-03-28T05:21:14.591851+0000","last_fullsized":"2024-03-28T05:21:14.591851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:44:25.566195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:14.591961+0000","last_change":"2024-03-28T05:21:08.050635+0000","last_active":"2024-03-28T05:21:14.591961+0000","last_peered":"2024-03-28T05:21:14.591961+0000","last_clean":"2024-03-28T05:21:14.591961+0000","last_became_active":"2024-03-28T05:21:08.050217+0000","last_became_peered":"2024-03-28T05:21:08.050217+0000","last_unstale":"2024-03-28T05:21:14.591961+0000","last_undegraded":"2024-03-28T05:21:14.591961+0000","last_fullsized":"2024-03-28T05:21:14.591961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:13:49.252755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876231+0000","last_change":"2024-03-28T05:21:08.048021+0000","last_active":"2024-03-28T05:21:13.876231+0000","last_peered":"2024-03-28T05:21:13.876231+0000","last_clean":"2024-03-28T05:21:13.876231+0000","last_became_active":"2024-03-28T05:21:08.047381+0000","last_became_peered":"2024-03-28T05:21:08.047381+0000","last_unstale":"2024-03-28T05:21:13.876231+0000","last_undegraded":"2024-03-28T05:21:13.876231+0000","last_fullsized":"2024-03-28T05:21:13.876231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:42:39.718804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876786+0000","last_change":"2024-03-28T05:21:08.044774+0000","last_active":"2024-03-28T05:21:13.876786+0000","last_peered":"2024-03-28T05:21:13.876786+0000","last_clean":"2024-03-28T05:21:13.876786+0000","last_became_active":"2024-03-28T05:21:08.044470+0000","last_became_peered":"2024-03-28T05:21:08.044470+0000","last_unstale":"2024-03-28T05:21:13.876786+0000","last_undegraded":"2024-03-28T05:21:13.876786+0000","last_fullsized":"2024-03-28T05:21:13.876786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:26:11.354288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272570+0000","last_change":"2024-03-28T05:21:08.042871+0000","last_active":"2024-03-28T05:21:13.272570+0000","last_peered":"2024-03-28T05:21:13.272570+0000","last_clean":"2024-03-28T05:21:13.272570+0000","last_became_active":"2024-03-28T05:21:08.042506+0000","last_became_peered":"2024-03-28T05:21:08.042506+0000","last_unstale":"2024-03-28T05:21:13.272570+0000","last_undegraded":"2024-03-28T05:21:13.272570+0000","last_fullsized":"2024-03-28T05:21:13.272570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:34:50.424380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272250+0000","last_change":"2024-03-28T05:21:08.045056+0000","last_active":"2024-03-28T05:21:13.272250+0000","last_peered":"2024-03-28T05:21:13.272250+0000","last_clean":"2024-03-28T05:21:13.272250+0000","last_became_active":"2024-03-28T05:21:08.044719+0000","last_became_peered":"2024-03-28T05:21:08.044719+0000","last_unstale":"2024-03-28T05:21:13.272250+0000","last_undegraded":"2024-03-28T05:21:13.272250+0000","last_fullsized":"2024-03-28T05:21:13.272250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:23:12.479829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272213+0000","last_change":"2024-03-28T05:21:12.295239+0000","last_active":"2024-03-28T05:21:13.272213+0000","last_peered":"2024-03-28T05:21:13.272213+0000","last_clean":"2024-03-28T05:21:13.272213+0000","last_became_active":"2024-03-28T05:21:12.294843+0000","last_became_peered":"2024-03-28T05:21:12.294843+0000","last_unstale":"2024-03-28T05:21:13.272213+0000","last_undegraded":"2024-03-28T05:21:13.272213+0000","last_fullsized":"2024-03-28T05:21:13.272213+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:36:35.291983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272616+0000","last_change":"2024-03-28T05:21:07.270833+0000","last_active":"2024-03-28T05:21:13.272616+0000","last_peered":"2024-03-28T05:21:13.272616+0000","last_clean":"2024-03-28T05:21:13.272616+0000","last_became_active":"2024-03-28T05:21:07.270288+0000","last_became_peered":"2024-03-28T05:21:07.270288+0000","last_unstale":"2024-03-28T05:21:13.272616+0000","last_undegraded":"2024-03-28T05:21:13.272616+0000","last_fullsized":"2024-03-28T05:21:13.272616+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:41:05.494851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876309+0000","last_change":"2024-03-28T05:21:08.048904+0000","last_active":"2024-03-28T05:21:13.876309+0000","last_peered":"2024-03-28T05:21:13.876309+0000","last_clean":"2024-03-28T05:21:13.876309+0000","last_became_active":"2024-03-28T05:21:08.048614+0000","last_became_peered":"2024-03-28T05:21:08.048614+0000","last_unstale":"2024-03-28T05:21:13.876309+0000","last_undegraded":"2024-03-28T05:21:13.876309+0000","last_fullsized":"2024-03-28T05:21:13.876309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:29:30.394022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876559+0000","last_change":"2024-03-28T05:21:12.295599+0000","last_active":"2024-03-28T05:21:13.876559+0000","last_peered":"2024-03-28T05:21:13.876559+0000","last_clean":"2024-03-28T05:21:13.876559+0000","last_became_active":"2024-03-28T05:21:12.295161+0000","last_became_peered":"2024-03-28T05:21:12.295161+0000","last_unstale":"2024-03-28T05:21:13.876559+0000","last_undegraded":"2024-03-28T05:21:13.876559+0000","last_fullsized":"2024-03-28T05:21:13.876559+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:28:39.097018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876657+0000","last_change":"2024-03-28T05:21:07.264532+0000","last_active":"2024-03-28T05:21:13.876657+0000","last_peered":"2024-03-28T05:21:13.876657+0000","last_clean":"2024-03-28T05:21:13.876657+0000","last_became_active":"2024-03-28T05:21:07.264121+0000","last_became_peered":"2024-03-28T05:21:07.264121+0000","last_unstale":"2024-03-28T05:21:13.876657+0000","last_undegraded":"2024-03-28T05:21:13.876657+0000","last_fullsized":"2024-03-28T05:21:13.876657+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:40:01.927165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:14.592571+0000","last_change":"2024-03-28T05:21:08.045547+0000","last_active":"2024-03-28T05:21:14.592571+0000","last_peered":"2024-03-28T05:21:14.592571+0000","last_clean":"2024-03-28T05:21:14.592571+0000","last_became_active":"2024-03-28T05:21:08.045243+0000","last_became_peered":"2024-03-28T05:21:08.045243+0000","last_unstale":"2024-03-28T05:21:14.592571+0000","last_undegraded":"2024-03-28T05:21:14.592571+0000","last_fullsized":"2024-03-28T05:21:14.592571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:24:51.178938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877111+0000","last_change":"2024-03-28T05:21:12.301461+0000","last_active":"2024-03-28T05:21:13.877111+0000","last_peered":"2024-03-28T05:21:13.877111+0000","last_clean":"2024-03-28T05:21:13.877111+0000","last_became_active":"2024-03-28T05:21:12.300765+0000","last_became_peered":"2024-03-28T05:21:12.300765+0000","last_unstale":"2024-03-28T05:21:13.877111+0000","last_undegraded":"2024-03-28T05:21:13.877111+0000","last_fullsized":"2024-03-28T05:21:13.877111+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:10:38.609978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876505+0000","last_change":"2024-03-28T05:21:07.269209+0000","last_active":"2024-03-28T05:21:13.876505+0000","last_peered":"2024-03-28T05:21:13.876505+0000","last_clean":"2024-03-28T05:21:13.876505+0000","last_became_active":"2024-03-28T05:21:07.268068+0000","last_became_peered":"2024-03-28T05:21:07.268068+0000","last_unstale":"2024-03-28T05:21:13.876505+0000","last_undegraded":"2024-03-28T05:21:13.876505+0000","last_fullsized":"2024-03-28T05:21:13.876505+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:44:53.135439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877831+0000","last_change":"2024-03-28T05:21:08.045233+0000","last_active":"2024-03-28T05:21:13.877831+0000","last_peered":"2024-03-28T05:21:13.877831+0000","last_clean":"2024-03-28T05:21:13.877831+0000","last_became_active":"2024-03-28T05:21:08.044878+0000","last_became_peered":"2024-03-28T05:21:08.044878+0000","last_unstale":"2024-03-28T05:21:13.877831+0000","last_undegraded":"2024-03-28T05:21:13.877831+0000","last_fullsized":"2024-03-28T05:21:13.877831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:15:50.830519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876493+0000","last_change":"2024-03-28T05:21:12.296326+0000","last_active":"2024-03-28T05:21:13.876493+0000","last_peered":"2024-03-28T05:21:13.876493+0000","last_clean":"2024-03-28T05:21:13.876493+0000","last_became_active":"2024-03-28T05:21:12.295863+0000","last_became_peered":"2024-03-28T05:21:12.295863+0000","last_unstale":"2024-03-28T05:21:13.876493+0000","last_undegraded":"2024-03-28T05:21:13.876493+0000","last_fullsized":"2024-03-28T05:21:13.876493+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:26:49.340900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876164+0000","last_change":"2024-03-28T05:21:07.263689+0000","last_active":"2024-03-28T05:21:13.876164+0000","last_peered":"2024-03-28T05:21:13.876164+0000","last_clean":"2024-03-28T05:21:13.876164+0000","last_became_active":"2024-03-28T05:21:07.263410+0000","last_became_peered":"2024-03-28T05:21:07.263410+0000","last_unstale":"2024-03-28T05:21:13.876164+0000","last_undegraded":"2024-03-28T05:21:13.876164+0000","last_fullsized":"2024-03-28T05:21:13.876164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:51:03.017620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759802+0000","last_change":"2024-03-28T05:21:08.046993+0000","last_active":"2024-03-28T05:21:13.759802+0000","last_peered":"2024-03-28T05:21:13.759802+0000","last_clean":"2024-03-28T05:21:13.759802+0000","last_became_active":"2024-03-28T05:21:08.045802+0000","last_became_peered":"2024-03-28T05:21:08.045802+0000","last_unstale":"2024-03-28T05:21:13.759802+0000","last_undegraded":"2024-03-28T05:21:13.759802+0000","last_fullsized":"2024-03-28T05:21:13.759802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:18:31.024546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272106+0000","last_change":"2024-03-28T05:21:12.284248+0000","last_active":"2024-03-28T05:21:13.272106+0000","last_peered":"2024-03-28T05:21:13.272106+0000","last_clean":"2024-03-28T05:21:13.272106+0000","last_became_active":"2024-03-28T05:21:12.283887+0000","last_became_peered":"2024-03-28T05:21:12.283887+0000","last_unstale":"2024-03-28T05:21:13.272106+0000","last_undegraded":"2024-03-28T05:21:13.272106+0000","last_fullsized":"2024-03-28T05:21:13.272106+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:50:50.871119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759416+0000","last_change":"2024-03-28T05:21:07.266436+0000","last_active":"2024-03-28T05:21:13.759416+0000","last_peered":"2024-03-28T05:21:13.759416+0000","last_clean":"2024-03-28T05:21:13.759416+0000","last_became_active":"2024-03-28T05:21:07.266115+0000","last_became_peered":"2024-03-28T05:21:07.266115+0000","last_unstale":"2024-03-28T05:21:13.759416+0000","last_undegraded":"2024-03-28T05:21:13.759416+0000","last_fullsized":"2024-03-28T05:21:13.759416+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:42:06.865456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876045+0000","last_change":"2024-03-28T05:21:08.047500+0000","last_active":"2024-03-28T05:21:13.876045+0000","last_peered":"2024-03-28T05:21:13.876045+0000","last_clean":"2024-03-28T05:21:13.876045+0000","last_became_active":"2024-03-28T05:21:08.047210+0000","last_became_peered":"2024-03-28T05:21:08.047210+0000","last_unstale":"2024-03-28T05:21:13.876045+0000","last_undegraded":"2024-03-28T05:21:13.876045+0000","last_fullsized":"2024-03-28T05:21:13.876045+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:31:02.393561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272013+0000","last_change":"2024-03-28T05:21:12.288979+0000","last_active":"2024-03-28T05:21:13.272013+0000","last_peered":"2024-03-28T05:21:13.272013+0000","last_clean":"2024-03-28T05:21:13.272013+0000","last_became_active":"2024-03-28T05:21:12.288621+0000","last_became_peered":"2024-03-28T05:21:12.288621+0000","last_unstale":"2024-03-28T05:21:13.272013+0000","last_undegraded":"2024-03-28T05:21:13.272013+0000","last_fullsized":"2024-03-28T05:21:13.272013+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:45:31.205059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876958+0000","last_change":"2024-03-28T05:21:07.265869+0000","last_active":"2024-03-28T05:21:13.876958+0000","last_peered":"2024-03-28T05:21:13.876958+0000","last_clean":"2024-03-28T05:21:13.876958+0000","last_became_active":"2024-03-28T05:21:07.265586+0000","last_became_peered":"2024-03-28T05:21:07.265586+0000","last_unstale":"2024-03-28T05:21:13.876958+0000","last_undegraded":"2024-03-28T05:21:13.876958+0000","last_fullsized":"2024-03-28T05:21:13.876958+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:17:39.328864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759509+0000","last_change":"2024-03-28T05:21:08.046780+0000","last_active":"2024-03-28T05:21:13.759509+0000","last_peered":"2024-03-28T05:21:13.759509+0000","last_clean":"2024-03-28T05:21:13.759509+0000","last_became_active":"2024-03-28T05:21:08.046484+0000","last_became_peered":"2024-03-28T05:21:08.046484+0000","last_unstale":"2024-03-28T05:21:13.759509+0000","last_undegraded":"2024-03-28T05:21:13.759509+0000","last_fullsized":"2024-03-28T05:21:13.759509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:02:08.388581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876880+0000","last_change":"2024-03-28T05:21:12.288723+0000","last_active":"2024-03-28T05:21:13.876880+0000","last_peered":"2024-03-28T05:21:13.876880+0000","last_clean":"2024-03-28T05:21:13.876880+0000","last_became_active":"2024-03-28T05:21:12.288310+0000","last_became_peered":"2024-03-28T05:21:12.288310+0000","last_unstale":"2024-03-28T05:21:13.876880+0000","last_undegraded":"2024-03-28T05:21:13.876880+0000","last_fullsized":"2024-03-28T05:21:13.876880+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:15:39.953732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.828127+0000","last_change":"2024-03-28T05:21:07.257433+0000","last_active":"2024-03-28T05:21:13.828127+0000","last_peered":"2024-03-28T05:21:13.828127+0000","last_clean":"2024-03-28T05:21:13.828127+0000","last_became_active":"2024-03-28T05:21:07.257099+0000","last_became_peered":"2024-03-28T05:21:07.257099+0000","last_unstale":"2024-03-28T05:21:13.828127+0000","last_undegraded":"2024-03-28T05:21:13.828127+0000","last_fullsized":"2024-03-28T05:21:13.828127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:59:24.580079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876106+0000","last_change":"2024-03-28T05:21:08.047963+0000","last_active":"2024-03-28T05:21:13.876106+0000","last_peered":"2024-03-28T05:21:13.876106+0000","last_clean":"2024-03-28T05:21:13.876106+0000","last_became_active":"2024-03-28T05:21:08.046403+0000","last_became_peered":"2024-03-28T05:21:08.046403+0000","last_unstale":"2024-03-28T05:21:13.876106+0000","last_undegraded":"2024-03-28T05:21:13.876106+0000","last_fullsized":"2024-03-28T05:21:13.876106+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:58:45.280549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:14.592578+0000","last_change":"2024-03-28T05:21:12.284618+0000","last_active":"2024-03-28T05:21:14.592578+0000","last_peered":"2024-03-28T05:21:14.592578+0000","last_clean":"2024-03-28T05:21:14.592578+0000","last_became_active":"2024-03-28T05:21:12.284185+0000","last_became_peered":"2024-03-28T05:21:12.284185+0000","last_unstale":"2024-03-28T05:21:14.592578+0000","last_undegraded":"2024-03-28T05:21:14.592578+0000","last_fullsized":"2024-03-28T05:21:14.592578+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:15:48.715607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759157+0000","last_change":"2024-03-28T05:21:07.265493+0000","last_active":"2024-03-28T05:21:13.759157+0000","last_peered":"2024-03-28T05:21:13.759157+0000","last_clean":"2024-03-28T05:21:13.759157+0000","last_became_active":"2024-03-28T05:21:07.265182+0000","last_became_peered":"2024-03-28T05:21:07.265182+0000","last_unstale":"2024-03-28T05:21:13.759157+0000","last_undegraded":"2024-03-28T05:21:13.759157+0000","last_fullsized":"2024-03-28T05:21:13.759157+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:55:00.151367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.271983+0000","last_change":"2024-03-28T05:21:08.041161+0000","last_active":"2024-03-28T05:21:13.271983+0000","last_peered":"2024-03-28T05:21:13.271983+0000","last_clean":"2024-03-28T05:21:13.271983+0000","last_became_active":"2024-03-28T05:21:08.040797+0000","last_became_peered":"2024-03-28T05:21:08.040797+0000","last_unstale":"2024-03-28T05:21:13.271983+0000","last_undegraded":"2024-03-28T05:21:13.271983+0000","last_fullsized":"2024-03-28T05:21:13.271983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:05:03.387916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876164+0000","last_change":"2024-03-28T05:21:12.293759+0000","last_active":"2024-03-28T05:21:13.876164+0000","last_peered":"2024-03-28T05:21:13.876164+0000","last_clean":"2024-03-28T05:21:13.876164+0000","last_became_active":"2024-03-28T05:21:12.293355+0000","last_became_peered":"2024-03-28T05:21:12.293355+0000","last_unstale":"2024-03-28T05:21:13.876164+0000","last_undegraded":"2024-03-28T05:21:13.876164+0000","last_fullsized":"2024-03-28T05:21:13.876164+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:29:52.424113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876757+0000","last_change":"2024-03-28T05:21:07.267179+0000","last_active":"2024-03-28T05:21:13.876757+0000","last_peered":"2024-03-28T05:21:13.876757+0000","last_clean":"2024-03-28T05:21:13.876757+0000","last_became_active":"2024-03-28T05:21:07.266837+0000","last_became_peered":"2024-03-28T05:21:07.266837+0000","last_unstale":"2024-03-28T05:21:13.876757+0000","last_undegraded":"2024-03-28T05:21:13.876757+0000","last_fullsized":"2024-03-28T05:21:13.876757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:58:17.424542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.878024+0000","last_change":"2024-03-28T05:21:08.045789+0000","last_active":"2024-03-28T05:21:13.878024+0000","last_peered":"2024-03-28T05:21:13.878024+0000","last_clean":"2024-03-28T05:21:13.878024+0000","last_became_active":"2024-03-28T05:21:08.045441+0000","last_became_peered":"2024-03-28T05:21:08.045441+0000","last_unstale":"2024-03-28T05:21:13.878024+0000","last_undegraded":"2024-03-28T05:21:13.878024+0000","last_fullsized":"2024-03-28T05:21:13.878024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:15:49.968174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272142+0000","last_change":"2024-03-28T05:21:12.290300+0000","last_active":"2024-03-28T05:21:13.272142+0000","last_peered":"2024-03-28T05:21:13.272142+0000","last_clean":"2024-03-28T05:21:13.272142+0000","last_became_active":"2024-03-28T05:21:12.289817+0000","last_became_peered":"2024-03-28T05:21:12.289817+0000","last_unstale":"2024-03-28T05:21:13.272142+0000","last_undegraded":"2024-03-28T05:21:13.272142+0000","last_fullsized":"2024-03-28T05:21:13.272142+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:53:50.705017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.276045+0000","last_change":"2024-03-28T05:21:07.261302+0000","last_active":"2024-03-28T05:21:13.276045+0000","last_peered":"2024-03-28T05:21:13.276045+0000","last_clean":"2024-03-28T05:21:13.276045+0000","last_became_active":"2024-03-28T05:21:07.260991+0000","last_became_peered":"2024-03-28T05:21:07.260991+0000","last_unstale":"2024-03-28T05:21:13.276045+0000","last_undegraded":"2024-03-28T05:21:13.276045+0000","last_fullsized":"2024-03-28T05:21:13.276045+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:57:58.411737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:14.592500+0000","last_change":"2024-03-28T05:21:08.039043+0000","last_active":"2024-03-28T05:21:14.592500+0000","last_peered":"2024-03-28T05:21:14.592500+0000","last_clean":"2024-03-28T05:21:14.592500+0000","last_became_active":"2024-03-28T05:21:08.038734+0000","last_became_peered":"2024-03-28T05:21:08.038734+0000","last_unstale":"2024-03-28T05:21:14.592500+0000","last_undegraded":"2024-03-28T05:21:14.592500+0000","last_fullsized":"2024-03-28T05:21:14.592500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:26:40.155570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877211+0000","last_change":"2024-03-28T05:21:12.290037+0000","last_active":"2024-03-28T05:21:13.877211+0000","last_peered":"2024-03-28T05:21:13.877211+0000","last_clean":"2024-03-28T05:21:13.877211+0000","last_became_active":"2024-03-28T05:21:12.289526+0000","last_became_peered":"2024-03-28T05:21:12.289526+0000","last_unstale":"2024-03-28T05:21:13.877211+0000","last_undegraded":"2024-03-28T05:21:13.877211+0000","last_fullsized":"2024-03-28T05:21:13.877211+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:06:03.705716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759694+0000","last_change":"2024-03-28T05:21:07.263945+0000","last_active":"2024-03-28T05:21:13.759694+0000","last_peered":"2024-03-28T05:21:13.759694+0000","last_clean":"2024-03-28T05:21:13.759694+0000","last_became_active":"2024-03-28T05:21:07.263648+0000","last_became_peered":"2024-03-28T05:21:07.263648+0000","last_unstale":"2024-03-28T05:21:13.759694+0000","last_undegraded":"2024-03-28T05:21:13.759694+0000","last_fullsized":"2024-03-28T05:21:13.759694+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:37:28.797262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877927+0000","last_change":"2024-03-28T05:21:08.041653+0000","last_active":"2024-03-28T05:21:13.877927+0000","last_peered":"2024-03-28T05:21:13.877927+0000","last_clean":"2024-03-28T05:21:13.877927+0000","last_became_active":"2024-03-28T05:21:08.041349+0000","last_became_peered":"2024-03-28T05:21:08.041349+0000","last_unstale":"2024-03-28T05:21:13.877927+0000","last_undegraded":"2024-03-28T05:21:13.877927+0000","last_fullsized":"2024-03-28T05:21:13.877927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:51:23.329132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272178+0000","last_change":"2024-03-28T05:21:12.283320+0000","last_active":"2024-03-28T05:21:13.272178+0000","last_peered":"2024-03-28T05:21:13.272178+0000","last_clean":"2024-03-28T05:21:13.272178+0000","last_became_active":"2024-03-28T05:21:12.282893+0000","last_became_peered":"2024-03-28T05:21:12.282893+0000","last_unstale":"2024-03-28T05:21:13.272178+0000","last_undegraded":"2024-03-28T05:21:13.272178+0000","last_fullsized":"2024-03-28T05:21:13.272178+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:53:19.885680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'17","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:34.388042+0000","last_change":"2024-03-28T05:21:07.263460+0000","last_active":"2024-03-28T05:21:34.388042+0000","last_peered":"2024-03-28T05:21:34.388042+0000","last_clean":"2024-03-28T05:21:34.388042+0000","last_became_active":"2024-03-28T05:21:07.263118+0000","last_became_peered":"2024-03-28T05:21:07.263118+0000","last_unstale":"2024-03-28T05:21:34.388042+0000","last_undegraded":"2024-03-28T05:21:34.388042+0000","last_fullsized":"2024-03-28T05:21:34.388042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:56:18.271666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23141,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":27,"num_write":17,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876389+0000","last_change":"2024-03-28T05:21:08.041307+0000","last_active":"2024-03-28T05:21:13.876389+0000","last_peered":"2024-03-28T05:21:13.876389+0000","last_clean":"2024-03-28T05:21:13.876389+0000","last_became_active":"2024-03-28T05:21:08.040961+0000","last_became_peered":"2024-03-28T05:21:08.040961+0000","last_unstale":"2024-03-28T05:21:13.876389+0000","last_undegraded":"2024-03-28T05:21:13.876389+0000","last_fullsized":"2024-03-28T05:21:13.876389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:12:40.318091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.760184+0000","last_change":"2024-03-28T05:21:12.287222+0000","last_active":"2024-03-28T05:21:13.760184+0000","last_peered":"2024-03-28T05:21:13.760184+0000","last_clean":"2024-03-28T05:21:13.760184+0000","last_became_active":"2024-03-28T05:21:12.286744+0000","last_became_peered":"2024-03-28T05:21:12.286744+0000","last_unstale":"2024-03-28T05:21:13.760184+0000","last_undegraded":"2024-03-28T05:21:13.760184+0000","last_fullsized":"2024-03-28T05:21:13.760184+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:20:45.698054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272057+0000","last_change":"2024-03-28T05:21:07.262932+0000","last_active":"2024-03-28T05:21:13.272057+0000","last_peered":"2024-03-28T05:21:13.272057+0000","last_clean":"2024-03-28T05:21:13.272057+0000","last_became_active":"2024-03-28T05:21:07.262497+0000","last_became_peered":"2024-03-28T05:21:07.262497+0000","last_unstale":"2024-03-28T05:21:13.272057+0000","last_undegraded":"2024-03-28T05:21:13.272057+0000","last_fullsized":"2024-03-28T05:21:13.272057+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:56:58.628442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.828230+0000","last_change":"2024-03-28T05:21:08.046052+0000","last_active":"2024-03-28T05:21:13.828230+0000","last_peered":"2024-03-28T05:21:13.828230+0000","last_clean":"2024-03-28T05:21:13.828230+0000","last_became_active":"2024-03-28T05:21:08.045720+0000","last_became_peered":"2024-03-28T05:21:08.045720+0000","last_unstale":"2024-03-28T05:21:13.828230+0000","last_undegraded":"2024-03-28T05:21:13.828230+0000","last_fullsized":"2024-03-28T05:21:13.828230+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:39:48.084984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876778+0000","last_change":"2024-03-28T05:21:12.294339+0000","last_active":"2024-03-28T05:21:13.876778+0000","last_peered":"2024-03-28T05:21:13.876778+0000","last_clean":"2024-03-28T05:21:13.876778+0000","last_became_active":"2024-03-28T05:21:12.293939+0000","last_became_peered":"2024-03-28T05:21:12.293939+0000","last_unstale":"2024-03-28T05:21:13.876778+0000","last_undegraded":"2024-03-28T05:21:13.876778+0000","last_fullsized":"2024-03-28T05:21:13.876778+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:38:59.851650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759432+0000","last_change":"2024-03-28T05:21:07.262414+0000","last_active":"2024-03-28T05:21:13.759432+0000","last_peered":"2024-03-28T05:21:13.759432+0000","last_clean":"2024-03-28T05:21:13.759432+0000","last_became_active":"2024-03-28T05:21:07.262121+0000","last_became_peered":"2024-03-28T05:21:07.262121+0000","last_unstale":"2024-03-28T05:21:13.759432+0000","last_undegraded":"2024-03-28T05:21:13.759432+0000","last_fullsized":"2024-03-28T05:21:13.759432+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:15:52.832449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.275896+0000","last_change":"2024-03-28T05:21:08.049263+0000","last_active":"2024-03-28T05:21:13.275896+0000","last_peered":"2024-03-28T05:21:13.275896+0000","last_clean":"2024-03-28T05:21:13.275896+0000","last_became_active":"2024-03-28T05:21:08.048912+0000","last_became_peered":"2024-03-28T05:21:08.048912+0000","last_unstale":"2024-03-28T05:21:13.275896+0000","last_undegraded":"2024-03-28T05:21:13.275896+0000","last_fullsized":"2024-03-28T05:21:13.275896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:36:06.562731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272523+0000","last_change":"2024-03-28T05:21:12.290345+0000","last_active":"2024-03-28T05:21:13.272523+0000","last_peered":"2024-03-28T05:21:13.272523+0000","last_clean":"2024-03-28T05:21:13.272523+0000","last_became_active":"2024-03-28T05:21:12.289945+0000","last_became_peered":"2024-03-28T05:21:12.289945+0000","last_unstale":"2024-03-28T05:21:13.272523+0000","last_undegraded":"2024-03-28T05:21:13.272523+0000","last_fullsized":"2024-03-28T05:21:13.272523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:41:43.024707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876845+0000","last_change":"2024-03-28T05:21:07.262588+0000","last_active":"2024-03-28T05:21:13.876845+0000","last_peered":"2024-03-28T05:21:13.876845+0000","last_clean":"2024-03-28T05:21:13.876845+0000","last_became_active":"2024-03-28T05:21:07.262237+0000","last_became_peered":"2024-03-28T05:21:07.262237+0000","last_unstale":"2024-03-28T05:21:13.876845+0000","last_undegraded":"2024-03-28T05:21:13.876845+0000","last_fullsized":"2024-03-28T05:21:13.876845+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:49:08.207031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.275533+0000","last_change":"2024-03-28T05:21:08.041473+0000","last_active":"2024-03-28T05:21:13.275533+0000","last_peered":"2024-03-28T05:21:13.275533+0000","last_clean":"2024-03-28T05:21:13.275533+0000","last_became_active":"2024-03-28T05:21:08.041171+0000","last_became_peered":"2024-03-28T05:21:08.041171+0000","last_unstale":"2024-03-28T05:21:13.275533+0000","last_undegraded":"2024-03-28T05:21:13.275533+0000","last_fullsized":"2024-03-28T05:21:13.275533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:56:26.652129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.828711+0000","last_change":"2024-03-28T05:21:12.296288+0000","last_active":"2024-03-28T05:21:13.828711+0000","last_peered":"2024-03-28T05:21:13.828711+0000","last_clean":"2024-03-28T05:21:13.828711+0000","last_became_active":"2024-03-28T05:21:12.295896+0000","last_became_peered":"2024-03-28T05:21:12.295896+0000","last_unstale":"2024-03-28T05:21:13.828711+0000","last_undegraded":"2024-03-28T05:21:13.828711+0000","last_fullsized":"2024-03-28T05:21:13.828711+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:05:05.779622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.875895+0000","last_change":"2024-03-28T05:21:07.262448+0000","last_active":"2024-03-28T05:21:13.875895+0000","last_peered":"2024-03-28T05:21:13.875895+0000","last_clean":"2024-03-28T05:21:13.875895+0000","last_became_active":"2024-03-28T05:21:07.261966+0000","last_became_peered":"2024-03-28T05:21:07.261966+0000","last_unstale":"2024-03-28T05:21:13.875895+0000","last_undegraded":"2024-03-28T05:21:13.875895+0000","last_fullsized":"2024-03-28T05:21:13.875895+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:45:37.904160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759270+0000","last_change":"2024-03-28T05:21:08.045929+0000","last_active":"2024-03-28T05:21:13.759270+0000","last_peered":"2024-03-28T05:21:13.759270+0000","last_clean":"2024-03-28T05:21:13.759270+0000","last_became_active":"2024-03-28T05:21:08.045640+0000","last_became_peered":"2024-03-28T05:21:08.045640+0000","last_unstale":"2024-03-28T05:21:13.759270+0000","last_undegraded":"2024-03-28T05:21:13.759270+0000","last_fullsized":"2024-03-28T05:21:13.759270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:38:01.546332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759293+0000","last_change":"2024-03-28T05:21:12.292516+0000","last_active":"2024-03-28T05:21:13.759293+0000","last_peered":"2024-03-28T05:21:13.759293+0000","last_clean":"2024-03-28T05:21:13.759293+0000","last_became_active":"2024-03-28T05:21:12.292129+0000","last_became_peered":"2024-03-28T05:21:12.292129+0000","last_unstale":"2024-03-28T05:21:13.759293+0000","last_undegraded":"2024-03-28T05:21:13.759293+0000","last_fullsized":"2024-03-28T05:21:13.759293+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:32:25.086143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876443+0000","last_change":"2024-03-28T05:21:07.267442+0000","last_active":"2024-03-28T05:21:13.876443+0000","last_peered":"2024-03-28T05:21:13.876443+0000","last_clean":"2024-03-28T05:21:13.876443+0000","last_became_active":"2024-03-28T05:21:07.267140+0000","last_became_peered":"2024-03-28T05:21:07.267140+0000","last_unstale":"2024-03-28T05:21:13.876443+0000","last_undegraded":"2024-03-28T05:21:13.876443+0000","last_fullsized":"2024-03-28T05:21:13.876443+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:43:05.543211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876743+0000","last_change":"2024-03-28T05:21:08.040304+0000","last_active":"2024-03-28T05:21:13.876743+0000","last_peered":"2024-03-28T05:21:13.876743+0000","last_clean":"2024-03-28T05:21:13.876743+0000","last_became_active":"2024-03-28T05:21:08.039976+0000","last_became_peered":"2024-03-28T05:21:08.039976+0000","last_unstale":"2024-03-28T05:21:13.876743+0000","last_undegraded":"2024-03-28T05:21:13.876743+0000","last_fullsized":"2024-03-28T05:21:13.876743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:57:33.062202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876082+0000","last_change":"2024-03-28T05:21:12.289177+0000","last_active":"2024-03-28T05:21:13.876082+0000","last_peered":"2024-03-28T05:21:13.876082+0000","last_clean":"2024-03-28T05:21:13.876082+0000","last_became_active":"2024-03-28T05:21:12.288615+0000","last_became_peered":"2024-03-28T05:21:12.288615+0000","last_unstale":"2024-03-28T05:21:13.876082+0000","last_undegraded":"2024-03-28T05:21:13.876082+0000","last_fullsized":"2024-03-28T05:21:13.876082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:27:51.819911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272089+0000","last_change":"2024-03-28T05:21:07.272325+0000","last_active":"2024-03-28T05:21:13.272089+0000","last_peered":"2024-03-28T05:21:13.272089+0000","last_clean":"2024-03-28T05:21:13.272089+0000","last_became_active":"2024-03-28T05:21:07.271965+0000","last_became_peered":"2024-03-28T05:21:07.271965+0000","last_unstale":"2024-03-28T05:21:13.272089+0000","last_undegraded":"2024-03-28T05:21:13.272089+0000","last_fullsized":"2024-03-28T05:21:13.272089+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:20:24.723627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877346+0000","last_change":"2024-03-28T05:21:08.047122+0000","last_active":"2024-03-28T05:21:13.877346+0000","last_peered":"2024-03-28T05:21:13.877346+0000","last_clean":"2024-03-28T05:21:13.877346+0000","last_became_active":"2024-03-28T05:21:08.046828+0000","last_became_peered":"2024-03-28T05:21:08.046828+0000","last_unstale":"2024-03-28T05:21:13.877346+0000","last_undegraded":"2024-03-28T05:21:13.877346+0000","last_fullsized":"2024-03-28T05:21:13.877346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:07:16.940600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.828348+0000","last_change":"2024-03-28T05:21:12.293325+0000","last_active":"2024-03-28T05:21:13.828348+0000","last_peered":"2024-03-28T05:21:13.828348+0000","last_clean":"2024-03-28T05:21:13.828348+0000","last_became_active":"2024-03-28T05:21:12.292876+0000","last_became_peered":"2024-03-28T05:21:12.292876+0000","last_unstale":"2024-03-28T05:21:13.828348+0000","last_undegraded":"2024-03-28T05:21:13.828348+0000","last_fullsized":"2024-03-28T05:21:13.828348+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:35:25.173848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.275661+0000","last_change":"2024-03-28T05:21:07.262435+0000","last_active":"2024-03-28T05:21:13.275661+0000","last_peered":"2024-03-28T05:21:13.275661+0000","last_clean":"2024-03-28T05:21:13.275661+0000","last_became_active":"2024-03-28T05:21:07.262019+0000","last_became_peered":"2024-03-28T05:21:07.262019+0000","last_unstale":"2024-03-28T05:21:13.275661+0000","last_undegraded":"2024-03-28T05:21:13.275661+0000","last_fullsized":"2024-03-28T05:21:13.275661+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:41:31.348300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876993+0000","last_change":"2024-03-28T05:21:08.043614+0000","last_active":"2024-03-28T05:21:13.876993+0000","last_peered":"2024-03-28T05:21:13.876993+0000","last_clean":"2024-03-28T05:21:13.876993+0000","last_became_active":"2024-03-28T05:21:08.043187+0000","last_became_peered":"2024-03-28T05:21:08.043187+0000","last_unstale":"2024-03-28T05:21:13.876993+0000","last_undegraded":"2024-03-28T05:21:13.876993+0000","last_fullsized":"2024-03-28T05:21:13.876993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:55:08.972707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.828505+0000","last_change":"2024-03-28T05:21:12.289381+0000","last_active":"2024-03-28T05:21:13.828505+0000","last_peered":"2024-03-28T05:21:13.828505+0000","last_clean":"2024-03-28T05:21:13.828505+0000","last_became_active":"2024-03-28T05:21:12.288987+0000","last_became_peered":"2024-03-28T05:21:12.288987+0000","last_unstale":"2024-03-28T05:21:13.828505+0000","last_undegraded":"2024-03-28T05:21:13.828505+0000","last_fullsized":"2024-03-28T05:21:13.828505+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:31:46.296664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759296+0000","last_change":"2024-03-28T05:21:08.047077+0000","last_active":"2024-03-28T05:21:13.759296+0000","last_peered":"2024-03-28T05:21:13.759296+0000","last_clean":"2024-03-28T05:21:13.759296+0000","last_became_active":"2024-03-28T05:21:08.046783+0000","last_became_peered":"2024-03-28T05:21:08.046783+0000","last_unstale":"2024-03-28T05:21:13.759296+0000","last_undegraded":"2024-03-28T05:21:13.759296+0000","last_fullsized":"2024-03-28T05:21:13.759296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:52:07.823247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.827997+0000","last_change":"2024-03-28T05:21:07.271028+0000","last_active":"2024-03-28T05:21:13.827997+0000","last_peered":"2024-03-28T05:21:13.827997+0000","last_clean":"2024-03-28T05:21:13.827997+0000","last_became_active":"2024-03-28T05:21:07.270404+0000","last_became_peered":"2024-03-28T05:21:07.270404+0000","last_unstale":"2024-03-28T05:21:13.827997+0000","last_undegraded":"2024-03-28T05:21:13.827997+0000","last_fullsized":"2024-03-28T05:21:13.827997+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:01:23.875231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.275924+0000","last_change":"2024-03-28T05:21:12.289881+0000","last_active":"2024-03-28T05:21:13.275924+0000","last_peered":"2024-03-28T05:21:13.275924+0000","last_clean":"2024-03-28T05:21:13.275924+0000","last_became_active":"2024-03-28T05:21:12.289476+0000","last_became_peered":"2024-03-28T05:21:12.289476+0000","last_unstale":"2024-03-28T05:21:13.275924+0000","last_undegraded":"2024-03-28T05:21:13.275924+0000","last_fullsized":"2024-03-28T05:21:13.275924+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:01:31.800429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876381+0000","last_change":"2024-03-28T05:21:08.044169+0000","last_active":"2024-03-28T05:21:13.876381+0000","last_peered":"2024-03-28T05:21:13.876381+0000","last_clean":"2024-03-28T05:21:13.876381+0000","last_became_active":"2024-03-28T05:21:08.043836+0000","last_became_peered":"2024-03-28T05:21:08.043836+0000","last_unstale":"2024-03-28T05:21:13.876381+0000","last_undegraded":"2024-03-28T05:21:13.876381+0000","last_fullsized":"2024-03-28T05:21:13.876381+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:07:41.128111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876912+0000","last_change":"2024-03-28T05:21:07.267368+0000","last_active":"2024-03-28T05:21:13.876912+0000","last_peered":"2024-03-28T05:21:13.876912+0000","last_clean":"2024-03-28T05:21:13.876912+0000","last_became_active":"2024-03-28T05:21:07.266995+0000","last_became_peered":"2024-03-28T05:21:07.266995+0000","last_unstale":"2024-03-28T05:21:13.876912+0000","last_undegraded":"2024-03-28T05:21:13.876912+0000","last_fullsized":"2024-03-28T05:21:13.876912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:24:40.146950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":356,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.275411+0000","last_change":"2024-03-28T05:19:57.662704+0000","last_active":"2024-03-28T05:21:13.275411+0000","last_peered":"2024-03-28T05:21:13.275411+0000","last_clean":"2024-03-28T05:21:13.275411+0000","last_became_active":"2024-03-28T05:19:57.662332+0000","last_became_peered":"2024-03-28T05:19:57.662332+0000","last_unstale":"2024-03-28T05:21:13.275411+0000","last_undegraded":"2024-03-28T05:21:13.275411+0000","last_fullsized":"2024-03-28T05:21:13.275411+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:16:33.255097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:16:33.255097+0000","last_clean_scrub_stamp":"2024-03-28T05:16:33.255097+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:40:47.655643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759653+0000","last_change":"2024-03-28T05:21:12.291993+0000","last_active":"2024-03-28T05:21:13.759653+0000","last_peered":"2024-03-28T05:21:13.759653+0000","last_clean":"2024-03-28T05:21:13.759653+0000","last_became_active":"2024-03-28T05:21:12.291641+0000","last_became_peered":"2024-03-28T05:21:12.291641+0000","last_unstale":"2024-03-28T05:21:13.759653+0000","last_undegraded":"2024-03-28T05:21:13.759653+0000","last_fullsized":"2024-03-28T05:21:13.759653+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:12:02.217568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.827988+0000","last_change":"2024-03-28T05:21:08.043036+0000","last_active":"2024-03-28T05:21:13.827988+0000","last_peered":"2024-03-28T05:21:13.827988+0000","last_clean":"2024-03-28T05:21:13.827988+0000","last_became_active":"2024-03-28T05:21:08.042467+0000","last_became_peered":"2024-03-28T05:21:08.042467+0000","last_unstale":"2024-03-28T05:21:13.827988+0000","last_undegraded":"2024-03-28T05:21:13.827988+0000","last_fullsized":"2024-03-28T05:21:13.827988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:16:41.718497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.275466+0000","last_change":"2024-03-28T05:21:07.260270+0000","last_active":"2024-03-28T05:21:13.275466+0000","last_peered":"2024-03-28T05:21:13.275466+0000","last_clean":"2024-03-28T05:21:13.275466+0000","last_became_active":"2024-03-28T05:21:07.259727+0000","last_became_peered":"2024-03-28T05:21:07.259727+0000","last_unstale":"2024-03-28T05:21:13.275466+0000","last_undegraded":"2024-03-28T05:21:13.275466+0000","last_fullsized":"2024-03-28T05:21:13.275466+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:19:15.883172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877032+0000","last_change":"2024-03-28T05:21:12.294439+0000","last_active":"2024-03-28T05:21:13.877032+0000","last_peered":"2024-03-28T05:21:13.877032+0000","last_clean":"2024-03-28T05:21:13.877032+0000","last_became_active":"2024-03-28T05:21:12.294044+0000","last_became_peered":"2024-03-28T05:21:12.294044+0000","last_unstale":"2024-03-28T05:21:13.877032+0000","last_undegraded":"2024-03-28T05:21:13.877032+0000","last_fullsized":"2024-03-28T05:21:13.877032+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:22:51.218266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876398+0000","last_change":"2024-03-28T05:21:07.272735+0000","last_active":"2024-03-28T05:21:13.876398+0000","last_peered":"2024-03-28T05:21:13.876398+0000","last_clean":"2024-03-28T05:21:13.876398+0000","last_became_active":"2024-03-28T05:21:07.272407+0000","last_became_peered":"2024-03-28T05:21:07.272407+0000","last_unstale":"2024-03-28T05:21:13.876398+0000","last_undegraded":"2024-03-28T05:21:13.876398+0000","last_fullsized":"2024-03-28T05:21:13.876398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:55:02.165183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.828555+0000","last_change":"2024-03-28T05:21:08.042577+0000","last_active":"2024-03-28T05:21:13.828555+0000","last_peered":"2024-03-28T05:21:13.828555+0000","last_clean":"2024-03-28T05:21:13.828555+0000","last_became_active":"2024-03-28T05:21:08.042248+0000","last_became_peered":"2024-03-28T05:21:08.042248+0000","last_unstale":"2024-03-28T05:21:13.828555+0000","last_undegraded":"2024-03-28T05:21:13.828555+0000","last_fullsized":"2024-03-28T05:21:13.828555+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:31:15.856177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876108+0000","last_change":"2024-03-28T05:21:12.294141+0000","last_active":"2024-03-28T05:21:13.876108+0000","last_peered":"2024-03-28T05:21:13.876108+0000","last_clean":"2024-03-28T05:21:13.876108+0000","last_became_active":"2024-03-28T05:21:12.293746+0000","last_became_peered":"2024-03-28T05:21:12.293746+0000","last_unstale":"2024-03-28T05:21:13.876108+0000","last_undegraded":"2024-03-28T05:21:13.876108+0000","last_fullsized":"2024-03-28T05:21:13.876108+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:06:18.871631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272615+0000","last_change":"2024-03-28T05:21:07.275515+0000","last_active":"2024-03-28T05:21:13.272615+0000","last_peered":"2024-03-28T05:21:13.272615+0000","last_clean":"2024-03-28T05:21:13.272615+0000","last_became_active":"2024-03-28T05:21:07.275188+0000","last_became_peered":"2024-03-28T05:21:07.275188+0000","last_unstale":"2024-03-28T05:21:13.272615+0000","last_undegraded":"2024-03-28T05:21:13.272615+0000","last_fullsized":"2024-03-28T05:21:13.272615+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:34:29.351759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876578+0000","last_change":"2024-03-28T05:21:08.051019+0000","last_active":"2024-03-28T05:21:13.876578+0000","last_peered":"2024-03-28T05:21:13.876578+0000","last_clean":"2024-03-28T05:21:13.876578+0000","last_became_active":"2024-03-28T05:21:08.050696+0000","last_became_peered":"2024-03-28T05:21:08.050696+0000","last_unstale":"2024-03-28T05:21:13.876578+0000","last_undegraded":"2024-03-28T05:21:13.876578+0000","last_fullsized":"2024-03-28T05:21:13.876578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:06:23.318153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.271749+0000","last_change":"2024-03-28T05:21:12.285980+0000","last_active":"2024-03-28T05:21:13.271749+0000","last_peered":"2024-03-28T05:21:13.271749+0000","last_clean":"2024-03-28T05:21:13.271749+0000","last_became_active":"2024-03-28T05:21:12.285578+0000","last_became_peered":"2024-03-28T05:21:12.285578+0000","last_unstale":"2024-03-28T05:21:13.271749+0000","last_undegraded":"2024-03-28T05:21:13.271749+0000","last_fullsized":"2024-03-28T05:21:13.271749+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:33:58.377753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877473+0000","last_change":"2024-03-28T05:21:07.256475+0000","last_active":"2024-03-28T05:21:13.877473+0000","last_peered":"2024-03-28T05:21:13.877473+0000","last_clean":"2024-03-28T05:21:13.877473+0000","last_became_active":"2024-03-28T05:21:07.256125+0000","last_became_peered":"2024-03-28T05:21:07.256125+0000","last_unstale":"2024-03-28T05:21:13.877473+0000","last_undegraded":"2024-03-28T05:21:13.877473+0000","last_fullsized":"2024-03-28T05:21:13.877473+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:42.263704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:14.592660+0000","last_change":"2024-03-28T05:21:08.044417+0000","last_active":"2024-03-28T05:21:14.592660+0000","last_peered":"2024-03-28T05:21:14.592660+0000","last_clean":"2024-03-28T05:21:14.592660+0000","last_became_active":"2024-03-28T05:21:08.043931+0000","last_became_peered":"2024-03-28T05:21:08.043931+0000","last_unstale":"2024-03-28T05:21:14.592660+0000","last_undegraded":"2024-03-28T05:21:14.592660+0000","last_fullsized":"2024-03-28T05:21:14.592660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:43:55.013625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876747+0000","last_change":"2024-03-28T05:21:12.300641+0000","last_active":"2024-03-28T05:21:13.876747+0000","last_peered":"2024-03-28T05:21:13.876747+0000","last_clean":"2024-03-28T05:21:13.876747+0000","last_became_active":"2024-03-28T05:21:12.300013+0000","last_became_peered":"2024-03-28T05:21:12.300013+0000","last_unstale":"2024-03-28T05:21:13.876747+0000","last_undegraded":"2024-03-28T05:21:13.876747+0000","last_fullsized":"2024-03-28T05:21:13.876747+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:38:42.474798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876368+0000","last_change":"2024-03-28T05:21:07.261548+0000","last_active":"2024-03-28T05:21:13.876368+0000","last_peered":"2024-03-28T05:21:13.876368+0000","last_clean":"2024-03-28T05:21:13.876368+0000","last_became_active":"2024-03-28T05:21:07.261201+0000","last_became_peered":"2024-03-28T05:21:07.261201+0000","last_unstale":"2024-03-28T05:21:13.876368+0000","last_undegraded":"2024-03-28T05:21:13.876368+0000","last_fullsized":"2024-03-28T05:21:13.876368+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:35:48.899064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.875866+0000","last_change":"2024-03-28T05:21:08.045168+0000","last_active":"2024-03-28T05:21:13.875866+0000","last_peered":"2024-03-28T05:21:13.875866+0000","last_clean":"2024-03-28T05:21:13.875866+0000","last_became_active":"2024-03-28T05:21:08.044654+0000","last_became_peered":"2024-03-28T05:21:08.044654+0000","last_unstale":"2024-03-28T05:21:13.875866+0000","last_undegraded":"2024-03-28T05:21:13.875866+0000","last_fullsized":"2024-03-28T05:21:13.875866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:48:14.592798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272530+0000","last_change":"2024-03-28T05:21:12.285593+0000","last_active":"2024-03-28T05:21:13.272530+0000","last_peered":"2024-03-28T05:21:13.272530+0000","last_clean":"2024-03-28T05:21:13.272530+0000","last_became_active":"2024-03-28T05:21:12.285202+0000","last_became_peered":"2024-03-28T05:21:12.285202+0000","last_unstale":"2024-03-28T05:21:13.272530+0000","last_undegraded":"2024-03-28T05:21:13.272530+0000","last_fullsized":"2024-03-28T05:21:13.272530+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:00:40.198255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876134+0000","last_change":"2024-03-28T05:21:07.271352+0000","last_active":"2024-03-28T05:21:13.876134+0000","last_peered":"2024-03-28T05:21:13.876134+0000","last_clean":"2024-03-28T05:21:13.876134+0000","last_became_active":"2024-03-28T05:21:07.271038+0000","last_became_peered":"2024-03-28T05:21:07.271038+0000","last_unstale":"2024-03-28T05:21:13.876134+0000","last_undegraded":"2024-03-28T05:21:13.876134+0000","last_fullsized":"2024-03-28T05:21:13.876134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:37:41.742550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876529+0000","last_change":"2024-03-28T05:21:08.043503+0000","last_active":"2024-03-28T05:21:13.876529+0000","last_peered":"2024-03-28T05:21:13.876529+0000","last_clean":"2024-03-28T05:21:13.876529+0000","last_became_active":"2024-03-28T05:21:08.043173+0000","last_became_peered":"2024-03-28T05:21:08.043173+0000","last_unstale":"2024-03-28T05:21:13.876529+0000","last_undegraded":"2024-03-28T05:21:13.876529+0000","last_fullsized":"2024-03-28T05:21:13.876529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:23:30.576539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877262+0000","last_change":"2024-03-28T05:21:12.300600+0000","last_active":"2024-03-28T05:21:13.877262+0000","last_peered":"2024-03-28T05:21:13.877262+0000","last_clean":"2024-03-28T05:21:13.877262+0000","last_became_active":"2024-03-28T05:21:12.299931+0000","last_became_peered":"2024-03-28T05:21:12.299931+0000","last_unstale":"2024-03-28T05:21:13.877262+0000","last_undegraded":"2024-03-28T05:21:13.877262+0000","last_fullsized":"2024-03-28T05:21:13.877262+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:29:22.405410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.271831+0000","last_change":"2024-03-28T05:21:07.265708+0000","last_active":"2024-03-28T05:21:13.271831+0000","last_peered":"2024-03-28T05:21:13.271831+0000","last_clean":"2024-03-28T05:21:13.271831+0000","last_became_active":"2024-03-28T05:21:07.265416+0000","last_became_peered":"2024-03-28T05:21:07.265416+0000","last_unstale":"2024-03-28T05:21:13.271831+0000","last_undegraded":"2024-03-28T05:21:13.271831+0000","last_fullsized":"2024-03-28T05:21:13.271831+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:14:05.003658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876256+0000","last_change":"2024-03-28T05:21:08.051458+0000","last_active":"2024-03-28T05:21:13.876256+0000","last_peered":"2024-03-28T05:21:13.876256+0000","last_clean":"2024-03-28T05:21:13.876256+0000","last_became_active":"2024-03-28T05:21:08.050997+0000","last_became_peered":"2024-03-28T05:21:08.050997+0000","last_unstale":"2024-03-28T05:21:13.876256+0000","last_undegraded":"2024-03-28T05:21:13.876256+0000","last_fullsized":"2024-03-28T05:21:13.876256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:33:41.046841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876764+0000","last_change":"2024-03-28T05:21:12.285977+0000","last_active":"2024-03-28T05:21:13.876764+0000","last_peered":"2024-03-28T05:21:13.876764+0000","last_clean":"2024-03-28T05:21:13.876764+0000","last_became_active":"2024-03-28T05:21:12.285439+0000","last_became_peered":"2024-03-28T05:21:12.285439+0000","last_unstale":"2024-03-28T05:21:13.876764+0000","last_undegraded":"2024-03-28T05:21:13.876764+0000","last_fullsized":"2024-03-28T05:21:13.876764+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:07:34.409001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876144+0000","last_change":"2024-03-28T05:21:07.268187+0000","last_active":"2024-03-28T05:21:13.876144+0000","last_peered":"2024-03-28T05:21:13.876144+0000","last_clean":"2024-03-28T05:21:13.876144+0000","last_became_active":"2024-03-28T05:21:07.266791+0000","last_became_peered":"2024-03-28T05:21:07.266791+0000","last_unstale":"2024-03-28T05:21:13.876144+0000","last_undegraded":"2024-03-28T05:21:13.876144+0000","last_fullsized":"2024-03-28T05:21:13.876144+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:22:03.683043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.275846+0000","last_change":"2024-03-28T05:21:08.054067+0000","last_active":"2024-03-28T05:21:13.275846+0000","last_peered":"2024-03-28T05:21:13.275846+0000","last_clean":"2024-03-28T05:21:13.275846+0000","last_became_active":"2024-03-28T05:21:08.053759+0000","last_became_peered":"2024-03-28T05:21:08.053759+0000","last_unstale":"2024-03-28T05:21:13.275846+0000","last_undegraded":"2024-03-28T05:21:13.275846+0000","last_fullsized":"2024-03-28T05:21:13.275846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:47:39.329564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876564+0000","last_change":"2024-03-28T05:21:12.294547+0000","last_active":"2024-03-28T05:21:13.876564+0000","last_peered":"2024-03-28T05:21:13.876564+0000","last_clean":"2024-03-28T05:21:13.876564+0000","last_became_active":"2024-03-28T05:21:12.294146+0000","last_became_peered":"2024-03-28T05:21:12.294146+0000","last_unstale":"2024-03-28T05:21:13.876564+0000","last_undegraded":"2024-03-28T05:21:13.876564+0000","last_fullsized":"2024-03-28T05:21:13.876564+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:24:28.939378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272145+0000","last_change":"2024-03-28T05:21:07.260749+0000","last_active":"2024-03-28T05:21:13.272145+0000","last_peered":"2024-03-28T05:21:13.272145+0000","last_clean":"2024-03-28T05:21:13.272145+0000","last_became_active":"2024-03-28T05:21:07.260439+0000","last_became_peered":"2024-03-28T05:21:07.260439+0000","last_unstale":"2024-03-28T05:21:13.272145+0000","last_undegraded":"2024-03-28T05:21:13.272145+0000","last_fullsized":"2024-03-28T05:21:13.272145+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:24:04.257080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.276133+0000","last_change":"2024-03-28T05:21:08.051079+0000","last_active":"2024-03-28T05:21:13.276133+0000","last_peered":"2024-03-28T05:21:13.276133+0000","last_clean":"2024-03-28T05:21:13.276133+0000","last_became_active":"2024-03-28T05:21:08.050790+0000","last_became_peered":"2024-03-28T05:21:08.050790+0000","last_unstale":"2024-03-28T05:21:13.276133+0000","last_undegraded":"2024-03-28T05:21:13.276133+0000","last_fullsized":"2024-03-28T05:21:13.276133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:04:39.205068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:14.592244+0000","last_change":"2024-03-28T05:21:12.291964+0000","last_active":"2024-03-28T05:21:14.592244+0000","last_peered":"2024-03-28T05:21:14.592244+0000","last_clean":"2024-03-28T05:21:14.592244+0000","last_became_active":"2024-03-28T05:21:12.291553+0000","last_became_peered":"2024-03-28T05:21:12.291553+0000","last_unstale":"2024-03-28T05:21:14.592244+0000","last_undegraded":"2024-03-28T05:21:14.592244+0000","last_fullsized":"2024-03-28T05:21:14.592244+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:10:34.180033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759896+0000","last_change":"2024-03-28T05:21:07.263204+0000","last_active":"2024-03-28T05:21:13.759896+0000","last_peered":"2024-03-28T05:21:13.759896+0000","last_clean":"2024-03-28T05:21:13.759896+0000","last_became_active":"2024-03-28T05:21:07.262919+0000","last_became_peered":"2024-03-28T05:21:07.262919+0000","last_unstale":"2024-03-28T05:21:13.759896+0000","last_undegraded":"2024-03-28T05:21:13.759896+0000","last_fullsized":"2024-03-28T05:21:13.759896+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:53:19.505448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.275833+0000","last_change":"2024-03-28T05:21:08.044499+0000","last_active":"2024-03-28T05:21:13.275833+0000","last_peered":"2024-03-28T05:21:13.275833+0000","last_clean":"2024-03-28T05:21:13.275833+0000","last_became_active":"2024-03-28T05:21:08.043371+0000","last_became_peered":"2024-03-28T05:21:08.043371+0000","last_unstale":"2024-03-28T05:21:13.275833+0000","last_undegraded":"2024-03-28T05:21:13.275833+0000","last_fullsized":"2024-03-28T05:21:13.275833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:44:35.901625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759128+0000","last_change":"2024-03-28T05:21:12.282443+0000","last_active":"2024-03-28T05:21:13.759128+0000","last_peered":"2024-03-28T05:21:13.759128+0000","last_clean":"2024-03-28T05:21:13.759128+0000","last_became_active":"2024-03-28T05:21:12.282065+0000","last_became_peered":"2024-03-28T05:21:12.282065+0000","last_unstale":"2024-03-28T05:21:13.759128+0000","last_undegraded":"2024-03-28T05:21:13.759128+0000","last_fullsized":"2024-03-28T05:21:13.759128+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:26:55.366554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.828687+0000","last_change":"2024-03-28T05:21:07.266038+0000","last_active":"2024-03-28T05:21:13.828687+0000","last_peered":"2024-03-28T05:21:13.828687+0000","last_clean":"2024-03-28T05:21:13.828687+0000","last_became_active":"2024-03-28T05:21:07.265728+0000","last_became_peered":"2024-03-28T05:21:07.265728+0000","last_unstale":"2024-03-28T05:21:13.828687+0000","last_undegraded":"2024-03-28T05:21:13.828687+0000","last_fullsized":"2024-03-28T05:21:13.828687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:06:05.539565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.276136+0000","last_change":"2024-03-28T05:21:08.047626+0000","last_active":"2024-03-28T05:21:13.276136+0000","last_peered":"2024-03-28T05:21:13.276136+0000","last_clean":"2024-03-28T05:21:13.276136+0000","last_became_active":"2024-03-28T05:21:08.047251+0000","last_became_peered":"2024-03-28T05:21:08.047251+0000","last_unstale":"2024-03-28T05:21:13.276136+0000","last_undegraded":"2024-03-28T05:21:13.276136+0000","last_fullsized":"2024-03-28T05:21:13.276136+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:25:36.043024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876594+0000","last_change":"2024-03-28T05:21:08.042216+0000","last_active":"2024-03-28T05:21:13.876594+0000","last_peered":"2024-03-28T05:21:13.876594+0000","last_clean":"2024-03-28T05:21:13.876594+0000","last_became_active":"2024-03-28T05:21:08.041830+0000","last_became_peered":"2024-03-28T05:21:08.041830+0000","last_unstale":"2024-03-28T05:21:13.876594+0000","last_undegraded":"2024-03-28T05:21:13.876594+0000","last_fullsized":"2024-03-28T05:21:13.876594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:56:19.785472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.271813+0000","last_change":"2024-03-28T05:21:08.042043+0000","last_active":"2024-03-28T05:21:13.271813+0000","last_peered":"2024-03-28T05:21:13.271813+0000","last_clean":"2024-03-28T05:21:13.271813+0000","last_became_active":"2024-03-28T05:21:08.041708+0000","last_became_peered":"2024-03-28T05:21:08.041708+0000","last_unstale":"2024-03-28T05:21:13.271813+0000","last_undegraded":"2024-03-28T05:21:13.271813+0000","last_fullsized":"2024-03-28T05:21:13.271813+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:11:36.620653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.271728+0000","last_change":"2024-03-28T05:21:08.043471+0000","last_active":"2024-03-28T05:21:13.271728+0000","last_peered":"2024-03-28T05:21:13.271728+0000","last_clean":"2024-03-28T05:21:13.271728+0000","last_became_active":"2024-03-28T05:21:08.043144+0000","last_became_peered":"2024-03-28T05:21:08.043144+0000","last_unstale":"2024-03-28T05:21:13.271728+0000","last_undegraded":"2024-03-28T05:21:13.271728+0000","last_fullsized":"2024-03-28T05:21:13.271728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:18:40.580589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:14.592076+0000","last_change":"2024-03-28T05:21:08.050105+0000","last_active":"2024-03-28T05:21:14.592076+0000","last_peered":"2024-03-28T05:21:14.592076+0000","last_clean":"2024-03-28T05:21:14.592076+0000","last_became_active":"2024-03-28T05:21:08.047056+0000","last_became_peered":"2024-03-28T05:21:08.047056+0000","last_unstale":"2024-03-28T05:21:14.592076+0000","last_undegraded":"2024-03-28T05:21:14.592076+0000","last_fullsized":"2024-03-28T05:21:14.592076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:59:54.341517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.828462+0000","last_change":"2024-03-28T05:21:08.043253+0000","last_active":"2024-03-28T05:21:13.828462+0000","last_peered":"2024-03-28T05:21:13.828462+0000","last_clean":"2024-03-28T05:21:13.828462+0000","last_became_active":"2024-03-28T05:21:08.042934+0000","last_became_peered":"2024-03-28T05:21:08.042934+0000","last_unstale":"2024-03-28T05:21:13.828462+0000","last_undegraded":"2024-03-28T05:21:13.828462+0000","last_fullsized":"2024-03-28T05:21:13.828462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:55:41.612580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876946+0000","last_change":"2024-03-28T05:21:08.045349+0000","last_active":"2024-03-28T05:21:13.876946+0000","last_peered":"2024-03-28T05:21:13.876946+0000","last_clean":"2024-03-28T05:21:13.876946+0000","last_became_active":"2024-03-28T05:21:08.044477+0000","last_became_peered":"2024-03-28T05:21:08.044477+0000","last_unstale":"2024-03-28T05:21:13.876946+0000","last_undegraded":"2024-03-28T05:21:13.876946+0000","last_fullsized":"2024-03-28T05:21:13.876946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:38:34.732765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272152+0000","last_change":"2024-03-28T05:21:08.044801+0000","last_active":"2024-03-28T05:21:13.272152+0000","last_peered":"2024-03-28T05:21:13.272152+0000","last_clean":"2024-03-28T05:21:13.272152+0000","last_became_active":"2024-03-28T05:21:08.042208+0000","last_became_peered":"2024-03-28T05:21:08.042208+0000","last_unstale":"2024-03-28T05:21:13.272152+0000","last_undegraded":"2024-03-28T05:21:13.272152+0000","last_fullsized":"2024-03-28T05:21:13.272152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:49:15.663953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876689+0000","last_change":"2024-03-28T05:21:08.047843+0000","last_active":"2024-03-28T05:21:13.876689+0000","last_peered":"2024-03-28T05:21:13.876689+0000","last_clean":"2024-03-28T05:21:13.876689+0000","last_became_active":"2024-03-28T05:21:08.047466+0000","last_became_peered":"2024-03-28T05:21:08.047466+0000","last_unstale":"2024-03-28T05:21:13.876689+0000","last_undegraded":"2024-03-28T05:21:13.876689+0000","last_fullsized":"2024-03-28T05:21:13.876689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:25:03.424233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759293+0000","last_change":"2024-03-28T05:21:08.040786+0000","last_active":"2024-03-28T05:21:13.759293+0000","last_peered":"2024-03-28T05:21:13.759293+0000","last_clean":"2024-03-28T05:21:13.759293+0000","last_became_active":"2024-03-28T05:21:08.040495+0000","last_became_peered":"2024-03-28T05:21:08.040495+0000","last_unstale":"2024-03-28T05:21:13.759293+0000","last_undegraded":"2024-03-28T05:21:13.759293+0000","last_fullsized":"2024-03-28T05:21:13.759293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:06:31.626221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876276+0000","last_change":"2024-03-28T05:21:08.050245+0000","last_active":"2024-03-28T05:21:13.876276+0000","last_peered":"2024-03-28T05:21:13.876276+0000","last_clean":"2024-03-28T05:21:13.876276+0000","last_became_active":"2024-03-28T05:21:08.049932+0000","last_became_peered":"2024-03-28T05:21:08.049932+0000","last_unstale":"2024-03-28T05:21:13.876276+0000","last_undegraded":"2024-03-28T05:21:13.876276+0000","last_fullsized":"2024-03-28T05:21:13.876276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:59:03.545226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877663+0000","last_change":"2024-03-28T05:21:08.049262+0000","last_active":"2024-03-28T05:21:13.877663+0000","last_peered":"2024-03-28T05:21:13.877663+0000","last_clean":"2024-03-28T05:21:13.877663+0000","last_became_active":"2024-03-28T05:21:08.048578+0000","last_became_peered":"2024-03-28T05:21:08.048578+0000","last_unstale":"2024-03-28T05:21:13.877663+0000","last_undegraded":"2024-03-28T05:21:13.877663+0000","last_fullsized":"2024-03-28T05:21:13.877663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:26:38.614430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.828435+0000","last_change":"2024-03-28T05:21:08.046612+0000","last_active":"2024-03-28T05:21:13.828435+0000","last_peered":"2024-03-28T05:21:13.828435+0000","last_clean":"2024-03-28T05:21:13.828435+0000","last_became_active":"2024-03-28T05:21:08.046312+0000","last_became_peered":"2024-03-28T05:21:08.046312+0000","last_unstale":"2024-03-28T05:21:13.828435+0000","last_undegraded":"2024-03-28T05:21:13.828435+0000","last_fullsized":"2024-03-28T05:21:13.828435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:20:08.379620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876725+0000","last_change":"2024-03-28T05:21:08.045205+0000","last_active":"2024-03-28T05:21:13.876725+0000","last_peered":"2024-03-28T05:21:13.876725+0000","last_clean":"2024-03-28T05:21:13.876725+0000","last_became_active":"2024-03-28T05:21:08.044901+0000","last_became_peered":"2024-03-28T05:21:08.044901+0000","last_unstale":"2024-03-28T05:21:13.876725+0000","last_undegraded":"2024-03-28T05:21:13.876725+0000","last_fullsized":"2024-03-28T05:21:13.876725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:31:11.045766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":23939,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":32,"num_write":56,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":52518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383696,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645334,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1622016,"data_stored":1090918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":481893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070496,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":991232,"data_stored":477819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332133,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28572,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716580,"statfs":{"total":95995035648,"available":95965777920,"internally_reserved":0,"allocated":1581056,"data_stored":1068165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495658,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":487657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":987136,"data_stored":477305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":481091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":477319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920821,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":477237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28576,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716576,"statfs":{"total":95995035648,"available":95965773824,"internally_reserved":0,"allocated":1589248,"data_stored":1067765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":23141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":23335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T05:21:36.164 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T05:21:36.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.305+0000 7f5e54f29640 1 -- 172.21.15.23:0/3017378500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e501016c0 msgr2=0x7f5e50101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:36.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.305+0000 7f5e54f29640 1 --2- 172.21.15.23:0/3017378500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e501016c0 0x7f5e50101ae0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5e40009900 tx=0x7f5e4002bc30 comp rx=0 tx=0).stop 2024-03-28T05:21:36.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.305+0000 7f5e54f29640 1 -- 172.21.15.23:0/3017378500 shutdown_connections 2024-03-28T05:21:36.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.305+0000 7f5e54f29640 1 --2- 172.21.15.23:0/3017378500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e50103270 0x7f5e50109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:36.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.305+0000 7f5e54f29640 1 --2- 172.21.15.23:0/3017378500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e501028b0 0x7f5e50102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:36.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.305+0000 7f5e54f29640 1 --2- 172.21.15.23:0/3017378500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e501016c0 0x7f5e50101ae0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:36.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.305+0000 7f5e54f29640 1 -- 172.21.15.23:0/3017378500 >> 172.21.15.23:0/3017378500 conn(0x7f5e500fcc50 msgr2=0x7f5e500ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:36.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.305+0000 7f5e54f29640 1 -- 172.21.15.23:0/3017378500 shutdown_connections 2024-03-28T05:21:36.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.305+0000 7f5e54f29640 1 -- 172.21.15.23:0/3017378500 wait complete. 2024-03-28T05:21:36.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.306+0000 7f5e54f29640 1 Processor -- start 2024-03-28T05:21:36.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.306+0000 7f5e54f29640 1 -- start start 2024-03-28T05:21:36.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.306+0000 7f5e54f29640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e501016c0 0x7f5e501979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:36.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.306+0000 7f5e54f29640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e501028b0 0x7f5e50197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:36.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.306+0000 7f5e54f29640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e50103270 0x7f5e5019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:36.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.306+0000 7f5e54f29640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5010d340 con 0x7f5e50103270 2024-03-28T05:21:36.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.306+0000 7f5e54f29640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5010d1c0 con 0x7f5e501016c0 2024-03-28T05:21:36.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.306+0000 7f5e54f29640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5010d4c0 con 0x7f5e501028b0 2024-03-28T05:21:36.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.306+0000 7f5e4ed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e50103270 0x7f5e5019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:36.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.306+0000 7f5e4ed9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e50103270 0x7f5e5019ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53360/0 (socket says 172.21.15.23:53360) 2024-03-28T05:21:36.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.306+0000 7f5e4ed9d640 1 -- 172.21.15.23:0/880575889 learned_addr learned my addr 172.21.15.23:0/880575889 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:36.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.306+0000 7f5e47fff640 1 --2- 172.21.15.23:0/880575889 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e501028b0 0x7f5e50197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:36.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.306+0000 7f5e4e59c640 1 --2- 172.21.15.23:0/880575889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e501016c0 0x7f5e501979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:36.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.306+0000 7f5e4ed9d640 1 -- 172.21.15.23:0/880575889 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e501028b0 msgr2=0x7f5e50197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:36.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.306+0000 7f5e4ed9d640 1 --2- 172.21.15.23:0/880575889 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e501028b0 0x7f5e50197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:36.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.306+0000 7f5e4ed9d640 1 -- 172.21.15.23:0/880575889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e501016c0 msgr2=0x7f5e501979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:36.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.306+0000 7f5e4ed9d640 1 --2- 172.21.15.23:0/880575889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e501016c0 0x7f5e501979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:36.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.306+0000 7f5e4ed9d640 1 -- 172.21.15.23:0/880575889 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5019f5d0 con 0x7f5e50103270 2024-03-28T05:21:36.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.307+0000 7f5e4ed9d640 1 --2- 172.21.15.23:0/880575889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e50103270 0x7f5e5019ef60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800eb10 tx=0x7f5e4800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:36.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.307+0000 7f5e477fe640 1 -- 172.21.15.23:0/880575889 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e48004b20 con 0x7f5e50103270 2024-03-28T05:21:36.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.307+0000 7f5e477fe640 1 -- 172.21.15.23:0/880575889 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e48004cc0 con 0x7f5e50103270 2024-03-28T05:21:36.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.307+0000 7f5e54f29640 1 -- 172.21.15.23:0/880575889 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5019f8c0 con 0x7f5e50103270 2024-03-28T05:21:36.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.307+0000 7f5e477fe640 1 -- 172.21.15.23:0/880575889 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e48010760 con 0x7f5e50103270 2024-03-28T05:21:36.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.307+0000 7f5e54f29640 1 -- 172.21.15.23:0/880575889 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5019fe10 con 0x7f5e50103270 2024-03-28T05:21:36.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.308+0000 7f5e54f29640 1 -- 172.21.15.23:0/880575889 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e501089e0 con 0x7f5e50103270 2024-03-28T05:21:36.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.309+0000 7f5e477fe640 1 -- 172.21.15.23:0/880575889 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f5e48005020 con 0x7f5e50103270 2024-03-28T05:21:36.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.310+0000 7f5e477fe640 1 --2- 172.21.15.23:0/880575889 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5e3c0a2890 0x7f5e3c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:36.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.310+0000 7f5e4e59c640 1 --2- 172.21.15.23:0/880575889 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5e3c0a2890 0x7f5e3c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:36.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.310+0000 7f5e477fe640 1 -- 172.21.15.23:0/880575889 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f5e480cf5c0 con 0x7f5e50103270 2024-03-28T05:21:36.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.310+0000 7f5e4e59c640 1 --2- 172.21.15.23:0/880575889 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5e3c0a2890 0x7f5e3c0a4d50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5e4002c7b0 tx=0x7f5e40005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:36.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.314+0000 7f5e477fe640 1 -- 172.21.15.23:0/880575889 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5e4809c2c0 con 0x7f5e50103270 2024-03-28T05:21:36.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.495+0000 7f5e54f29640 1 -- 172.21.15.23:0/880575889 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5e50061ae0 con 0x7f5e3c0a2890 2024-03-28T05:21:36.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.505+0000 7f5e477fe640 1 -- 172.21.15.23:0/880575889 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340209 (secure 0 0 0) 0x7f5e50061ae0 con 0x7f5e3c0a2890 2024-03-28T05:21:36.509 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:21:36.511 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-28T05:21:36.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.508+0000 7f5e54f29640 1 -- 172.21.15.23:0/880575889 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5e3c0a2890 msgr2=0x7f5e3c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:36.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.508+0000 7f5e54f29640 1 --2- 172.21.15.23:0/880575889 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5e3c0a2890 0x7f5e3c0a4d50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5e4002c7b0 tx=0x7f5e40005fd0 comp rx=0 tx=0).stop 2024-03-28T05:21:36.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.508+0000 7f5e54f29640 1 -- 172.21.15.23:0/880575889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e50103270 msgr2=0x7f5e5019ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:36.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.508+0000 7f5e54f29640 1 --2- 172.21.15.23:0/880575889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e50103270 0x7f5e5019ef60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800eb10 tx=0x7f5e4800c2d0 comp rx=0 tx=0).stop 2024-03-28T05:21:36.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.508+0000 7f5e54f29640 1 -- 172.21.15.23:0/880575889 shutdown_connections 2024-03-28T05:21:36.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.508+0000 7f5e54f29640 1 --2- 172.21.15.23:0/880575889 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5e3c0a2890 0x7f5e3c0a4d50 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:36.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.508+0000 7f5e54f29640 1 --2- 172.21.15.23:0/880575889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e50103270 0x7f5e5019ef60 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:36.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.508+0000 7f5e54f29640 1 --2- 172.21.15.23:0/880575889 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e501028b0 0x7f5e50197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:36.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.508+0000 7f5e54f29640 1 --2- 172.21.15.23:0/880575889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e501016c0 0x7f5e501979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:36.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.508+0000 7f5e54f29640 1 -- 172.21.15.23:0/880575889 >> 172.21.15.23:0/880575889 conn(0x7f5e500fcc50 msgr2=0x7f5e50105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:36.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.508+0000 7f5e54f29640 1 -- 172.21.15.23:0/880575889 shutdown_connections 2024-03-28T05:21:36.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:36.508+0000 7f5e54f29640 1 -- 172.21.15.23:0/880575889 wait complete. 2024-03-28T05:21:36.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:36 smithi023 ceph-mon[31954]: pgmap v302: 129 pgs: 129 active+clean; 600 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.2 KiB/s wr, 1 op/s 2024-03-28T05:21:36.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:36 smithi023 ceph-mon[31954]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:21:36.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:36 smithi023 ceph-mon[31954]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:21:36.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:36 smithi023 ceph-mon[31954]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-28T05:21:36.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:36 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1551061551' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:21:36.518 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:36 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1704576560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:36.519 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-03-28T05:21:34.726040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":614307,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":150,"num_read_kb":151,"num_write":237,"num_write_kb":2979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":187,"ondisk_log_size":187,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":358428,"kb_used_data":13388,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124583396,"statfs":{"total":1151940427776,"available":1151573397504,"internally_reserved":0,"allocated":13709312,"data_stored":7541599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21080,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":32,"num_write":10,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003506"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876886+0000","last_change":"2024-03-28T05:21:08.043720+0000","last_active":"2024-03-28T05:21:13.876886+0000","last_peered":"2024-03-28T05:21:13.876886+0000","last_clean":"2024-03-28T05:21:13.876886+0000","last_became_active":"2024-03-28T05:21:08.043250+0000","last_became_peered":"2024-03-28T05:21:08.043250+0000","last_unstale":"2024-03-28T05:21:13.876886+0000","last_undegraded":"2024-03-28T05:21:13.876886+0000","last_fullsized":"2024-03-28T05:21:13.876886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:29:10.144086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:14.592555+0000","last_change":"2024-03-28T05:21:08.053247+0000","last_active":"2024-03-28T05:21:14.592555+0000","last_peered":"2024-03-28T05:21:14.592555+0000","last_clean":"2024-03-28T05:21:14.592555+0000","last_became_active":"2024-03-28T05:21:08.052941+0000","last_became_peered":"2024-03-28T05:21:08.052941+0000","last_unstale":"2024-03-28T05:21:14.592555+0000","last_undegraded":"2024-03-28T05:21:14.592555+0000","last_fullsized":"2024-03-28T05:21:14.592555+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:41:22.223858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:14.592448+0000","last_change":"2024-03-28T05:21:08.051938+0000","last_active":"2024-03-28T05:21:14.592448+0000","last_peered":"2024-03-28T05:21:14.592448+0000","last_clean":"2024-03-28T05:21:14.592448+0000","last_became_active":"2024-03-28T05:21:08.050366+0000","last_became_peered":"2024-03-28T05:21:08.050366+0000","last_unstale":"2024-03-28T05:21:14.592448+0000","last_undegraded":"2024-03-28T05:21:14.592448+0000","last_fullsized":"2024-03-28T05:21:14.592448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:27:29.325009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877382+0000","last_change":"2024-03-28T05:21:08.043371+0000","last_active":"2024-03-28T05:21:13.877382+0000","last_peered":"2024-03-28T05:21:13.877382+0000","last_clean":"2024-03-28T05:21:13.877382+0000","last_became_active":"2024-03-28T05:21:08.043038+0000","last_became_peered":"2024-03-28T05:21:08.043038+0000","last_unstale":"2024-03-28T05:21:13.877382+0000","last_undegraded":"2024-03-28T05:21:13.877382+0000","last_fullsized":"2024-03-28T05:21:13.877382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:16:14.137248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876032+0000","last_change":"2024-03-28T05:21:08.048678+0000","last_active":"2024-03-28T05:21:13.876032+0000","last_peered":"2024-03-28T05:21:13.876032+0000","last_clean":"2024-03-28T05:21:13.876032+0000","last_became_active":"2024-03-28T05:21:08.048272+0000","last_became_peered":"2024-03-28T05:21:08.048272+0000","last_unstale":"2024-03-28T05:21:13.876032+0000","last_undegraded":"2024-03-28T05:21:13.876032+0000","last_fullsized":"2024-03-28T05:21:13.876032+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:34:29.029249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877106+0000","last_change":"2024-03-28T05:21:08.041764+0000","last_active":"2024-03-28T05:21:13.877106+0000","last_peered":"2024-03-28T05:21:13.877106+0000","last_clean":"2024-03-28T05:21:13.877106+0000","last_became_active":"2024-03-28T05:21:08.041337+0000","last_became_peered":"2024-03-28T05:21:08.041337+0000","last_unstale":"2024-03-28T05:21:13.877106+0000","last_undegraded":"2024-03-28T05:21:13.877106+0000","last_fullsized":"2024-03-28T05:21:13.877106+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:42:29.136829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.275455+0000","last_change":"2024-03-28T05:21:08.046554+0000","last_active":"2024-03-28T05:21:13.275455+0000","last_peered":"2024-03-28T05:21:13.275455+0000","last_clean":"2024-03-28T05:21:13.275455+0000","last_became_active":"2024-03-28T05:21:08.046245+0000","last_became_peered":"2024-03-28T05:21:08.046245+0000","last_unstale":"2024-03-28T05:21:13.275455+0000","last_undegraded":"2024-03-28T05:21:13.275455+0000","last_fullsized":"2024-03-28T05:21:13.275455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:40:05.549254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.875924+0000","last_change":"2024-03-28T05:21:08.048605+0000","last_active":"2024-03-28T05:21:13.875924+0000","last_peered":"2024-03-28T05:21:13.875924+0000","last_clean":"2024-03-28T05:21:13.875924+0000","last_became_active":"2024-03-28T05:21:08.048279+0000","last_became_peered":"2024-03-28T05:21:08.048279+0000","last_unstale":"2024-03-28T05:21:13.875924+0000","last_undegraded":"2024-03-28T05:21:13.875924+0000","last_fullsized":"2024-03-28T05:21:13.875924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:34:00.935133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.878182+0000","last_change":"2024-03-28T05:21:08.051357+0000","last_active":"2024-03-28T05:21:13.878182+0000","last_peered":"2024-03-28T05:21:13.878182+0000","last_clean":"2024-03-28T05:21:13.878182+0000","last_became_active":"2024-03-28T05:21:08.051012+0000","last_became_peered":"2024-03-28T05:21:08.051012+0000","last_unstale":"2024-03-28T05:21:13.878182+0000","last_undegraded":"2024-03-28T05:21:13.878182+0000","last_fullsized":"2024-03-28T05:21:13.878182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:43:42.090619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877368+0000","last_change":"2024-03-28T05:21:08.038727+0000","last_active":"2024-03-28T05:21:13.877368+0000","last_peered":"2024-03-28T05:21:13.877368+0000","last_clean":"2024-03-28T05:21:13.877368+0000","last_became_active":"2024-03-28T05:21:08.038425+0000","last_became_peered":"2024-03-28T05:21:08.038425+0000","last_unstale":"2024-03-28T05:21:13.877368+0000","last_undegraded":"2024-03-28T05:21:13.877368+0000","last_fullsized":"2024-03-28T05:21:13.877368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:06:53.786136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.276082+0000","last_change":"2024-03-28T05:21:08.054698+0000","last_active":"2024-03-28T05:21:13.276082+0000","last_peered":"2024-03-28T05:21:13.276082+0000","last_clean":"2024-03-28T05:21:13.276082+0000","last_became_active":"2024-03-28T05:21:08.054310+0000","last_became_peered":"2024-03-28T05:21:08.054310+0000","last_unstale":"2024-03-28T05:21:13.276082+0000","last_undegraded":"2024-03-28T05:21:13.276082+0000","last_fullsized":"2024-03-28T05:21:13.276082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:09:47.996381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877139+0000","last_change":"2024-03-28T05:21:08.042826+0000","last_active":"2024-03-28T05:21:13.877139+0000","last_peered":"2024-03-28T05:21:13.877139+0000","last_clean":"2024-03-28T05:21:13.877139+0000","last_became_active":"2024-03-28T05:21:08.042487+0000","last_became_peered":"2024-03-28T05:21:08.042487+0000","last_unstale":"2024-03-28T05:21:13.877139+0000","last_undegraded":"2024-03-28T05:21:13.877139+0000","last_fullsized":"2024-03-28T05:21:13.877139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:47:06.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.828392+0000","last_change":"2024-03-28T05:21:08.030791+0000","last_active":"2024-03-28T05:21:13.828392+0000","last_peered":"2024-03-28T05:21:13.828392+0000","last_clean":"2024-03-28T05:21:13.828392+0000","last_became_active":"2024-03-28T05:21:08.030447+0000","last_became_peered":"2024-03-28T05:21:08.030447+0000","last_unstale":"2024-03-28T05:21:13.828392+0000","last_undegraded":"2024-03-28T05:21:13.828392+0000","last_fullsized":"2024-03-28T05:21:13.828392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:54:53.454037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:14.591851+0000","last_change":"2024-03-28T05:21:08.042477+0000","last_active":"2024-03-28T05:21:14.591851+0000","last_peered":"2024-03-28T05:21:14.591851+0000","last_clean":"2024-03-28T05:21:14.591851+0000","last_became_active":"2024-03-28T05:21:08.042164+0000","last_became_peered":"2024-03-28T05:21:08.042164+0000","last_unstale":"2024-03-28T05:21:14.591851+0000","last_undegraded":"2024-03-28T05:21:14.591851+0000","last_fullsized":"2024-03-28T05:21:14.591851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:44:25.566195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:14.591961+0000","last_change":"2024-03-28T05:21:08.050635+0000","last_active":"2024-03-28T05:21:14.591961+0000","last_peered":"2024-03-28T05:21:14.591961+0000","last_clean":"2024-03-28T05:21:14.591961+0000","last_became_active":"2024-03-28T05:21:08.050217+0000","last_became_peered":"2024-03-28T05:21:08.050217+0000","last_unstale":"2024-03-28T05:21:14.591961+0000","last_undegraded":"2024-03-28T05:21:14.591961+0000","last_fullsized":"2024-03-28T05:21:14.591961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:13:49.252755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876231+0000","last_change":"2024-03-28T05:21:08.048021+0000","last_active":"2024-03-28T05:21:13.876231+0000","last_peered":"2024-03-28T05:21:13.876231+0000","last_clean":"2024-03-28T05:21:13.876231+0000","last_became_active":"2024-03-28T05:21:08.047381+0000","last_became_peered":"2024-03-28T05:21:08.047381+0000","last_unstale":"2024-03-28T05:21:13.876231+0000","last_undegraded":"2024-03-28T05:21:13.876231+0000","last_fullsized":"2024-03-28T05:21:13.876231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:42:39.718804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876786+0000","last_change":"2024-03-28T05:21:08.044774+0000","last_active":"2024-03-28T05:21:13.876786+0000","last_peered":"2024-03-28T05:21:13.876786+0000","last_clean":"2024-03-28T05:21:13.876786+0000","last_became_active":"2024-03-28T05:21:08.044470+0000","last_became_peered":"2024-03-28T05:21:08.044470+0000","last_unstale":"2024-03-28T05:21:13.876786+0000","last_undegraded":"2024-03-28T05:21:13.876786+0000","last_fullsized":"2024-03-28T05:21:13.876786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:26:11.354288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272570+0000","last_change":"2024-03-28T05:21:08.042871+0000","last_active":"2024-03-28T05:21:13.272570+0000","last_peered":"2024-03-28T05:21:13.272570+0000","last_clean":"2024-03-28T05:21:13.272570+0000","last_became_active":"2024-03-28T05:21:08.042506+0000","last_became_peered":"2024-03-28T05:21:08.042506+0000","last_unstale":"2024-03-28T05:21:13.272570+0000","last_undegraded":"2024-03-28T05:21:13.272570+0000","last_fullsized":"2024-03-28T05:21:13.272570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:34:50.424380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272250+0000","last_change":"2024-03-28T05:21:08.045056+0000","last_active":"2024-03-28T05:21:13.272250+0000","last_peered":"2024-03-28T05:21:13.272250+0000","last_clean":"2024-03-28T05:21:13.272250+0000","last_became_active":"2024-03-28T05:21:08.044719+0000","last_became_peered":"2024-03-28T05:21:08.044719+0000","last_unstale":"2024-03-28T05:21:13.272250+0000","last_undegraded":"2024-03-28T05:21:13.272250+0000","last_fullsized":"2024-03-28T05:21:13.272250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:23:12.479829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272213+0000","last_change":"2024-03-28T05:21:12.295239+0000","last_active":"2024-03-28T05:21:13.272213+0000","last_peered":"2024-03-28T05:21:13.272213+0000","last_clean":"2024-03-28T05:21:13.272213+0000","last_became_active":"2024-03-28T05:21:12.294843+0000","last_became_peered":"2024-03-28T05:21:12.294843+0000","last_unstale":"2024-03-28T05:21:13.272213+0000","last_undegraded":"2024-03-28T05:21:13.272213+0000","last_fullsized":"2024-03-28T05:21:13.272213+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:36:35.291983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272616+0000","last_change":"2024-03-28T05:21:07.270833+0000","last_active":"2024-03-28T05:21:13.272616+0000","last_peered":"2024-03-28T05:21:13.272616+0000","last_clean":"2024-03-28T05:21:13.272616+0000","last_became_active":"2024-03-28T05:21:07.270288+0000","last_became_peered":"2024-03-28T05:21:07.270288+0000","last_unstale":"2024-03-28T05:21:13.272616+0000","last_undegraded":"2024-03-28T05:21:13.272616+0000","last_fullsized":"2024-03-28T05:21:13.272616+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:41:05.494851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876309+0000","last_change":"2024-03-28T05:21:08.048904+0000","last_active":"2024-03-28T05:21:13.876309+0000","last_peered":"2024-03-28T05:21:13.876309+0000","last_clean":"2024-03-28T05:21:13.876309+0000","last_became_active":"2024-03-28T05:21:08.048614+0000","last_became_peered":"2024-03-28T05:21:08.048614+0000","last_unstale":"2024-03-28T05:21:13.876309+0000","last_undegraded":"2024-03-28T05:21:13.876309+0000","last_fullsized":"2024-03-28T05:21:13.876309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:29:30.394022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876559+0000","last_change":"2024-03-28T05:21:12.295599+0000","last_active":"2024-03-28T05:21:13.876559+0000","last_peered":"2024-03-28T05:21:13.876559+0000","last_clean":"2024-03-28T05:21:13.876559+0000","last_became_active":"2024-03-28T05:21:12.295161+0000","last_became_peered":"2024-03-28T05:21:12.295161+0000","last_unstale":"2024-03-28T05:21:13.876559+0000","last_undegraded":"2024-03-28T05:21:13.876559+0000","last_fullsized":"2024-03-28T05:21:13.876559+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:28:39.097018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876657+0000","last_change":"2024-03-28T05:21:07.264532+0000","last_active":"2024-03-28T05:21:13.876657+0000","last_peered":"2024-03-28T05:21:13.876657+0000","last_clean":"2024-03-28T05:21:13.876657+0000","last_became_active":"2024-03-28T05:21:07.264121+0000","last_became_peered":"2024-03-28T05:21:07.264121+0000","last_unstale":"2024-03-28T05:21:13.876657+0000","last_undegraded":"2024-03-28T05:21:13.876657+0000","last_fullsized":"2024-03-28T05:21:13.876657+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:40:01.927165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:14.592571+0000","last_change":"2024-03-28T05:21:08.045547+0000","last_active":"2024-03-28T05:21:14.592571+0000","last_peered":"2024-03-28T05:21:14.592571+0000","last_clean":"2024-03-28T05:21:14.592571+0000","last_became_active":"2024-03-28T05:21:08.045243+0000","last_became_peered":"2024-03-28T05:21:08.045243+0000","last_unstale":"2024-03-28T05:21:14.592571+0000","last_undegraded":"2024-03-28T05:21:14.592571+0000","last_fullsized":"2024-03-28T05:21:14.592571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:24:51.178938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877111+0000","last_change":"2024-03-28T05:21:12.301461+0000","last_active":"2024-03-28T05:21:13.877111+0000","last_peered":"2024-03-28T05:21:13.877111+0000","last_clean":"2024-03-28T05:21:13.877111+0000","last_became_active":"2024-03-28T05:21:12.300765+0000","last_became_peered":"2024-03-28T05:21:12.300765+0000","last_unstale":"2024-03-28T05:21:13.877111+0000","last_undegraded":"2024-03-28T05:21:13.877111+0000","last_fullsized":"2024-03-28T05:21:13.877111+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:10:38.609978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876505+0000","last_change":"2024-03-28T05:21:07.269209+0000","last_active":"2024-03-28T05:21:13.876505+0000","last_peered":"2024-03-28T05:21:13.876505+0000","last_clean":"2024-03-28T05:21:13.876505+0000","last_became_active":"2024-03-28T05:21:07.268068+0000","last_became_peered":"2024-03-28T05:21:07.268068+0000","last_unstale":"2024-03-28T05:21:13.876505+0000","last_undegraded":"2024-03-28T05:21:13.876505+0000","last_fullsized":"2024-03-28T05:21:13.876505+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:44:53.135439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877831+0000","last_change":"2024-03-28T05:21:08.045233+0000","last_active":"2024-03-28T05:21:13.877831+0000","last_peered":"2024-03-28T05:21:13.877831+0000","last_clean":"2024-03-28T05:21:13.877831+0000","last_became_active":"2024-03-28T05:21:08.044878+0000","last_became_peered":"2024-03-28T05:21:08.044878+0000","last_unstale":"2024-03-28T05:21:13.877831+0000","last_undegraded":"2024-03-28T05:21:13.877831+0000","last_fullsized":"2024-03-28T05:21:13.877831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:15:50.830519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876493+0000","last_change":"2024-03-28T05:21:12.296326+0000","last_active":"2024-03-28T05:21:13.876493+0000","last_peered":"2024-03-28T05:21:13.876493+0000","last_clean":"2024-03-28T05:21:13.876493+0000","last_became_active":"2024-03-28T05:21:12.295863+0000","last_became_peered":"2024-03-28T05:21:12.295863+0000","last_unstale":"2024-03-28T05:21:13.876493+0000","last_undegraded":"2024-03-28T05:21:13.876493+0000","last_fullsized":"2024-03-28T05:21:13.876493+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:26:49.340900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876164+0000","last_change":"2024-03-28T05:21:07.263689+0000","last_active":"2024-03-28T05:21:13.876164+0000","last_peered":"2024-03-28T05:21:13.876164+0000","last_clean":"2024-03-28T05:21:13.876164+0000","last_became_active":"2024-03-28T05:21:07.263410+0000","last_became_peered":"2024-03-28T05:21:07.263410+0000","last_unstale":"2024-03-28T05:21:13.876164+0000","last_undegraded":"2024-03-28T05:21:13.876164+0000","last_fullsized":"2024-03-28T05:21:13.876164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:51:03.017620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759802+0000","last_change":"2024-03-28T05:21:08.046993+0000","last_active":"2024-03-28T05:21:13.759802+0000","last_peered":"2024-03-28T05:21:13.759802+0000","last_clean":"2024-03-28T05:21:13.759802+0000","last_became_active":"2024-03-28T05:21:08.045802+0000","last_became_peered":"2024-03-28T05:21:08.045802+0000","last_unstale":"2024-03-28T05:21:13.759802+0000","last_undegraded":"2024-03-28T05:21:13.759802+0000","last_fullsized":"2024-03-28T05:21:13.759802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:18:31.024546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272106+0000","last_change":"2024-03-28T05:21:12.284248+0000","last_active":"2024-03-28T05:21:13.272106+0000","last_peered":"2024-03-28T05:21:13.272106+0000","last_clean":"2024-03-28T05:21:13.272106+0000","last_became_active":"2024-03-28T05:21:12.283887+0000","last_became_peered":"2024-03-28T05:21:12.283887+0000","last_unstale":"2024-03-28T05:21:13.272106+0000","last_undegraded":"2024-03-28T05:21:13.272106+0000","last_fullsized":"2024-03-28T05:21:13.272106+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:50:50.871119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759416+0000","last_change":"2024-03-28T05:21:07.266436+0000","last_active":"2024-03-28T05:21:13.759416+0000","last_peered":"2024-03-28T05:21:13.759416+0000","last_clean":"2024-03-28T05:21:13.759416+0000","last_became_active":"2024-03-28T05:21:07.266115+0000","last_became_peered":"2024-03-28T05:21:07.266115+0000","last_unstale":"2024-03-28T05:21:13.759416+0000","last_undegraded":"2024-03-28T05:21:13.759416+0000","last_fullsized":"2024-03-28T05:21:13.759416+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:42:06.865456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876045+0000","last_change":"2024-03-28T05:21:08.047500+0000","last_active":"2024-03-28T05:21:13.876045+0000","last_peered":"2024-03-28T05:21:13.876045+0000","last_clean":"2024-03-28T05:21:13.876045+0000","last_became_active":"2024-03-28T05:21:08.047210+0000","last_became_peered":"2024-03-28T05:21:08.047210+0000","last_unstale":"2024-03-28T05:21:13.876045+0000","last_undegraded":"2024-03-28T05:21:13.876045+0000","last_fullsized":"2024-03-28T05:21:13.876045+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:31:02.393561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272013+0000","last_change":"2024-03-28T05:21:12.288979+0000","last_active":"2024-03-28T05:21:13.272013+0000","last_peered":"2024-03-28T05:21:13.272013+0000","last_clean":"2024-03-28T05:21:13.272013+0000","last_became_active":"2024-03-28T05:21:12.288621+0000","last_became_peered":"2024-03-28T05:21:12.288621+0000","last_unstale":"2024-03-28T05:21:13.272013+0000","last_undegraded":"2024-03-28T05:21:13.272013+0000","last_fullsized":"2024-03-28T05:21:13.272013+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:45:31.205059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876958+0000","last_change":"2024-03-28T05:21:07.265869+0000","last_active":"2024-03-28T05:21:13.876958+0000","last_peered":"2024-03-28T05:21:13.876958+0000","last_clean":"2024-03-28T05:21:13.876958+0000","last_became_active":"2024-03-28T05:21:07.265586+0000","last_became_peered":"2024-03-28T05:21:07.265586+0000","last_unstale":"2024-03-28T05:21:13.876958+0000","last_undegraded":"2024-03-28T05:21:13.876958+0000","last_fullsized":"2024-03-28T05:21:13.876958+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:17:39.328864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759509+0000","last_change":"2024-03-28T05:21:08.046780+0000","last_active":"2024-03-28T05:21:13.759509+0000","last_peered":"2024-03-28T05:21:13.759509+0000","last_clean":"2024-03-28T05:21:13.759509+0000","last_became_active":"2024-03-28T05:21:08.046484+0000","last_became_peered":"2024-03-28T05:21:08.046484+0000","last_unstale":"2024-03-28T05:21:13.759509+0000","last_undegraded":"2024-03-28T05:21:13.759509+0000","last_fullsized":"2024-03-28T05:21:13.759509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:02:08.388581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876880+0000","last_change":"2024-03-28T05:21:12.288723+0000","last_active":"2024-03-28T05:21:13.876880+0000","last_peered":"2024-03-28T05:21:13.876880+0000","last_clean":"2024-03-28T05:21:13.876880+0000","last_became_active":"2024-03-28T05:21:12.288310+0000","last_became_peered":"2024-03-28T05:21:12.288310+0000","last_unstale":"2024-03-28T05:21:13.876880+0000","last_undegraded":"2024-03-28T05:21:13.876880+0000","last_fullsized":"2024-03-28T05:21:13.876880+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:15:39.953732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.828127+0000","last_change":"2024-03-28T05:21:07.257433+0000","last_active":"2024-03-28T05:21:13.828127+0000","last_peered":"2024-03-28T05:21:13.828127+0000","last_clean":"2024-03-28T05:21:13.828127+0000","last_became_active":"2024-03-28T05:21:07.257099+0000","last_became_peered":"2024-03-28T05:21:07.257099+0000","last_unstale":"2024-03-28T05:21:13.828127+0000","last_undegraded":"2024-03-28T05:21:13.828127+0000","last_fullsized":"2024-03-28T05:21:13.828127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:59:24.580079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876106+0000","last_change":"2024-03-28T05:21:08.047963+0000","last_active":"2024-03-28T05:21:13.876106+0000","last_peered":"2024-03-28T05:21:13.876106+0000","last_clean":"2024-03-28T05:21:13.876106+0000","last_became_active":"2024-03-28T05:21:08.046403+0000","last_became_peered":"2024-03-28T05:21:08.046403+0000","last_unstale":"2024-03-28T05:21:13.876106+0000","last_undegraded":"2024-03-28T05:21:13.876106+0000","last_fullsized":"2024-03-28T05:21:13.876106+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:58:45.280549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:14.592578+0000","last_change":"2024-03-28T05:21:12.284618+0000","last_active":"2024-03-28T05:21:14.592578+0000","last_peered":"2024-03-28T05:21:14.592578+0000","last_clean":"2024-03-28T05:21:14.592578+0000","last_became_active":"2024-03-28T05:21:12.284185+0000","last_became_peered":"2024-03-28T05:21:12.284185+0000","last_unstale":"2024-03-28T05:21:14.592578+0000","last_undegraded":"2024-03-28T05:21:14.592578+0000","last_fullsized":"2024-03-28T05:21:14.592578+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:15:48.715607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759157+0000","last_change":"2024-03-28T05:21:07.265493+0000","last_active":"2024-03-28T05:21:13.759157+0000","last_peered":"2024-03-28T05:21:13.759157+0000","last_clean":"2024-03-28T05:21:13.759157+0000","last_became_active":"2024-03-28T05:21:07.265182+0000","last_became_peered":"2024-03-28T05:21:07.265182+0000","last_unstale":"2024-03-28T05:21:13.759157+0000","last_undegraded":"2024-03-28T05:21:13.759157+0000","last_fullsized":"2024-03-28T05:21:13.759157+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:55:00.151367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.271983+0000","last_change":"2024-03-28T05:21:08.041161+0000","last_active":"2024-03-28T05:21:13.271983+0000","last_peered":"2024-03-28T05:21:13.271983+0000","last_clean":"2024-03-28T05:21:13.271983+0000","last_became_active":"2024-03-28T05:21:08.040797+0000","last_became_peered":"2024-03-28T05:21:08.040797+0000","last_unstale":"2024-03-28T05:21:13.271983+0000","last_undegraded":"2024-03-28T05:21:13.271983+0000","last_fullsized":"2024-03-28T05:21:13.271983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:05:03.387916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876164+0000","last_change":"2024-03-28T05:21:12.293759+0000","last_active":"2024-03-28T05:21:13.876164+0000","last_peered":"2024-03-28T05:21:13.876164+0000","last_clean":"2024-03-28T05:21:13.876164+0000","last_became_active":"2024-03-28T05:21:12.293355+0000","last_became_peered":"2024-03-28T05:21:12.293355+0000","last_unstale":"2024-03-28T05:21:13.876164+0000","last_undegraded":"2024-03-28T05:21:13.876164+0000","last_fullsized":"2024-03-28T05:21:13.876164+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:29:52.424113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876757+0000","last_change":"2024-03-28T05:21:07.267179+0000","last_active":"2024-03-28T05:21:13.876757+0000","last_peered":"2024-03-28T05:21:13.876757+0000","last_clean":"2024-03-28T05:21:13.876757+0000","last_became_active":"2024-03-28T05:21:07.266837+0000","last_became_peered":"2024-03-28T05:21:07.266837+0000","last_unstale":"2024-03-28T05:21:13.876757+0000","last_undegraded":"2024-03-28T05:21:13.876757+0000","last_fullsized":"2024-03-28T05:21:13.876757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:58:17.424542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.878024+0000","last_change":"2024-03-28T05:21:08.045789+0000","last_active":"2024-03-28T05:21:13.878024+0000","last_peered":"2024-03-28T05:21:13.878024+0000","last_clean":"2024-03-28T05:21:13.878024+0000","last_became_active":"2024-03-28T05:21:08.045441+0000","last_became_peered":"2024-03-28T05:21:08.045441+0000","last_unstale":"2024-03-28T05:21:13.878024+0000","last_undegraded":"2024-03-28T05:21:13.878024+0000","last_fullsized":"2024-03-28T05:21:13.878024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:15:49.968174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272142+0000","last_change":"2024-03-28T05:21:12.290300+0000","last_active":"2024-03-28T05:21:13.272142+0000","last_peered":"2024-03-28T05:21:13.272142+0000","last_clean":"2024-03-28T05:21:13.272142+0000","last_became_active":"2024-03-28T05:21:12.289817+0000","last_became_peered":"2024-03-28T05:21:12.289817+0000","last_unstale":"2024-03-28T05:21:13.272142+0000","last_undegraded":"2024-03-28T05:21:13.272142+0000","last_fullsized":"2024-03-28T05:21:13.272142+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:53:50.705017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.276045+0000","last_change":"2024-03-28T05:21:07.261302+0000","last_active":"2024-03-28T05:21:13.276045+0000","last_peered":"2024-03-28T05:21:13.276045+0000","last_clean":"2024-03-28T05:21:13.276045+0000","last_became_active":"2024-03-28T05:21:07.260991+0000","last_became_peered":"2024-03-28T05:21:07.260991+0000","last_unstale":"2024-03-28T05:21:13.276045+0000","last_undegraded":"2024-03-28T05:21:13.276045+0000","last_fullsized":"2024-03-28T05:21:13.276045+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:57:58.411737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:14.592500+0000","last_change":"2024-03-28T05:21:08.039043+0000","last_active":"2024-03-28T05:21:14.592500+0000","last_peered":"2024-03-28T05:21:14.592500+0000","last_clean":"2024-03-28T05:21:14.592500+0000","last_became_active":"2024-03-28T05:21:08.038734+0000","last_became_peered":"2024-03-28T05:21:08.038734+0000","last_unstale":"2024-03-28T05:21:14.592500+0000","last_undegraded":"2024-03-28T05:21:14.592500+0000","last_fullsized":"2024-03-28T05:21:14.592500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:26:40.155570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877211+0000","last_change":"2024-03-28T05:21:12.290037+0000","last_active":"2024-03-28T05:21:13.877211+0000","last_peered":"2024-03-28T05:21:13.877211+0000","last_clean":"2024-03-28T05:21:13.877211+0000","last_became_active":"2024-03-28T05:21:12.289526+0000","last_became_peered":"2024-03-28T05:21:12.289526+0000","last_unstale":"2024-03-28T05:21:13.877211+0000","last_undegraded":"2024-03-28T05:21:13.877211+0000","last_fullsized":"2024-03-28T05:21:13.877211+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:06:03.705716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759694+0000","last_change":"2024-03-28T05:21:07.263945+0000","last_active":"2024-03-28T05:21:13.759694+0000","last_peered":"2024-03-28T05:21:13.759694+0000","last_clean":"2024-03-28T05:21:13.759694+0000","last_became_active":"2024-03-28T05:21:07.263648+0000","last_became_peered":"2024-03-28T05:21:07.263648+0000","last_unstale":"2024-03-28T05:21:13.759694+0000","last_undegraded":"2024-03-28T05:21:13.759694+0000","last_fullsized":"2024-03-28T05:21:13.759694+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:37:28.797262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877927+0000","last_change":"2024-03-28T05:21:08.041653+0000","last_active":"2024-03-28T05:21:13.877927+0000","last_peered":"2024-03-28T05:21:13.877927+0000","last_clean":"2024-03-28T05:21:13.877927+0000","last_became_active":"2024-03-28T05:21:08.041349+0000","last_became_peered":"2024-03-28T05:21:08.041349+0000","last_unstale":"2024-03-28T05:21:13.877927+0000","last_undegraded":"2024-03-28T05:21:13.877927+0000","last_fullsized":"2024-03-28T05:21:13.877927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:51:23.329132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272178+0000","last_change":"2024-03-28T05:21:12.283320+0000","last_active":"2024-03-28T05:21:13.272178+0000","last_peered":"2024-03-28T05:21:13.272178+0000","last_clean":"2024-03-28T05:21:13.272178+0000","last_became_active":"2024-03-28T05:21:12.282893+0000","last_became_peered":"2024-03-28T05:21:12.282893+0000","last_unstale":"2024-03-28T05:21:13.272178+0000","last_undegraded":"2024-03-28T05:21:13.272178+0000","last_fullsized":"2024-03-28T05:21:13.272178+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:53:19.885680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'17","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:34.388042+0000","last_change":"2024-03-28T05:21:07.263460+0000","last_active":"2024-03-28T05:21:34.388042+0000","last_peered":"2024-03-28T05:21:34.388042+0000","last_clean":"2024-03-28T05:21:34.388042+0000","last_became_active":"2024-03-28T05:21:07.263118+0000","last_became_peered":"2024-03-28T05:21:07.263118+0000","last_unstale":"2024-03-28T05:21:34.388042+0000","last_undegraded":"2024-03-28T05:21:34.388042+0000","last_fullsized":"2024-03-28T05:21:34.388042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:56:18.271666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23141,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":27,"num_write":17,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876389+0000","last_change":"2024-03-28T05:21:08.041307+0000","last_active":"2024-03-28T05:21:13.876389+0000","last_peered":"2024-03-28T05:21:13.876389+0000","last_clean":"2024-03-28T05:21:13.876389+0000","last_became_active":"2024-03-28T05:21:08.040961+0000","last_became_peered":"2024-03-28T05:21:08.040961+0000","last_unstale":"2024-03-28T05:21:13.876389+0000","last_undegraded":"2024-03-28T05:21:13.876389+0000","last_fullsized":"2024-03-28T05:21:13.876389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:12:40.318091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.760184+0000","last_change":"2024-03-28T05:21:12.287222+0000","last_active":"2024-03-28T05:21:13.760184+0000","last_peered":"2024-03-28T05:21:13.760184+0000","last_clean":"2024-03-28T05:21:13.760184+0000","last_became_active":"2024-03-28T05:21:12.286744+0000","last_became_peered":"2024-03-28T05:21:12.286744+0000","last_unstale":"2024-03-28T05:21:13.760184+0000","last_undegraded":"2024-03-28T05:21:13.760184+0000","last_fullsized":"2024-03-28T05:21:13.760184+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:20:45.698054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272057+0000","last_change":"2024-03-28T05:21:07.262932+0000","last_active":"2024-03-28T05:21:13.272057+0000","last_peered":"2024-03-28T05:21:13.272057+0000","last_clean":"2024-03-28T05:21:13.272057+0000","last_became_active":"2024-03-28T05:21:07.262497+0000","last_became_peered":"2024-03-28T05:21:07.262497+0000","last_unstale":"2024-03-28T05:21:13.272057+0000","last_undegraded":"2024-03-28T05:21:13.272057+0000","last_fullsized":"2024-03-28T05:21:13.272057+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:56:58.628442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.828230+0000","last_change":"2024-03-28T05:21:08.046052+0000","last_active":"2024-03-28T05:21:13.828230+0000","last_peered":"2024-03-28T05:21:13.828230+0000","last_clean":"2024-03-28T05:21:13.828230+0000","last_became_active":"2024-03-28T05:21:08.045720+0000","last_became_peered":"2024-03-28T05:21:08.045720+0000","last_unstale":"2024-03-28T05:21:13.828230+0000","last_undegraded":"2024-03-28T05:21:13.828230+0000","last_fullsized":"2024-03-28T05:21:13.828230+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:39:48.084984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876778+0000","last_change":"2024-03-28T05:21:12.294339+0000","last_active":"2024-03-28T05:21:13.876778+0000","last_peered":"2024-03-28T05:21:13.876778+0000","last_clean":"2024-03-28T05:21:13.876778+0000","last_became_active":"2024-03-28T05:21:12.293939+0000","last_became_peered":"2024-03-28T05:21:12.293939+0000","last_unstale":"2024-03-28T05:21:13.876778+0000","last_undegraded":"2024-03-28T05:21:13.876778+0000","last_fullsized":"2024-03-28T05:21:13.876778+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:38:59.851650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759432+0000","last_change":"2024-03-28T05:21:07.262414+0000","last_active":"2024-03-28T05:21:13.759432+0000","last_peered":"2024-03-28T05:21:13.759432+0000","last_clean":"2024-03-28T05:21:13.759432+0000","last_became_active":"2024-03-28T05:21:07.262121+0000","last_became_peered":"2024-03-28T05:21:07.262121+0000","last_unstale":"2024-03-28T05:21:13.759432+0000","last_undegraded":"2024-03-28T05:21:13.759432+0000","last_fullsized":"2024-03-28T05:21:13.759432+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:15:52.832449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.275896+0000","last_change":"2024-03-28T05:21:08.049263+0000","last_active":"2024-03-28T05:21:13.275896+0000","last_peered":"2024-03-28T05:21:13.275896+0000","last_clean":"2024-03-28T05:21:13.275896+0000","last_became_active":"2024-03-28T05:21:08.048912+0000","last_became_peered":"2024-03-28T05:21:08.048912+0000","last_unstale":"2024-03-28T05:21:13.275896+0000","last_undegraded":"2024-03-28T05:21:13.275896+0000","last_fullsized":"2024-03-28T05:21:13.275896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:36:06.562731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272523+0000","last_change":"2024-03-28T05:21:12.290345+0000","last_active":"2024-03-28T05:21:13.272523+0000","last_peered":"2024-03-28T05:21:13.272523+0000","last_clean":"2024-03-28T05:21:13.272523+0000","last_became_active":"2024-03-28T05:21:12.289945+0000","last_became_peered":"2024-03-28T05:21:12.289945+0000","last_unstale":"2024-03-28T05:21:13.272523+0000","last_undegraded":"2024-03-28T05:21:13.272523+0000","last_fullsized":"2024-03-28T05:21:13.272523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:41:43.024707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876845+0000","last_change":"2024-03-28T05:21:07.262588+0000","last_active":"2024-03-28T05:21:13.876845+0000","last_peered":"2024-03-28T05:21:13.876845+0000","last_clean":"2024-03-28T05:21:13.876845+0000","last_became_active":"2024-03-28T05:21:07.262237+0000","last_became_peered":"2024-03-28T05:21:07.262237+0000","last_unstale":"2024-03-28T05:21:13.876845+0000","last_undegraded":"2024-03-28T05:21:13.876845+0000","last_fullsized":"2024-03-28T05:21:13.876845+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:49:08.207031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.275533+0000","last_change":"2024-03-28T05:21:08.041473+0000","last_active":"2024-03-28T05:21:13.275533+0000","last_peered":"2024-03-28T05:21:13.275533+0000","last_clean":"2024-03-28T05:21:13.275533+0000","last_became_active":"2024-03-28T05:21:08.041171+0000","last_became_peered":"2024-03-28T05:21:08.041171+0000","last_unstale":"2024-03-28T05:21:13.275533+0000","last_undegraded":"2024-03-28T05:21:13.275533+0000","last_fullsized":"2024-03-28T05:21:13.275533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:56:26.652129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.828711+0000","last_change":"2024-03-28T05:21:12.296288+0000","last_active":"2024-03-28T05:21:13.828711+0000","last_peered":"2024-03-28T05:21:13.828711+0000","last_clean":"2024-03-28T05:21:13.828711+0000","last_became_active":"2024-03-28T05:21:12.295896+0000","last_became_peered":"2024-03-28T05:21:12.295896+0000","last_unstale":"2024-03-28T05:21:13.828711+0000","last_undegraded":"2024-03-28T05:21:13.828711+0000","last_fullsized":"2024-03-28T05:21:13.828711+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:05:05.779622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.875895+0000","last_change":"2024-03-28T05:21:07.262448+0000","last_active":"2024-03-28T05:21:13.875895+0000","last_peered":"2024-03-28T05:21:13.875895+0000","last_clean":"2024-03-28T05:21:13.875895+0000","last_became_active":"2024-03-28T05:21:07.261966+0000","last_became_peered":"2024-03-28T05:21:07.261966+0000","last_unstale":"2024-03-28T05:21:13.875895+0000","last_undegraded":"2024-03-28T05:21:13.875895+0000","last_fullsized":"2024-03-28T05:21:13.875895+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:45:37.904160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759270+0000","last_change":"2024-03-28T05:21:08.045929+0000","last_active":"2024-03-28T05:21:13.759270+0000","last_peered":"2024-03-28T05:21:13.759270+0000","last_clean":"2024-03-28T05:21:13.759270+0000","last_became_active":"2024-03-28T05:21:08.045640+0000","last_became_peered":"2024-03-28T05:21:08.045640+0000","last_unstale":"2024-03-28T05:21:13.759270+0000","last_undegraded":"2024-03-28T05:21:13.759270+0000","last_fullsized":"2024-03-28T05:21:13.759270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:38:01.546332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759293+0000","last_change":"2024-03-28T05:21:12.292516+0000","last_active":"2024-03-28T05:21:13.759293+0000","last_peered":"2024-03-28T05:21:13.759293+0000","last_clean":"2024-03-28T05:21:13.759293+0000","last_became_active":"2024-03-28T05:21:12.292129+0000","last_became_peered":"2024-03-28T05:21:12.292129+0000","last_unstale":"2024-03-28T05:21:13.759293+0000","last_undegraded":"2024-03-28T05:21:13.759293+0000","last_fullsized":"2024-03-28T05:21:13.759293+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:32:25.086143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876443+0000","last_change":"2024-03-28T05:21:07.267442+0000","last_active":"2024-03-28T05:21:13.876443+0000","last_peered":"2024-03-28T05:21:13.876443+0000","last_clean":"2024-03-28T05:21:13.876443+0000","last_became_active":"2024-03-28T05:21:07.267140+0000","last_became_peered":"2024-03-28T05:21:07.267140+0000","last_unstale":"2024-03-28T05:21:13.876443+0000","last_undegraded":"2024-03-28T05:21:13.876443+0000","last_fullsized":"2024-03-28T05:21:13.876443+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:43:05.543211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876743+0000","last_change":"2024-03-28T05:21:08.040304+0000","last_active":"2024-03-28T05:21:13.876743+0000","last_peered":"2024-03-28T05:21:13.876743+0000","last_clean":"2024-03-28T05:21:13.876743+0000","last_became_active":"2024-03-28T05:21:08.039976+0000","last_became_peered":"2024-03-28T05:21:08.039976+0000","last_unstale":"2024-03-28T05:21:13.876743+0000","last_undegraded":"2024-03-28T05:21:13.876743+0000","last_fullsized":"2024-03-28T05:21:13.876743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:57:33.062202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876082+0000","last_change":"2024-03-28T05:21:12.289177+0000","last_active":"2024-03-28T05:21:13.876082+0000","last_peered":"2024-03-28T05:21:13.876082+0000","last_clean":"2024-03-28T05:21:13.876082+0000","last_became_active":"2024-03-28T05:21:12.288615+0000","last_became_peered":"2024-03-28T05:21:12.288615+0000","last_unstale":"2024-03-28T05:21:13.876082+0000","last_undegraded":"2024-03-28T05:21:13.876082+0000","last_fullsized":"2024-03-28T05:21:13.876082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:27:51.819911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272089+0000","last_change":"2024-03-28T05:21:07.272325+0000","last_active":"2024-03-28T05:21:13.272089+0000","last_peered":"2024-03-28T05:21:13.272089+0000","last_clean":"2024-03-28T05:21:13.272089+0000","last_became_active":"2024-03-28T05:21:07.271965+0000","last_became_peered":"2024-03-28T05:21:07.271965+0000","last_unstale":"2024-03-28T05:21:13.272089+0000","last_undegraded":"2024-03-28T05:21:13.272089+0000","last_fullsized":"2024-03-28T05:21:13.272089+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:20:24.723627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877346+0000","last_change":"2024-03-28T05:21:08.047122+0000","last_active":"2024-03-28T05:21:13.877346+0000","last_peered":"2024-03-28T05:21:13.877346+0000","last_clean":"2024-03-28T05:21:13.877346+0000","last_became_active":"2024-03-28T05:21:08.046828+0000","last_became_peered":"2024-03-28T05:21:08.046828+0000","last_unstale":"2024-03-28T05:21:13.877346+0000","last_undegraded":"2024-03-28T05:21:13.877346+0000","last_fullsized":"2024-03-28T05:21:13.877346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:07:16.940600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.828348+0000","last_change":"2024-03-28T05:21:12.293325+0000","last_active":"2024-03-28T05:21:13.828348+0000","last_peered":"2024-03-28T05:21:13.828348+0000","last_clean":"2024-03-28T05:21:13.828348+0000","last_became_active":"2024-03-28T05:21:12.292876+0000","last_became_peered":"2024-03-28T05:21:12.292876+0000","last_unstale":"2024-03-28T05:21:13.828348+0000","last_undegraded":"2024-03-28T05:21:13.828348+0000","last_fullsized":"2024-03-28T05:21:13.828348+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:35:25.173848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.275661+0000","last_change":"2024-03-28T05:21:07.262435+0000","last_active":"2024-03-28T05:21:13.275661+0000","last_peered":"2024-03-28T05:21:13.275661+0000","last_clean":"2024-03-28T05:21:13.275661+0000","last_became_active":"2024-03-28T05:21:07.262019+0000","last_became_peered":"2024-03-28T05:21:07.262019+0000","last_unstale":"2024-03-28T05:21:13.275661+0000","last_undegraded":"2024-03-28T05:21:13.275661+0000","last_fullsized":"2024-03-28T05:21:13.275661+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:41:31.348300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876993+0000","last_change":"2024-03-28T05:21:08.043614+0000","last_active":"2024-03-28T05:21:13.876993+0000","last_peered":"2024-03-28T05:21:13.876993+0000","last_clean":"2024-03-28T05:21:13.876993+0000","last_became_active":"2024-03-28T05:21:08.043187+0000","last_became_peered":"2024-03-28T05:21:08.043187+0000","last_unstale":"2024-03-28T05:21:13.876993+0000","last_undegraded":"2024-03-28T05:21:13.876993+0000","last_fullsized":"2024-03-28T05:21:13.876993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:55:08.972707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.828505+0000","last_change":"2024-03-28T05:21:12.289381+0000","last_active":"2024-03-28T05:21:13.828505+0000","last_peered":"2024-03-28T05:21:13.828505+0000","last_clean":"2024-03-28T05:21:13.828505+0000","last_became_active":"2024-03-28T05:21:12.288987+0000","last_became_peered":"2024-03-28T05:21:12.288987+0000","last_unstale":"2024-03-28T05:21:13.828505+0000","last_undegraded":"2024-03-28T05:21:13.828505+0000","last_fullsized":"2024-03-28T05:21:13.828505+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:31:46.296664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759296+0000","last_change":"2024-03-28T05:21:08.047077+0000","last_active":"2024-03-28T05:21:13.759296+0000","last_peered":"2024-03-28T05:21:13.759296+0000","last_clean":"2024-03-28T05:21:13.759296+0000","last_became_active":"2024-03-28T05:21:08.046783+0000","last_became_peered":"2024-03-28T05:21:08.046783+0000","last_unstale":"2024-03-28T05:21:13.759296+0000","last_undegraded":"2024-03-28T05:21:13.759296+0000","last_fullsized":"2024-03-28T05:21:13.759296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:52:07.823247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.827997+0000","last_change":"2024-03-28T05:21:07.271028+0000","last_active":"2024-03-28T05:21:13.827997+0000","last_peered":"2024-03-28T05:21:13.827997+0000","last_clean":"2024-03-28T05:21:13.827997+0000","last_became_active":"2024-03-28T05:21:07.270404+0000","last_became_peered":"2024-03-28T05:21:07.270404+0000","last_unstale":"2024-03-28T05:21:13.827997+0000","last_undegraded":"2024-03-28T05:21:13.827997+0000","last_fullsized":"2024-03-28T05:21:13.827997+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:01:23.875231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.275924+0000","last_change":"2024-03-28T05:21:12.289881+0000","last_active":"2024-03-28T05:21:13.275924+0000","last_peered":"2024-03-28T05:21:13.275924+0000","last_clean":"2024-03-28T05:21:13.275924+0000","last_became_active":"2024-03-28T05:21:12.289476+0000","last_became_peered":"2024-03-28T05:21:12.289476+0000","last_unstale":"2024-03-28T05:21:13.275924+0000","last_undegraded":"2024-03-28T05:21:13.275924+0000","last_fullsized":"2024-03-28T05:21:13.275924+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:01:31.800429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876381+0000","last_change":"2024-03-28T05:21:08.044169+0000","last_active":"2024-03-28T05:21:13.876381+0000","last_peered":"2024-03-28T05:21:13.876381+0000","last_clean":"2024-03-28T05:21:13.876381+0000","last_became_active":"2024-03-28T05:21:08.043836+0000","last_became_peered":"2024-03-28T05:21:08.043836+0000","last_unstale":"2024-03-28T05:21:13.876381+0000","last_undegraded":"2024-03-28T05:21:13.876381+0000","last_fullsized":"2024-03-28T05:21:13.876381+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:07:41.128111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876912+0000","last_change":"2024-03-28T05:21:07.267368+0000","last_active":"2024-03-28T05:21:13.876912+0000","last_peered":"2024-03-28T05:21:13.876912+0000","last_clean":"2024-03-28T05:21:13.876912+0000","last_became_active":"2024-03-28T05:21:07.266995+0000","last_became_peered":"2024-03-28T05:21:07.266995+0000","last_unstale":"2024-03-28T05:21:13.876912+0000","last_undegraded":"2024-03-28T05:21:13.876912+0000","last_fullsized":"2024-03-28T05:21:13.876912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:24:40.146950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":356,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.275411+0000","last_change":"2024-03-28T05:19:57.662704+0000","last_active":"2024-03-28T05:21:13.275411+0000","last_peered":"2024-03-28T05:21:13.275411+0000","last_clean":"2024-03-28T05:21:13.275411+0000","last_became_active":"2024-03-28T05:19:57.662332+0000","last_became_peered":"2024-03-28T05:19:57.662332+0000","last_unstale":"2024-03-28T05:21:13.275411+0000","last_undegraded":"2024-03-28T05:21:13.275411+0000","last_fullsized":"2024-03-28T05:21:13.275411+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:16:33.255097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:16:33.255097+0000","last_clean_scrub_stamp":"2024-03-28T05:16:33.255097+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:40:47.655643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759653+0000","last_change":"2024-03-28T05:21:12.291993+0000","last_active":"2024-03-28T05:21:13.759653+0000","last_peered":"2024-03-28T05:21:13.759653+0000","last_clean":"2024-03-28T05:21:13.759653+0000","last_became_active":"2024-03-28T05:21:12.291641+0000","last_became_peered":"2024-03-28T05:21:12.291641+0000","last_unstale":"2024-03-28T05:21:13.759653+0000","last_undegraded":"2024-03-28T05:21:13.759653+0000","last_fullsized":"2024-03-28T05:21:13.759653+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:12:02.217568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.827988+0000","last_change":"2024-03-28T05:21:08.043036+0000","last_active":"2024-03-28T05:21:13.827988+0000","last_peered":"2024-03-28T05:21:13.827988+0000","last_clean":"2024-03-28T05:21:13.827988+0000","last_became_active":"2024-03-28T05:21:08.042467+0000","last_became_peered":"2024-03-28T05:21:08.042467+0000","last_unstale":"2024-03-28T05:21:13.827988+0000","last_undegraded":"2024-03-28T05:21:13.827988+0000","last_fullsized":"2024-03-28T05:21:13.827988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:16:41.718497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.275466+0000","last_change":"2024-03-28T05:21:07.260270+0000","last_active":"2024-03-28T05:21:13.275466+0000","last_peered":"2024-03-28T05:21:13.275466+0000","last_clean":"2024-03-28T05:21:13.275466+0000","last_became_active":"2024-03-28T05:21:07.259727+0000","last_became_peered":"2024-03-28T05:21:07.259727+0000","last_unstale":"2024-03-28T05:21:13.275466+0000","last_undegraded":"2024-03-28T05:21:13.275466+0000","last_fullsized":"2024-03-28T05:21:13.275466+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:19:15.883172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877032+0000","last_change":"2024-03-28T05:21:12.294439+0000","last_active":"2024-03-28T05:21:13.877032+0000","last_peered":"2024-03-28T05:21:13.877032+0000","last_clean":"2024-03-28T05:21:13.877032+0000","last_became_active":"2024-03-28T05:21:12.294044+0000","last_became_peered":"2024-03-28T05:21:12.294044+0000","last_unstale":"2024-03-28T05:21:13.877032+0000","last_undegraded":"2024-03-28T05:21:13.877032+0000","last_fullsized":"2024-03-28T05:21:13.877032+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:22:51.218266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876398+0000","last_change":"2024-03-28T05:21:07.272735+0000","last_active":"2024-03-28T05:21:13.876398+0000","last_peered":"2024-03-28T05:21:13.876398+0000","last_clean":"2024-03-28T05:21:13.876398+0000","last_became_active":"2024-03-28T05:21:07.272407+0000","last_became_peered":"2024-03-28T05:21:07.272407+0000","last_unstale":"2024-03-28T05:21:13.876398+0000","last_undegraded":"2024-03-28T05:21:13.876398+0000","last_fullsized":"2024-03-28T05:21:13.876398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:55:02.165183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.828555+0000","last_change":"2024-03-28T05:21:08.042577+0000","last_active":"2024-03-28T05:21:13.828555+0000","last_peered":"2024-03-28T05:21:13.828555+0000","last_clean":"2024-03-28T05:21:13.828555+0000","last_became_active":"2024-03-28T05:21:08.042248+0000","last_became_peered":"2024-03-28T05:21:08.042248+0000","last_unstale":"2024-03-28T05:21:13.828555+0000","last_undegraded":"2024-03-28T05:21:13.828555+0000","last_fullsized":"2024-03-28T05:21:13.828555+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:31:15.856177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876108+0000","last_change":"2024-03-28T05:21:12.294141+0000","last_active":"2024-03-28T05:21:13.876108+0000","last_peered":"2024-03-28T05:21:13.876108+0000","last_clean":"2024-03-28T05:21:13.876108+0000","last_became_active":"2024-03-28T05:21:12.293746+0000","last_became_peered":"2024-03-28T05:21:12.293746+0000","last_unstale":"2024-03-28T05:21:13.876108+0000","last_undegraded":"2024-03-28T05:21:13.876108+0000","last_fullsized":"2024-03-28T05:21:13.876108+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:06:18.871631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272615+0000","last_change":"2024-03-28T05:21:07.275515+0000","last_active":"2024-03-28T05:21:13.272615+0000","last_peered":"2024-03-28T05:21:13.272615+0000","last_clean":"2024-03-28T05:21:13.272615+0000","last_became_active":"2024-03-28T05:21:07.275188+0000","last_became_peered":"2024-03-28T05:21:07.275188+0000","last_unstale":"2024-03-28T05:21:13.272615+0000","last_undegraded":"2024-03-28T05:21:13.272615+0000","last_fullsized":"2024-03-28T05:21:13.272615+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:34:29.351759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876578+0000","last_change":"2024-03-28T05:21:08.051019+0000","last_active":"2024-03-28T05:21:13.876578+0000","last_peered":"2024-03-28T05:21:13.876578+0000","last_clean":"2024-03-28T05:21:13.876578+0000","last_became_active":"2024-03-28T05:21:08.050696+0000","last_became_peered":"2024-03-28T05:21:08.050696+0000","last_unstale":"2024-03-28T05:21:13.876578+0000","last_undegraded":"2024-03-28T05:21:13.876578+0000","last_fullsized":"2024-03-28T05:21:13.876578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:06:23.318153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.271749+0000","last_change":"2024-03-28T05:21:12.285980+0000","last_active":"2024-03-28T05:21:13.271749+0000","last_peered":"2024-03-28T05:21:13.271749+0000","last_clean":"2024-03-28T05:21:13.271749+0000","last_became_active":"2024-03-28T05:21:12.285578+0000","last_became_peered":"2024-03-28T05:21:12.285578+0000","last_unstale":"2024-03-28T05:21:13.271749+0000","last_undegraded":"2024-03-28T05:21:13.271749+0000","last_fullsized":"2024-03-28T05:21:13.271749+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:33:58.377753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877473+0000","last_change":"2024-03-28T05:21:07.256475+0000","last_active":"2024-03-28T05:21:13.877473+0000","last_peered":"2024-03-28T05:21:13.877473+0000","last_clean":"2024-03-28T05:21:13.877473+0000","last_became_active":"2024-03-28T05:21:07.256125+0000","last_became_peered":"2024-03-28T05:21:07.256125+0000","last_unstale":"2024-03-28T05:21:13.877473+0000","last_undegraded":"2024-03-28T05:21:13.877473+0000","last_fullsized":"2024-03-28T05:21:13.877473+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:42.263704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:14.592660+0000","last_change":"2024-03-28T05:21:08.044417+0000","last_active":"2024-03-28T05:21:14.592660+0000","last_peered":"2024-03-28T05:21:14.592660+0000","last_clean":"2024-03-28T05:21:14.592660+0000","last_became_active":"2024-03-28T05:21:08.043931+0000","last_became_peered":"2024-03-28T05:21:08.043931+0000","last_unstale":"2024-03-28T05:21:14.592660+0000","last_undegraded":"2024-03-28T05:21:14.592660+0000","last_fullsized":"2024-03-28T05:21:14.592660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:43:55.013625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876747+0000","last_change":"2024-03-28T05:21:12.300641+0000","last_active":"2024-03-28T05:21:13.876747+0000","last_peered":"2024-03-28T05:21:13.876747+0000","last_clean":"2024-03-28T05:21:13.876747+0000","last_became_active":"2024-03-28T05:21:12.300013+0000","last_became_peered":"2024-03-28T05:21:12.300013+0000","last_unstale":"2024-03-28T05:21:13.876747+0000","last_undegraded":"2024-03-28T05:21:13.876747+0000","last_fullsized":"2024-03-28T05:21:13.876747+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:38:42.474798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876368+0000","last_change":"2024-03-28T05:21:07.261548+0000","last_active":"2024-03-28T05:21:13.876368+0000","last_peered":"2024-03-28T05:21:13.876368+0000","last_clean":"2024-03-28T05:21:13.876368+0000","last_became_active":"2024-03-28T05:21:07.261201+0000","last_became_peered":"2024-03-28T05:21:07.261201+0000","last_unstale":"2024-03-28T05:21:13.876368+0000","last_undegraded":"2024-03-28T05:21:13.876368+0000","last_fullsized":"2024-03-28T05:21:13.876368+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:35:48.899064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.875866+0000","last_change":"2024-03-28T05:21:08.045168+0000","last_active":"2024-03-28T05:21:13.875866+0000","last_peered":"2024-03-28T05:21:13.875866+0000","last_clean":"2024-03-28T05:21:13.875866+0000","last_became_active":"2024-03-28T05:21:08.044654+0000","last_became_peered":"2024-03-28T05:21:08.044654+0000","last_unstale":"2024-03-28T05:21:13.875866+0000","last_undegraded":"2024-03-28T05:21:13.875866+0000","last_fullsized":"2024-03-28T05:21:13.875866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:48:14.592798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272530+0000","last_change":"2024-03-28T05:21:12.285593+0000","last_active":"2024-03-28T05:21:13.272530+0000","last_peered":"2024-03-28T05:21:13.272530+0000","last_clean":"2024-03-28T05:21:13.272530+0000","last_became_active":"2024-03-28T05:21:12.285202+0000","last_became_peered":"2024-03-28T05:21:12.285202+0000","last_unstale":"2024-03-28T05:21:13.272530+0000","last_undegraded":"2024-03-28T05:21:13.272530+0000","last_fullsized":"2024-03-28T05:21:13.272530+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:00:40.198255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876134+0000","last_change":"2024-03-28T05:21:07.271352+0000","last_active":"2024-03-28T05:21:13.876134+0000","last_peered":"2024-03-28T05:21:13.876134+0000","last_clean":"2024-03-28T05:21:13.876134+0000","last_became_active":"2024-03-28T05:21:07.271038+0000","last_became_peered":"2024-03-28T05:21:07.271038+0000","last_unstale":"2024-03-28T05:21:13.876134+0000","last_undegraded":"2024-03-28T05:21:13.876134+0000","last_fullsized":"2024-03-28T05:21:13.876134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:37:41.742550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876529+0000","last_change":"2024-03-28T05:21:08.043503+0000","last_active":"2024-03-28T05:21:13.876529+0000","last_peered":"2024-03-28T05:21:13.876529+0000","last_clean":"2024-03-28T05:21:13.876529+0000","last_became_active":"2024-03-28T05:21:08.043173+0000","last_became_peered":"2024-03-28T05:21:08.043173+0000","last_unstale":"2024-03-28T05:21:13.876529+0000","last_undegraded":"2024-03-28T05:21:13.876529+0000","last_fullsized":"2024-03-28T05:21:13.876529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:23:30.576539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877262+0000","last_change":"2024-03-28T05:21:12.300600+0000","last_active":"2024-03-28T05:21:13.877262+0000","last_peered":"2024-03-28T05:21:13.877262+0000","last_clean":"2024-03-28T05:21:13.877262+0000","last_became_active":"2024-03-28T05:21:12.299931+0000","last_became_peered":"2024-03-28T05:21:12.299931+0000","last_unstale":"2024-03-28T05:21:13.877262+0000","last_undegraded":"2024-03-28T05:21:13.877262+0000","last_fullsized":"2024-03-28T05:21:13.877262+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:29:22.405410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.271831+0000","last_change":"2024-03-28T05:21:07.265708+0000","last_active":"2024-03-28T05:21:13.271831+0000","last_peered":"2024-03-28T05:21:13.271831+0000","last_clean":"2024-03-28T05:21:13.271831+0000","last_became_active":"2024-03-28T05:21:07.265416+0000","last_became_peered":"2024-03-28T05:21:07.265416+0000","last_unstale":"2024-03-28T05:21:13.271831+0000","last_undegraded":"2024-03-28T05:21:13.271831+0000","last_fullsized":"2024-03-28T05:21:13.271831+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:14:05.003658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876256+0000","last_change":"2024-03-28T05:21:08.051458+0000","last_active":"2024-03-28T05:21:13.876256+0000","last_peered":"2024-03-28T05:21:13.876256+0000","last_clean":"2024-03-28T05:21:13.876256+0000","last_became_active":"2024-03-28T05:21:08.050997+0000","last_became_peered":"2024-03-28T05:21:08.050997+0000","last_unstale":"2024-03-28T05:21:13.876256+0000","last_undegraded":"2024-03-28T05:21:13.876256+0000","last_fullsized":"2024-03-28T05:21:13.876256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:33:41.046841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876764+0000","last_change":"2024-03-28T05:21:12.285977+0000","last_active":"2024-03-28T05:21:13.876764+0000","last_peered":"2024-03-28T05:21:13.876764+0000","last_clean":"2024-03-28T05:21:13.876764+0000","last_became_active":"2024-03-28T05:21:12.285439+0000","last_became_peered":"2024-03-28T05:21:12.285439+0000","last_unstale":"2024-03-28T05:21:13.876764+0000","last_undegraded":"2024-03-28T05:21:13.876764+0000","last_fullsized":"2024-03-28T05:21:13.876764+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:07:34.409001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876144+0000","last_change":"2024-03-28T05:21:07.268187+0000","last_active":"2024-03-28T05:21:13.876144+0000","last_peered":"2024-03-28T05:21:13.876144+0000","last_clean":"2024-03-28T05:21:13.876144+0000","last_became_active":"2024-03-28T05:21:07.266791+0000","last_became_peered":"2024-03-28T05:21:07.266791+0000","last_unstale":"2024-03-28T05:21:13.876144+0000","last_undegraded":"2024-03-28T05:21:13.876144+0000","last_fullsized":"2024-03-28T05:21:13.876144+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:22:03.683043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.275846+0000","last_change":"2024-03-28T05:21:08.054067+0000","last_active":"2024-03-28T05:21:13.275846+0000","last_peered":"2024-03-28T05:21:13.275846+0000","last_clean":"2024-03-28T05:21:13.275846+0000","last_became_active":"2024-03-28T05:21:08.053759+0000","last_became_peered":"2024-03-28T05:21:08.053759+0000","last_unstale":"2024-03-28T05:21:13.275846+0000","last_undegraded":"2024-03-28T05:21:13.275846+0000","last_fullsized":"2024-03-28T05:21:13.275846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:47:39.329564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876564+0000","last_change":"2024-03-28T05:21:12.294547+0000","last_active":"2024-03-28T05:21:13.876564+0000","last_peered":"2024-03-28T05:21:13.876564+0000","last_clean":"2024-03-28T05:21:13.876564+0000","last_became_active":"2024-03-28T05:21:12.294146+0000","last_became_peered":"2024-03-28T05:21:12.294146+0000","last_unstale":"2024-03-28T05:21:13.876564+0000","last_undegraded":"2024-03-28T05:21:13.876564+0000","last_fullsized":"2024-03-28T05:21:13.876564+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:24:28.939378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272145+0000","last_change":"2024-03-28T05:21:07.260749+0000","last_active":"2024-03-28T05:21:13.272145+0000","last_peered":"2024-03-28T05:21:13.272145+0000","last_clean":"2024-03-28T05:21:13.272145+0000","last_became_active":"2024-03-28T05:21:07.260439+0000","last_became_peered":"2024-03-28T05:21:07.260439+0000","last_unstale":"2024-03-28T05:21:13.272145+0000","last_undegraded":"2024-03-28T05:21:13.272145+0000","last_fullsized":"2024-03-28T05:21:13.272145+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:24:04.257080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.276133+0000","last_change":"2024-03-28T05:21:08.051079+0000","last_active":"2024-03-28T05:21:13.276133+0000","last_peered":"2024-03-28T05:21:13.276133+0000","last_clean":"2024-03-28T05:21:13.276133+0000","last_became_active":"2024-03-28T05:21:08.050790+0000","last_became_peered":"2024-03-28T05:21:08.050790+0000","last_unstale":"2024-03-28T05:21:13.276133+0000","last_undegraded":"2024-03-28T05:21:13.276133+0000","last_fullsized":"2024-03-28T05:21:13.276133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:04:39.205068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:14.592244+0000","last_change":"2024-03-28T05:21:12.291964+0000","last_active":"2024-03-28T05:21:14.592244+0000","last_peered":"2024-03-28T05:21:14.592244+0000","last_clean":"2024-03-28T05:21:14.592244+0000","last_became_active":"2024-03-28T05:21:12.291553+0000","last_became_peered":"2024-03-28T05:21:12.291553+0000","last_unstale":"2024-03-28T05:21:14.592244+0000","last_undegraded":"2024-03-28T05:21:14.592244+0000","last_fullsized":"2024-03-28T05:21:14.592244+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:10:34.180033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759896+0000","last_change":"2024-03-28T05:21:07.263204+0000","last_active":"2024-03-28T05:21:13.759896+0000","last_peered":"2024-03-28T05:21:13.759896+0000","last_clean":"2024-03-28T05:21:13.759896+0000","last_became_active":"2024-03-28T05:21:07.262919+0000","last_became_peered":"2024-03-28T05:21:07.262919+0000","last_unstale":"2024-03-28T05:21:13.759896+0000","last_undegraded":"2024-03-28T05:21:13.759896+0000","last_fullsized":"2024-03-28T05:21:13.759896+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:53:19.505448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.275833+0000","last_change":"2024-03-28T05:21:08.044499+0000","last_active":"2024-03-28T05:21:13.275833+0000","last_peered":"2024-03-28T05:21:13.275833+0000","last_clean":"2024-03-28T05:21:13.275833+0000","last_became_active":"2024-03-28T05:21:08.043371+0000","last_became_peered":"2024-03-28T05:21:08.043371+0000","last_unstale":"2024-03-28T05:21:13.275833+0000","last_undegraded":"2024-03-28T05:21:13.275833+0000","last_fullsized":"2024-03-28T05:21:13.275833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:44:35.901625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759128+0000","last_change":"2024-03-28T05:21:12.282443+0000","last_active":"2024-03-28T05:21:13.759128+0000","last_peered":"2024-03-28T05:21:13.759128+0000","last_clean":"2024-03-28T05:21:13.759128+0000","last_became_active":"2024-03-28T05:21:12.282065+0000","last_became_peered":"2024-03-28T05:21:12.282065+0000","last_unstale":"2024-03-28T05:21:13.759128+0000","last_undegraded":"2024-03-28T05:21:13.759128+0000","last_fullsized":"2024-03-28T05:21:13.759128+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:26:55.366554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.828687+0000","last_change":"2024-03-28T05:21:07.266038+0000","last_active":"2024-03-28T05:21:13.828687+0000","last_peered":"2024-03-28T05:21:13.828687+0000","last_clean":"2024-03-28T05:21:13.828687+0000","last_became_active":"2024-03-28T05:21:07.265728+0000","last_became_peered":"2024-03-28T05:21:07.265728+0000","last_unstale":"2024-03-28T05:21:13.828687+0000","last_undegraded":"2024-03-28T05:21:13.828687+0000","last_fullsized":"2024-03-28T05:21:13.828687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:06:05.539565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.276136+0000","last_change":"2024-03-28T05:21:08.047626+0000","last_active":"2024-03-28T05:21:13.276136+0000","last_peered":"2024-03-28T05:21:13.276136+0000","last_clean":"2024-03-28T05:21:13.276136+0000","last_became_active":"2024-03-28T05:21:08.047251+0000","last_became_peered":"2024-03-28T05:21:08.047251+0000","last_unstale":"2024-03-28T05:21:13.276136+0000","last_undegraded":"2024-03-28T05:21:13.276136+0000","last_fullsized":"2024-03-28T05:21:13.276136+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:25:36.043024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876594+0000","last_change":"2024-03-28T05:21:08.042216+0000","last_active":"2024-03-28T05:21:13.876594+0000","last_peered":"2024-03-28T05:21:13.876594+0000","last_clean":"2024-03-28T05:21:13.876594+0000","last_became_active":"2024-03-28T05:21:08.041830+0000","last_became_peered":"2024-03-28T05:21:08.041830+0000","last_unstale":"2024-03-28T05:21:13.876594+0000","last_undegraded":"2024-03-28T05:21:13.876594+0000","last_fullsized":"2024-03-28T05:21:13.876594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:56:19.785472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.271813+0000","last_change":"2024-03-28T05:21:08.042043+0000","last_active":"2024-03-28T05:21:13.271813+0000","last_peered":"2024-03-28T05:21:13.271813+0000","last_clean":"2024-03-28T05:21:13.271813+0000","last_became_active":"2024-03-28T05:21:08.041708+0000","last_became_peered":"2024-03-28T05:21:08.041708+0000","last_unstale":"2024-03-28T05:21:13.271813+0000","last_undegraded":"2024-03-28T05:21:13.271813+0000","last_fullsized":"2024-03-28T05:21:13.271813+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:11:36.620653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.271728+0000","last_change":"2024-03-28T05:21:08.043471+0000","last_active":"2024-03-28T05:21:13.271728+0000","last_peered":"2024-03-28T05:21:13.271728+0000","last_clean":"2024-03-28T05:21:13.271728+0000","last_became_active":"2024-03-28T05:21:08.043144+0000","last_became_peered":"2024-03-28T05:21:08.043144+0000","last_unstale":"2024-03-28T05:21:13.271728+0000","last_undegraded":"2024-03-28T05:21:13.271728+0000","last_fullsized":"2024-03-28T05:21:13.271728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:18:40.580589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:14.592076+0000","last_change":"2024-03-28T05:21:08.050105+0000","last_active":"2024-03-28T05:21:14.592076+0000","last_peered":"2024-03-28T05:21:14.592076+0000","last_clean":"2024-03-28T05:21:14.592076+0000","last_became_active":"2024-03-28T05:21:08.047056+0000","last_became_peered":"2024-03-28T05:21:08.047056+0000","last_unstale":"2024-03-28T05:21:14.592076+0000","last_undegraded":"2024-03-28T05:21:14.592076+0000","last_fullsized":"2024-03-28T05:21:14.592076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:59:54.341517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.828462+0000","last_change":"2024-03-28T05:21:08.043253+0000","last_active":"2024-03-28T05:21:13.828462+0000","last_peered":"2024-03-28T05:21:13.828462+0000","last_clean":"2024-03-28T05:21:13.828462+0000","last_became_active":"2024-03-28T05:21:08.042934+0000","last_became_peered":"2024-03-28T05:21:08.042934+0000","last_unstale":"2024-03-28T05:21:13.828462+0000","last_undegraded":"2024-03-28T05:21:13.828462+0000","last_fullsized":"2024-03-28T05:21:13.828462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:55:41.612580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876946+0000","last_change":"2024-03-28T05:21:08.045349+0000","last_active":"2024-03-28T05:21:13.876946+0000","last_peered":"2024-03-28T05:21:13.876946+0000","last_clean":"2024-03-28T05:21:13.876946+0000","last_became_active":"2024-03-28T05:21:08.044477+0000","last_became_peered":"2024-03-28T05:21:08.044477+0000","last_unstale":"2024-03-28T05:21:13.876946+0000","last_undegraded":"2024-03-28T05:21:13.876946+0000","last_fullsized":"2024-03-28T05:21:13.876946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:38:34.732765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.272152+0000","last_change":"2024-03-28T05:21:08.044801+0000","last_active":"2024-03-28T05:21:13.272152+0000","last_peered":"2024-03-28T05:21:13.272152+0000","last_clean":"2024-03-28T05:21:13.272152+0000","last_became_active":"2024-03-28T05:21:08.042208+0000","last_became_peered":"2024-03-28T05:21:08.042208+0000","last_unstale":"2024-03-28T05:21:13.272152+0000","last_undegraded":"2024-03-28T05:21:13.272152+0000","last_fullsized":"2024-03-28T05:21:13.272152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:49:15.663953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876689+0000","last_change":"2024-03-28T05:21:08.047843+0000","last_active":"2024-03-28T05:21:13.876689+0000","last_peered":"2024-03-28T05:21:13.876689+0000","last_clean":"2024-03-28T05:21:13.876689+0000","last_became_active":"2024-03-28T05:21:08.047466+0000","last_became_peered":"2024-03-28T05:21:08.047466+0000","last_unstale":"2024-03-28T05:21:13.876689+0000","last_undegraded":"2024-03-28T05:21:13.876689+0000","last_fullsized":"2024-03-28T05:21:13.876689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:25:03.424233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.759293+0000","last_change":"2024-03-28T05:21:08.040786+0000","last_active":"2024-03-28T05:21:13.759293+0000","last_peered":"2024-03-28T05:21:13.759293+0000","last_clean":"2024-03-28T05:21:13.759293+0000","last_became_active":"2024-03-28T05:21:08.040495+0000","last_became_peered":"2024-03-28T05:21:08.040495+0000","last_unstale":"2024-03-28T05:21:13.759293+0000","last_undegraded":"2024-03-28T05:21:13.759293+0000","last_fullsized":"2024-03-28T05:21:13.759293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:06:31.626221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876276+0000","last_change":"2024-03-28T05:21:08.050245+0000","last_active":"2024-03-28T05:21:13.876276+0000","last_peered":"2024-03-28T05:21:13.876276+0000","last_clean":"2024-03-28T05:21:13.876276+0000","last_became_active":"2024-03-28T05:21:08.049932+0000","last_became_peered":"2024-03-28T05:21:08.049932+0000","last_unstale":"2024-03-28T05:21:13.876276+0000","last_undegraded":"2024-03-28T05:21:13.876276+0000","last_fullsized":"2024-03-28T05:21:13.876276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:59:03.545226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.877663+0000","last_change":"2024-03-28T05:21:08.049262+0000","last_active":"2024-03-28T05:21:13.877663+0000","last_peered":"2024-03-28T05:21:13.877663+0000","last_clean":"2024-03-28T05:21:13.877663+0000","last_became_active":"2024-03-28T05:21:08.048578+0000","last_became_peered":"2024-03-28T05:21:08.048578+0000","last_unstale":"2024-03-28T05:21:13.877663+0000","last_undegraded":"2024-03-28T05:21:13.877663+0000","last_fullsized":"2024-03-28T05:21:13.877663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:26:38.614430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.828435+0000","last_change":"2024-03-28T05:21:08.046612+0000","last_active":"2024-03-28T05:21:13.828435+0000","last_peered":"2024-03-28T05:21:13.828435+0000","last_clean":"2024-03-28T05:21:13.828435+0000","last_became_active":"2024-03-28T05:21:08.046312+0000","last_became_peered":"2024-03-28T05:21:08.046312+0000","last_unstale":"2024-03-28T05:21:13.828435+0000","last_undegraded":"2024-03-28T05:21:13.828435+0000","last_fullsized":"2024-03-28T05:21:13.828435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:20:08.379620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-28T05:21:13.876725+0000","last_change":"2024-03-28T05:21:08.045205+0000","last_active":"2024-03-28T05:21:13.876725+0000","last_peered":"2024-03-28T05:21:13.876725+0000","last_clean":"2024-03-28T05:21:13.876725+0000","last_became_active":"2024-03-28T05:21:08.044901+0000","last_became_peered":"2024-03-28T05:21:08.044901+0000","last_unstale":"2024-03-28T05:21:13.876725+0000","last_undegraded":"2024-03-28T05:21:13.876725+0000","last_fullsized":"2024-03-28T05:21:13.876725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:31:11.045766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":23939,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":14,"num_read_kb":32,"num_write":56,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":52518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383696,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645334,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1622016,"data_stored":1090918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":481893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070496,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":991232,"data_stored":477819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332133,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28572,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716580,"statfs":{"total":95995035648,"available":95965777920,"internally_reserved":0,"allocated":1581056,"data_stored":1068165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495658,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":487657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":987136,"data_stored":477305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":481091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":477319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920821,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":477237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28576,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716576,"statfs":{"total":95995035648,"available":95965773824,"internally_reserved":0,"allocated":1589248,"data_stored":1067765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":23141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":23335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T05:21:36.524 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T05:21:36.524 INFO:tasks.cephadm:Setting up client nodes... 2024-03-28T05:21:36.524 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-28T05:21:36.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:36 smithi113 ceph-mon[37164]: pgmap v302: 129 pgs: 129 active+clean; 600 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.2 KiB/s wr, 1 op/s 2024-03-28T05:21:36.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:36 smithi113 ceph-mon[37164]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:21:36.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:36 smithi113 ceph-mon[37164]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:21:36.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:36 smithi113 ceph-mon[37164]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-28T05:21:36.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:36 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1551061551' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:21:36.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:36 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1704576560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:36.804 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:21:36.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:36 smithi098 ceph-mon[36087]: pgmap v302: 129 pgs: 129 active+clean; 600 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.2 KiB/s wr, 1 op/s 2024-03-28T05:21:36.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:36 smithi098 ceph-mon[36087]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:21:36.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:36 smithi098 ceph-mon[36087]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:21:36.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:36 smithi098 ceph-mon[36087]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-28T05:21:36.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:36 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1551061551' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:21:36.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:36 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1704576560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:37.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:37 smithi023 ceph-mon[31954]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T05:21:37.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:37 smithi023 ceph-mon[31954]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T05:21:37.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:37 smithi113 ceph-mon[37164]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T05:21:37.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:37 smithi113 ceph-mon[37164]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T05:21:37.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:37 smithi098 ceph-mon[36087]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T05:21:37.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:37 smithi098 ceph-mon[36087]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T05:21:38.594 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:38 smithi023 ceph-mon[31954]: pgmap v303: 129 pgs: 129 active+clean; 600 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 1 op/s 2024-03-28T05:21:38.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.714+0000 7fbb0a5c0700 1 -- 172.21.15.23:0/934352959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb041022d0 msgr2=0x7fbb04102730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:38.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.714+0000 7fbb0a5c0700 1 --2- 172.21.15.23:0/934352959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb041022d0 0x7fbb04102730 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbaf80099c0 tx=0x7fbaf8009cd0 comp rx=0 tx=0).stop 2024-03-28T05:21:38.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.714+0000 7fbb0a5c0700 1 -- 172.21.15.23:0/934352959 shutdown_connections 2024-03-28T05:21:38.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.714+0000 7fbb0a5c0700 1 --2- 172.21.15.23:0/934352959 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb04102c70 0x7fbb04109520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:38.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.714+0000 7fbb0a5c0700 1 --2- 172.21.15.23:0/934352959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb041010d0 0x7fbb041014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:38.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.714+0000 7fbb0a5c0700 1 --2- 172.21.15.23:0/934352959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb041022d0 0x7fbb04102730 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:38.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.714+0000 7fbb0a5c0700 1 -- 172.21.15.23:0/934352959 >> 172.21.15.23:0/934352959 conn(0x7fbb040fc690 msgr2=0x7fbb040feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:38.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.714+0000 7fbb0a5c0700 1 -- 172.21.15.23:0/934352959 shutdown_connections 2024-03-28T05:21:38.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.714+0000 7fbb0a5c0700 1 -- 172.21.15.23:0/934352959 wait complete. 2024-03-28T05:21:38.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.715+0000 7fbb0a5c0700 1 Processor -- start 2024-03-28T05:21:38.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.715+0000 7fbb0a5c0700 1 -- start start 2024-03-28T05:21:38.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.715+0000 7fbb0a5c0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb041010d0 0x7fbb04105060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:38.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.715+0000 7fbb0a5c0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb041022d0 0x7fbb041036b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:38.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.715+0000 7fbb0a5c0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb04102c70 0x7fbb04103bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:38.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.715+0000 7fbb0a5c0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0410cc90 con 0x7fbb041010d0 2024-03-28T05:21:38.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.715+0000 7fbb0a5c0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0410cb10 con 0x7fbb04102c70 2024-03-28T05:21:38.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.715+0000 7fbb0a5c0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0410ce10 con 0x7fbb041022d0 2024-03-28T05:21:38.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.715+0000 7fbb03fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb041010d0 0x7fbb04105060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:38.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.715+0000 7fbb03fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb041010d0 0x7fbb04105060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53382/0 (socket says 172.21.15.23:53382) 2024-03-28T05:21:38.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.715+0000 7fbb03fff700 1 -- 172.21.15.23:0/156818824 learned_addr learned my addr 172.21.15.23:0/156818824 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:38.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.715+0000 7fbb037fe700 1 --2- 172.21.15.23:0/156818824 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb041022d0 0x7fbb041036b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:38.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.715+0000 7fbb08b5d700 1 --2- 172.21.15.23:0/156818824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb04102c70 0x7fbb04103bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:38.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.715+0000 7fbb03fff700 1 -- 172.21.15.23:0/156818824 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb041022d0 msgr2=0x7fbb041036b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:38.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.715+0000 7fbb03fff700 1 --2- 172.21.15.23:0/156818824 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb041022d0 0x7fbb041036b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:38.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.715+0000 7fbb03fff700 1 -- 172.21.15.23:0/156818824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb04102c70 msgr2=0x7fbb04103bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:38.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.715+0000 7fbb03fff700 1 --2- 172.21.15.23:0/156818824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb04102c70 0x7fbb04103bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:38.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.715+0000 7fbb03fff700 1 -- 172.21.15.23:0/156818824 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb041044a0 con 0x7fbb041010d0 2024-03-28T05:21:38.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.715+0000 7fbb08b5d700 1 --2- 172.21.15.23:0/156818824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb04102c70 0x7fbb04103bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:21:38.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.716+0000 7fbb03fff700 1 --2- 172.21.15.23:0/156818824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb041010d0 0x7fbb04105060 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fbaf400d980 tx=0x7fbaf400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:38.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.716+0000 7fbb017fa700 1 -- 172.21.15.23:0/156818824 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4017030 con 0x7fbb041010d0 2024-03-28T05:21:38.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.716+0000 7fbb0a5c0700 1 -- 172.21.15.23:0/156818824 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb041a2490 con 0x7fbb041010d0 2024-03-28T05:21:38.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.716+0000 7fbb017fa700 1 -- 172.21.15.23:0/156818824 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbaf4004500 con 0x7fbb041010d0 2024-03-28T05:21:38.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.716+0000 7fbb017fa700 1 -- 172.21.15.23:0/156818824 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4003aa0 con 0x7fbb041010d0 2024-03-28T05:21:38.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.716+0000 7fbb0a5c0700 1 -- 172.21.15.23:0/156818824 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb041a29a0 con 0x7fbb041010d0 2024-03-28T05:21:38.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.717+0000 7fbb0a5c0700 1 -- 172.21.15.23:0/156818824 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb040506a0 con 0x7fbb041010d0 2024-03-28T05:21:38.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.718+0000 7fbb017fa700 1 -- 172.21.15.23:0/156818824 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fbaf401a070 con 0x7fbb041010d0 2024-03-28T05:21:38.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.718+0000 7fbb017fa700 1 --2- 172.21.15.23:0/156818824 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fbaec0a29d0 0x7fbaec0a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:38.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.718+0000 7fbb037fe700 1 --2- 172.21.15.23:0/156818824 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fbaec0a29d0 0x7fbaec0a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:38.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.718+0000 7fbb017fa700 1 -- 172.21.15.23:0/156818824 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fbaf40ce770 con 0x7fbb041010d0 2024-03-28T05:21:38.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.719+0000 7fbb037fe700 1 --2- 172.21.15.23:0/156818824 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fbaec0a29d0 0x7fbaec0a4e90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8000c00 tx=0x7fbaf8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:38.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:38.723+0000 7fbb017fa700 1 -- 172.21.15.23:0/156818824 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbaf409b470 con 0x7fbb041010d0 2024-03-28T05:21:38.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:38 smithi113 ceph-mon[37164]: pgmap v303: 129 pgs: 129 active+clean; 600 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 1 op/s 2024-03-28T05:21:38.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:38 smithi098 ceph-mon[36087]: pgmap v303: 129 pgs: 129 active+clean; 600 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.8 KiB/s wr, 1 op/s 2024-03-28T05:21:39.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:39.014+0000 7fbb0a5c0700 1 -- 172.21.15.23:0/156818824 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fbb041a32c0 con 0x7fbb041010d0 2024-03-28T05:21:39.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:39.017+0000 7fbb017fa700 1 -- 172.21.15.23:0/156818824 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7fbaf409ad70 con 0x7fbb041010d0 2024-03-28T05:21:39.016 INFO:teuthology.orchestra.run.smithi023.stdout:[client.0] 2024-03-28T05:21:39.016 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQBj/gRmtvL8ABAAPeoG67UYmdMD8EP7R8xUpA== 2024-03-28T05:21:39.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:39.020+0000 7fbb0a5c0700 1 -- 172.21.15.23:0/156818824 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fbaec0a29d0 msgr2=0x7fbaec0a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:39.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:39.020+0000 7fbb0a5c0700 1 --2- 172.21.15.23:0/156818824 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fbaec0a29d0 0x7fbaec0a4e90 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8000c00 tx=0x7fbaf8005fb0 comp rx=0 tx=0).stop 2024-03-28T05:21:39.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:39.020+0000 7fbb0a5c0700 1 -- 172.21.15.23:0/156818824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb041010d0 msgr2=0x7fbb04105060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:39.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:39.020+0000 7fbb0a5c0700 1 --2- 172.21.15.23:0/156818824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb041010d0 0x7fbb04105060 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fbaf400d980 tx=0x7fbaf400dc90 comp rx=0 tx=0).stop 2024-03-28T05:21:39.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:39.020+0000 7fbb0a5c0700 1 -- 172.21.15.23:0/156818824 shutdown_connections 2024-03-28T05:21:39.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:39.020+0000 7fbb0a5c0700 1 --2- 172.21.15.23:0/156818824 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fbaec0a29d0 0x7fbaec0a4e90 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:39.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:39.020+0000 7fbb0a5c0700 1 --2- 172.21.15.23:0/156818824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb04102c70 0x7fbb04103bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:39.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:39.020+0000 7fbb0a5c0700 1 --2- 172.21.15.23:0/156818824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb041010d0 0x7fbb04105060 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:39.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:39.020+0000 7fbb0a5c0700 1 --2- 172.21.15.23:0/156818824 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbb041022d0 0x7fbb041036b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:39.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:39.020+0000 7fbb0a5c0700 1 -- 172.21.15.23:0/156818824 >> 172.21.15.23:0/156818824 conn(0x7fbb040fc690 msgr2=0x7fbb040fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:39.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:39.020+0000 7fbb0a5c0700 1 -- 172.21.15.23:0/156818824 shutdown_connections 2024-03-28T05:21:39.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:39.020+0000 7fbb0a5c0700 1 -- 172.21.15.23:0/156818824 wait complete. 2024-03-28T05:21:39.641 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:39 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/156818824' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T05:21:39.641 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:39 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/156818824' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T05:21:39.667 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T05:21:39.667 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-28T05:21:39.668 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-28T05:21:39.710 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-28T05:21:39.711 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-28T05:21:39.711 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-28T05:21:39.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:39 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/156818824' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T05:21:39.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:39 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/156818824' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T05:21:39.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:39 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/156818824' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T05:21:39.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:39 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/156818824' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T05:21:39.997 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:21:40.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:40 smithi113 ceph-mon[37164]: pgmap v304: 129 pgs: 129 active+clean; 600 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-28T05:21:40.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:40 smithi023 ceph-mon[31954]: pgmap v304: 129 pgs: 129 active+clean; 600 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-28T05:21:40.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:40 smithi098 ceph-mon[36087]: pgmap v304: 129 pgs: 129 active+clean; 600 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-28T05:21:42.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.018+0000 7f449ad35700 1 -- 172.21.15.23:0/4259073714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44941021e0 msgr2=0x7f4494102660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:42.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.018+0000 7f449ad35700 1 --2- 172.21.15.23:0/4259073714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44941021e0 0x7f4494102660 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f44800099c0 tx=0x7f4480009cd0 comp rx=0 tx=0).stop 2024-03-28T05:21:42.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.018+0000 7f449ad35700 1 -- 172.21.15.23:0/4259073714 shutdown_connections 2024-03-28T05:21:42.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.018+0000 7f449ad35700 1 --2- 172.21.15.23:0/4259073714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4494102ba0 0x7f449410efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:42.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.018+0000 7f449ad35700 1 --2- 172.21.15.23:0/4259073714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44941081c0 0x7f44941085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:42.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.018+0000 7f449ad35700 1 --2- 172.21.15.23:0/4259073714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44941021e0 0x7f4494102660 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:42.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.018+0000 7f449ad35700 1 -- 172.21.15.23:0/4259073714 >> 172.21.15.23:0/4259073714 conn(0x7f44940792c0 msgr2=0x7f44940796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:42.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.018+0000 7f449ad35700 1 -- 172.21.15.23:0/4259073714 shutdown_connections 2024-03-28T05:21:42.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.018+0000 7f449ad35700 1 -- 172.21.15.23:0/4259073714 wait complete. 2024-03-28T05:21:42.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.019+0000 7f449ad35700 1 Processor -- start 2024-03-28T05:21:42.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.019+0000 7f449ad35700 1 -- start start 2024-03-28T05:21:42.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.019+0000 7f449ad35700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44941021e0 0x7f449419e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:42.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.019+0000 7f449ad35700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4494102ba0 0x7f449419eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:42.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.019+0000 7f449ad35700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44941081c0 0x7f44941986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:42.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.019+0000 7f449ad35700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4494112440 con 0x7f44941081c0 2024-03-28T05:21:42.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.019+0000 7f449ad35700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44941122c0 con 0x7f44941021e0 2024-03-28T05:21:42.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.019+0000 7f449ad35700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44941125c0 con 0x7f4494102ba0 2024-03-28T05:21:42.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.019+0000 7f44992d2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44941081c0 0x7f44941986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:42.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.019+0000 7f44992d2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44941081c0 0x7f44941986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53404/0 (socket says 172.21.15.23:53404) 2024-03-28T05:21:42.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.019+0000 7f44992d2700 1 -- 172.21.15.23:0/1256894203 learned_addr learned my addr 172.21.15.23:0/1256894203 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:42.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.019+0000 7f4493fff700 1 --2- 172.21.15.23:0/1256894203 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4494102ba0 0x7f449419eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:42.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.019+0000 7f4498ad1700 1 --2- 172.21.15.23:0/1256894203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44941021e0 0x7f449419e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:42.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.020+0000 7f4493fff700 1 -- 172.21.15.23:0/1256894203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44941021e0 msgr2=0x7f449419e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:42.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.020+0000 7f4493fff700 1 --2- 172.21.15.23:0/1256894203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44941021e0 0x7f449419e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:42.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.020+0000 7f4493fff700 1 -- 172.21.15.23:0/1256894203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44941081c0 msgr2=0x7f44941986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:42.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.020+0000 7f4493fff700 1 --2- 172.21.15.23:0/1256894203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44941081c0 0x7f44941986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:42.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.020+0000 7f4493fff700 1 -- 172.21.15.23:0/1256894203 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4494198f80 con 0x7f4494102ba0 2024-03-28T05:21:42.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.020+0000 7f44992d2700 1 --2- 172.21.15.23:0/1256894203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44941081c0 0x7f44941986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:21:42.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.020+0000 7f4493fff700 1 --2- 172.21.15.23:0/1256894203 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4494102ba0 0x7f449419eb10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f44800098e0 tx=0x7f448000fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:42.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.020+0000 7f4491ffb700 1 -- 172.21.15.23:0/1256894203 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4480027070 con 0x7f4494102ba0 2024-03-28T05:21:42.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.020+0000 7f4491ffb700 1 -- 172.21.15.23:0/1256894203 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4480003680 con 0x7f4494102ba0 2024-03-28T05:21:42.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.020+0000 7f449ad35700 1 -- 172.21.15.23:0/1256894203 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4494199210 con 0x7f4494102ba0 2024-03-28T05:21:42.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.020+0000 7f4491ffb700 1 -- 172.21.15.23:0/1256894203 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44800173f0 con 0x7f4494102ba0 2024-03-28T05:21:42.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.020+0000 7f449ad35700 1 -- 172.21.15.23:0/1256894203 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44941a5380 con 0x7f4494102ba0 2024-03-28T05:21:42.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.021+0000 7f449ad35700 1 -- 172.21.15.23:0/1256894203 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4494103870 con 0x7f4494102ba0 2024-03-28T05:21:42.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.022+0000 7f4491ffb700 1 -- 172.21.15.23:0/1256894203 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f448002a070 con 0x7f4494102ba0 2024-03-28T05:21:42.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.022+0000 7f4491ffb700 1 --2- 172.21.15.23:0/1256894203 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f44840a2920 0x7f44840a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:42.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.023+0000 7f4498ad1700 1 --2- 172.21.15.23:0/1256894203 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f44840a2920 0x7f44840a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:42.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.023+0000 7f4491ffb700 1 -- 172.21.15.23:0/1256894203 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f44800cfbd0 con 0x7f4494102ba0 2024-03-28T05:21:42.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.023+0000 7f4498ad1700 1 --2- 172.21.15.23:0/1256894203 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f44840a2920 0x7f44840a4de0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4488009680 tx=0x7f4488009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:42.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.026+0000 7f4491ffb700 1 -- 172.21.15.23:0/1256894203 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4480013070 con 0x7f4494102ba0 2024-03-28T05:21:42.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.287+0000 7f449ad35700 1 -- 172.21.15.23:0/1256894203 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f449402da30 con 0x7f4494102ba0 2024-03-28T05:21:42.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.292+0000 7f4491ffb700 1 -- 172.21.15.23:0/1256894203 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+262655 (secure 0 0 0) 0x7f448009c900 con 0x7f4494102ba0 2024-03-28T05:21:42.295 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:21:42.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.295+0000 7f449ad35700 1 -- 172.21.15.23:0/1256894203 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f44840a2920 msgr2=0x7f44840a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:42.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.295+0000 7f449ad35700 1 --2- 172.21.15.23:0/1256894203 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f44840a2920 0x7f44840a4de0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4488009680 tx=0x7f4488009260 comp rx=0 tx=0).stop 2024-03-28T05:21:42.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.295+0000 7f449ad35700 1 -- 172.21.15.23:0/1256894203 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4494102ba0 msgr2=0x7f449419eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:42.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.295+0000 7f449ad35700 1 --2- 172.21.15.23:0/1256894203 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4494102ba0 0x7f449419eb10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f44800098e0 tx=0x7f448000fa00 comp rx=0 tx=0).stop 2024-03-28T05:21:42.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.295+0000 7f449ad35700 1 -- 172.21.15.23:0/1256894203 shutdown_connections 2024-03-28T05:21:42.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.295+0000 7f449ad35700 1 --2- 172.21.15.23:0/1256894203 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f44840a2920 0x7f44840a4de0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:42.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.295+0000 7f449ad35700 1 --2- 172.21.15.23:0/1256894203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44941081c0 0x7f44941986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:42.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.295+0000 7f449ad35700 1 --2- 172.21.15.23:0/1256894203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44941021e0 0x7f449419e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:42.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.295+0000 7f449ad35700 1 --2- 172.21.15.23:0/1256894203 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4494102ba0 0x7f449419eb10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:42.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.295+0000 7f449ad35700 1 -- 172.21.15.23:0/1256894203 >> 172.21.15.23:0/1256894203 conn(0x7f44940792c0 msgr2=0x7f4494100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:42.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.295+0000 7f449ad35700 1 -- 172.21.15.23:0/1256894203 shutdown_connections 2024-03-28T05:21:42.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:42.295+0000 7f449ad35700 1 -- 172.21.15.23:0/1256894203 wait complete. 2024-03-28T05:21:42.642 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:42 smithi023 ceph-mon[31954]: pgmap v305: 129 pgs: 129 active+clean; 622 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-03-28T05:21:42.642 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:42 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1256894203' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T05:21:42.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:42 smithi113 ceph-mon[37164]: pgmap v305: 129 pgs: 129 active+clean; 622 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-03-28T05:21:42.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:42 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1256894203' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T05:21:42.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:42 smithi098 ceph-mon[36087]: pgmap v305: 129 pgs: 129 active+clean; 622 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-03-28T05:21:42.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:42 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1256894203' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T05:21:42.996 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":222178108},{"type":"v1","addr":"172.21.15.23:6801","nonce":222178108}]},"active_addr":"172.21.15.23:6801/222178108","active_change":"2024-03-28T05:14:08.640447+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.23:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":84692814}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":1135235867}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":2772113246}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":944304909}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":2866536403}]}]} 2024-03-28T05:21:43.000 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-28T05:21:43.001 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-28T05:21:43.001 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T05:21:43.249 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:21:44.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:44 smithi023 ceph-mon[31954]: pgmap v306: 129 pgs: 129 active+clean; 622 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-03-28T05:21:44.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:44 smithi113 ceph-mon[37164]: pgmap v306: 129 pgs: 129 active+clean; 622 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-03-28T05:21:44.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:44 smithi098 ceph-mon[36087]: pgmap v306: 129 pgs: 129 active+clean; 622 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-03-28T05:21:45.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.019+0000 7fb7f9e2b700 1 -- 172.21.15.23:0/3602156501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7f406a7e0 msgr2=0x7fb7f410f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:45.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.019+0000 7fb7f9e2b700 1 --2- 172.21.15.23:0/3602156501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7f406a7e0 0x7fb7f410f020 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec00b0d0 tx=0x7fb7ec00b3e0 comp rx=0 tx=0).stop 2024-03-28T05:21:45.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.019+0000 7fb7f9e2b700 1 -- 172.21.15.23:0/3602156501 shutdown_connections 2024-03-28T05:21:45.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.019+0000 7fb7f9e2b700 1 --2- 172.21.15.23:0/3602156501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7f406a7e0 0x7fb7f410f020 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:45.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.019+0000 7fb7f9e2b700 1 --2- 172.21.15.23:0/3602156501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7f41081e0 0x7fb7f41085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:45.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.019+0000 7fb7f9e2b700 1 --2- 172.21.15.23:0/3602156501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f4069e40 0x7fb7f406a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:45.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.019+0000 7fb7f9e2b700 1 -- 172.21.15.23:0/3602156501 >> 172.21.15.23:0/3602156501 conn(0x7fb7f4077fc0 msgr2=0x7fb7f40783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:45.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.019+0000 7fb7f9e2b700 1 -- 172.21.15.23:0/3602156501 shutdown_connections 2024-03-28T05:21:45.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.019+0000 7fb7f9e2b700 1 -- 172.21.15.23:0/3602156501 wait complete. 2024-03-28T05:21:45.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.019+0000 7fb7f9e2b700 1 Processor -- start 2024-03-28T05:21:45.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.020+0000 7fb7f9e2b700 1 -- start start 2024-03-28T05:21:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.020+0000 7fb7f9e2b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7f4069e40 0x7fb7f419e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.020+0000 7fb7f9e2b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f406a7e0 0x7fb7f419eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.020+0000 7fb7f9e2b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7f41081e0 0x7fb7f4198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.020+0000 7fb7f9e2b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f4112460 con 0x7fb7f4069e40 2024-03-28T05:21:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.020+0000 7fb7f9e2b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f41122e0 con 0x7fb7f41081e0 2024-03-28T05:21:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.020+0000 7fb7f9e2b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f41125e0 con 0x7fb7f406a7e0 2024-03-28T05:21:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.020+0000 7fb7f37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7f4069e40 0x7fb7f419e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.020+0000 7fb7f37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7f4069e40 0x7fb7f419e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53598/0 (socket says 172.21.15.23:53598) 2024-03-28T05:21:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.020+0000 7fb7f37fe700 1 -- 172.21.15.23:0/2675337866 learned_addr learned my addr 172.21.15.23:0/2675337866 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.020+0000 7fb7f2ffd700 1 --2- 172.21.15.23:0/2675337866 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f406a7e0 0x7fb7f419eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.020+0000 7fb7f3fff700 1 --2- 172.21.15.23:0/2675337866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7f41081e0 0x7fb7f4198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.020+0000 7fb7f37fe700 1 -- 172.21.15.23:0/2675337866 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f406a7e0 msgr2=0x7fb7f419eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.020+0000 7fb7f37fe700 1 --2- 172.21.15.23:0/2675337866 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f406a7e0 0x7fb7f419eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.020+0000 7fb7f37fe700 1 -- 172.21.15.23:0/2675337866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7f41081e0 msgr2=0x7fb7f4198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:45.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.020+0000 7fb7f37fe700 1 --2- 172.21.15.23:0/2675337866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7f41081e0 0x7fb7f4198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:45.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.020+0000 7fb7f37fe700 1 -- 172.21.15.23:0/2675337866 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f4199000 con 0x7fb7f4069e40 2024-03-28T05:21:45.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.020+0000 7fb7f37fe700 1 --2- 172.21.15.23:0/2675337866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7f4069e40 0x7fb7f419e600 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400d950 tx=0x7fb7e400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:45.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.021+0000 7fb7f0ff9700 1 -- 172.21.15.23:0/2675337866 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e4016070 con 0x7fb7f4069e40 2024-03-28T05:21:45.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.021+0000 7fb7f9e2b700 1 -- 172.21.15.23:0/2675337866 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f4199290 con 0x7fb7f4069e40 2024-03-28T05:21:45.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.021+0000 7fb7f0ff9700 1 -- 172.21.15.23:0/2675337866 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7e4009e20 con 0x7fb7f4069e40 2024-03-28T05:21:45.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.021+0000 7fb7f0ff9700 1 -- 172.21.15.23:0/2675337866 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e400f620 con 0x7fb7f4069e40 2024-03-28T05:21:45.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.021+0000 7fb7f9e2b700 1 -- 172.21.15.23:0/2675337866 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f4101260 con 0x7fb7f4069e40 2024-03-28T05:21:45.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.022+0000 7fb7f9e2b700 1 -- 172.21.15.23:0/2675337866 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f4103890 con 0x7fb7f4069e40 2024-03-28T05:21:45.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.023+0000 7fb7f0ff9700 1 -- 172.21.15.23:0/2675337866 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fb7e4019070 con 0x7fb7f4069e40 2024-03-28T05:21:45.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.023+0000 7fb7f0ff9700 1 --2- 172.21.15.23:0/2675337866 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb7dc0a2920 0x7fb7dc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:45.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.023+0000 7fb7f2ffd700 1 --2- 172.21.15.23:0/2675337866 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb7dc0a2920 0x7fb7dc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:45.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.023+0000 7fb7f0ff9700 1 -- 172.21.15.23:0/2675337866 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb7e40ce9a0 con 0x7fb7f4069e40 2024-03-28T05:21:45.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.024+0000 7fb7f2ffd700 1 --2- 172.21.15.23:0/2675337866 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb7dc0a2920 0x7fb7dc0a4de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb7f4199e90 tx=0x7fb7e8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:45.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.028+0000 7fb7f0ff9700 1 -- 172.21.15.23:0/2675337866 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb7e409b6a0 con 0x7fb7f4069e40 2024-03-28T05:21:45.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.235+0000 7fb7f9e2b700 1 -- 172.21.15.23:0/2675337866 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7f4199c00 con 0x7fb7f4069e40 2024-03-28T05:21:45.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.237+0000 7fb7f0ff9700 1 -- 172.21.15.23:0/2675337866 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24082 (secure 0 0 0) 0x7fb7e409afa0 con 0x7fb7f4069e40 2024-03-28T05:21:45.236 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:21:45.236 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":89,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","created":"2024-03-28T05:13:13.453653+0000","modified":"2024-03-28T05:21:12.641185+0000","last_up_change":"2024-03-28T05:20:20.221527+0000","last_in_change":"2024-03-28T05:20:00.365478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T05:16:32.986974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T05:21:06.103894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T05:21:06.827190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T05:21:10.507418+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"18150942-59a4-4c3b-83d4-c1fc3ccf1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6803","nonce":1219956881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6805","nonce":1219956881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6809","nonce":1219956881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6807","nonce":1219956881}]},"public_addr":"172.21.15.23:6803/1219956881","cluster_addr":"172.21.15.23:6805/1219956881","heartbeat_back_addr":"172.21.15.23:6809/1219956881","heartbeat_front_addr":"172.21.15.23:6807/1219956881","state":["exists","up"]},{"osd":1,"uuid":"ba6d1a95-5135-4670-95f9-7878c0e7bd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6801","nonce":555491234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6803","nonce":555491234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6807","nonce":555491234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6805","nonce":555491234}]},"public_addr":"172.21.15.98:6801/555491234","cluster_addr":"172.21.15.98:6803/555491234","heartbeat_back_addr":"172.21.15.98:6807/555491234","heartbeat_front_addr":"172.21.15.98:6805/555491234","state":["exists","up"]},{"osd":2,"uuid":"873bdd60-2a47-403b-b1f4-ee9226a95067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6801","nonce":1535856914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6803","nonce":1535856914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6807","nonce":1535856914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6805","nonce":1535856914}]},"public_addr":"172.21.15.113:6801/1535856914","cluster_addr":"172.21.15.113:6803/1535856914","heartbeat_back_addr":"172.21.15.113:6807/1535856914","heartbeat_front_addr":"172.21.15.113:6805/1535856914","state":["exists","up"]},{"osd":3,"uuid":"83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6811","nonce":2949415020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6813","nonce":2949415020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6817","nonce":2949415020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6815","nonce":2949415020}]},"public_addr":"172.21.15.23:6811/2949415020","cluster_addr":"172.21.15.23:6813/2949415020","heartbeat_back_addr":"172.21.15.23:6817/2949415020","heartbeat_front_addr":"172.21.15.23:6815/2949415020","state":["exists","up"]},{"osd":4,"uuid":"76a90bac-b221-40a3-8238-40e8049cec36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6809","nonce":2615902819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6811","nonce":2615902819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6815","nonce":2615902819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6813","nonce":2615902819}]},"public_addr":"172.21.15.98:6809/2615902819","cluster_addr":"172.21.15.98:6811/2615902819","heartbeat_back_addr":"172.21.15.98:6815/2615902819","heartbeat_front_addr":"172.21.15.98:6813/2615902819","state":["exists","up"]},{"osd":5,"uuid":"b53fdb55-a8df-40a8-bd1b-8bcef634e26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6809","nonce":625271528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6811","nonce":625271528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6815","nonce":625271528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6813","nonce":625271528}]},"public_addr":"172.21.15.113:6809/625271528","cluster_addr":"172.21.15.113:6811/625271528","heartbeat_back_addr":"172.21.15.113:6815/625271528","heartbeat_front_addr":"172.21.15.113:6813/625271528","state":["exists","up"]},{"osd":6,"uuid":"f00283d2-8d67-4e97-a575-9a9d3b55d5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6819","nonce":4188464176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6821","nonce":4188464176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6825","nonce":4188464176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6823","nonce":4188464176}]},"public_addr":"172.21.15.23:6819/4188464176","cluster_addr":"172.21.15.23:6821/4188464176","heartbeat_back_addr":"172.21.15.23:6825/4188464176","heartbeat_front_addr":"172.21.15.23:6823/4188464176","state":["exists","up"]},{"osd":7,"uuid":"edc008a7-b07b-40a8-88b7-1614ba38a775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6817","nonce":1985546028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6819","nonce":1985546028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6823","nonce":1985546028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6821","nonce":1985546028}]},"public_addr":"172.21.15.98:6817/1985546028","cluster_addr":"172.21.15.98:6819/1985546028","heartbeat_back_addr":"172.21.15.98:6823/1985546028","heartbeat_front_addr":"172.21.15.98:6821/1985546028","state":["exists","up"]},{"osd":8,"uuid":"56e7b2d1-81f0-47b2-8638-f29832bed8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6817","nonce":3534876081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6819","nonce":3534876081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6823","nonce":3534876081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6821","nonce":3534876081}]},"public_addr":"172.21.15.113:6817/3534876081","cluster_addr":"172.21.15.113:6819/3534876081","heartbeat_back_addr":"172.21.15.113:6823/3534876081","heartbeat_front_addr":"172.21.15.113:6821/3534876081","state":["exists","up"]},{"osd":9,"uuid":"a2e90caa-6e9c-42bf-9f18-81f7f47880cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6827","nonce":672264207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6829","nonce":672264207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6833","nonce":672264207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6831","nonce":672264207}]},"public_addr":"172.21.15.23:6827/672264207","cluster_addr":"172.21.15.23:6829/672264207","heartbeat_back_addr":"172.21.15.23:6833/672264207","heartbeat_front_addr":"172.21.15.23:6831/672264207","state":["exists","up"]},{"osd":10,"uuid":"0eda917f-020c-437f-95ae-b26203f6c68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6825","nonce":2699920595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6827","nonce":2699920595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6831","nonce":2699920595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6829","nonce":2699920595}]},"public_addr":"172.21.15.98:6825/2699920595","cluster_addr":"172.21.15.98:6827/2699920595","heartbeat_back_addr":"172.21.15.98:6831/2699920595","heartbeat_front_addr":"172.21.15.98:6829/2699920595","state":["exists","up"]},{"osd":11,"uuid":"aae78501-78a8-469a-be20-3006f5ec5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6825","nonce":2113644020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6827","nonce":2113644020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6831","nonce":2113644020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6829","nonce":2113644020}]},"public_addr":"172.21.15.113:6825/2113644020","cluster_addr":"172.21.15.113:6827/2113644020","heartbeat_back_addr":"172.21.15.113:6831/2113644020","heartbeat_front_addr":"172.21.15.113:6829/2113644020","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:15:43.527475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:07.006538+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:30.594176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:55.810999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:20.595811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:45.474138+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:10.500409+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:35.787638+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:01.001188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:26.761115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:52.952752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:20:18.559832+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:6800/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1876451381":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2253559862":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2653562762":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/3337463830":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6800/1735221490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/2784771635":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2389902490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6801/1735221490":"2024-03-29T05:13:42.579607+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T05:21:45.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.239+0000 7fb7f9e2b700 1 -- 172.21.15.23:0/2675337866 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb7dc0a2920 msgr2=0x7fb7dc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:45.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.239+0000 7fb7f9e2b700 1 --2- 172.21.15.23:0/2675337866 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb7dc0a2920 0x7fb7dc0a4de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb7f4199e90 tx=0x7fb7e8009260 comp rx=0 tx=0).stop 2024-03-28T05:21:45.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.239+0000 7fb7f9e2b700 1 -- 172.21.15.23:0/2675337866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7f4069e40 msgr2=0x7fb7f419e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:45.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.239+0000 7fb7f9e2b700 1 --2- 172.21.15.23:0/2675337866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7f4069e40 0x7fb7f419e600 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400d950 tx=0x7fb7e400dc60 comp rx=0 tx=0).stop 2024-03-28T05:21:45.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.239+0000 7fb7f9e2b700 1 -- 172.21.15.23:0/2675337866 shutdown_connections 2024-03-28T05:21:45.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.239+0000 7fb7f9e2b700 1 --2- 172.21.15.23:0/2675337866 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb7dc0a2920 0x7fb7dc0a4de0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:45.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.239+0000 7fb7f9e2b700 1 --2- 172.21.15.23:0/2675337866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7f41081e0 0x7fb7f4198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:45.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.239+0000 7fb7f9e2b700 1 --2- 172.21.15.23:0/2675337866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7f4069e40 0x7fb7f419e600 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:45.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.239+0000 7fb7f9e2b700 1 --2- 172.21.15.23:0/2675337866 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f406a7e0 0x7fb7f419eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:45.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.239+0000 7fb7f9e2b700 1 -- 172.21.15.23:0/2675337866 >> 172.21.15.23:0/2675337866 conn(0x7fb7f4077fc0 msgr2=0x7fb7f40ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:45.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.239+0000 7fb7f9e2b700 1 -- 172.21.15.23:0/2675337866 shutdown_connections 2024-03-28T05:21:45.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:45.239+0000 7fb7f9e2b700 1 -- 172.21.15.23:0/2675337866 wait complete. 2024-03-28T05:21:45.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:45 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2675337866' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:45.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:45 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2675337866' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:45.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:45 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2675337866' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:45.937 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-28T05:21:45.937 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T05:21:46.186 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:21:46.635 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:46 smithi023 ceph-mon[31954]: pgmap v307: 129 pgs: 129 active+clean; 634 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.6 KiB/s wr, 5 op/s 2024-03-28T05:21:46.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:46 smithi113 ceph-mon[37164]: pgmap v307: 129 pgs: 129 active+clean; 634 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.6 KiB/s wr, 5 op/s 2024-03-28T05:21:46.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:46 smithi098 ceph-mon[36087]: pgmap v307: 129 pgs: 129 active+clean; 634 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.6 KiB/s wr, 5 op/s 2024-03-28T05:21:48.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.044+0000 7f34d40a2700 1 -- 172.21.15.23:0/1516595079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34cc104600 msgr2=0x7f34cc1049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.044+0000 7f34d40a2700 1 --2- 172.21.15.23:0/1516595079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34cc104600 0x7f34cc1049e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f34c80099c0 tx=0x7f34c8009cd0 comp rx=0 tx=0).stop 2024-03-28T05:21:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.044+0000 7f34d40a2700 1 -- 172.21.15.23:0/1516595079 shutdown_connections 2024-03-28T05:21:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.044+0000 7f34d40a2700 1 --2- 172.21.15.23:0/1516595079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34cc1096e0 0x7f34cc1112b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.044+0000 7f34d40a2700 1 --2- 172.21.15.23:0/1516595079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34cc104600 0x7f34cc1049e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.044+0000 7f34d40a2700 1 --2- 172.21.15.23:0/1516595079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34cc104fb0 0x7f34cc109000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.044+0000 7f34d40a2700 1 -- 172.21.15.23:0/1516595079 >> 172.21.15.23:0/1516595079 conn(0x7f34cc0ffe90 msgr2=0x7f34cc1022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.045+0000 7f34d40a2700 1 -- 172.21.15.23:0/1516595079 shutdown_connections 2024-03-28T05:21:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.045+0000 7f34d40a2700 1 -- 172.21.15.23:0/1516595079 wait complete. 2024-03-28T05:21:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.045+0000 7f34d40a2700 1 Processor -- start 2024-03-28T05:21:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.045+0000 7f34d40a2700 1 -- start start 2024-03-28T05:21:48.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.046+0000 7f34d40a2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34cc104600 0x7f34cc1a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:48.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.046+0000 7f34d40a2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34cc104fb0 0x7f34cc1a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:48.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.046+0000 7f34d40a2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34cc1096e0 0x7f34cc19a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:48.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.046+0000 7f34d40a2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc1146f0 con 0x7f34cc104fb0 2024-03-28T05:21:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.046+0000 7f34d40a2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc114570 con 0x7f34cc1096e0 2024-03-28T05:21:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.046+0000 7f34d40a2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc114870 con 0x7f34cc104600 2024-03-28T05:21:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.046+0000 7f34d163d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34cc104fb0 0x7f34cc1a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.046+0000 7f34d163d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34cc104fb0 0x7f34cc1a0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53614/0 (socket says 172.21.15.23:53614) 2024-03-28T05:21:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.046+0000 7f34d163d700 1 -- 172.21.15.23:0/3712403014 learned_addr learned my addr 172.21.15.23:0/3712403014 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.046+0000 7f34d1e3e700 1 --2- 172.21.15.23:0/3712403014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34cc104600 0x7f34cc1a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.046+0000 7f34d263f700 1 --2- 172.21.15.23:0/3712403014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34cc1096e0 0x7f34cc19a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.046+0000 7f34d1e3e700 1 -- 172.21.15.23:0/3712403014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34cc1096e0 msgr2=0x7f34cc19a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.046+0000 7f34d1e3e700 1 --2- 172.21.15.23:0/3712403014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34cc1096e0 0x7f34cc19a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.046+0000 7f34d1e3e700 1 -- 172.21.15.23:0/3712403014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34cc104fb0 msgr2=0x7f34cc1a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.046+0000 7f34d1e3e700 1 --2- 172.21.15.23:0/3712403014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34cc104fb0 0x7f34cc1a0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.046+0000 7f34d1e3e700 1 -- 172.21.15.23:0/3712403014 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34cc19b0f0 con 0x7f34cc104600 2024-03-28T05:21:48.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.047+0000 7f34d1e3e700 1 --2- 172.21.15.23:0/3712403014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34cc104600 0x7f34cc1a0780 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f34c80098e0 tx=0x7f34c8005f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:48.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.047+0000 7f34c2ffd700 1 -- 172.21.15.23:0/3712403014 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c8027070 con 0x7f34cc104600 2024-03-28T05:21:48.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.047+0000 7f34c2ffd700 1 -- 172.21.15.23:0/3712403014 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f34c8003680 con 0x7f34cc104600 2024-03-28T05:21:48.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.047+0000 7f34d40a2700 1 -- 172.21.15.23:0/3712403014 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34cc19b380 con 0x7f34cc104600 2024-03-28T05:21:48.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.047+0000 7f34c2ffd700 1 -- 172.21.15.23:0/3712403014 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c8022e10 con 0x7f34cc104600 2024-03-28T05:21:48.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.047+0000 7f34d40a2700 1 -- 172.21.15.23:0/3712403014 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34cc1a7650 con 0x7f34cc104600 2024-03-28T05:21:48.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.048+0000 7f34d40a2700 1 -- 172.21.15.23:0/3712403014 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34cc105ae0 con 0x7f34cc104600 2024-03-28T05:21:48.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.049+0000 7f34c2ffd700 1 -- 172.21.15.23:0/3712403014 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f34c802a070 con 0x7f34cc104600 2024-03-28T05:21:48.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.050+0000 7f34c2ffd700 1 --2- 172.21.15.23:0/3712403014 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f34b80a29a0 0x7f34b80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:48.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.050+0000 7f34d163d700 1 --2- 172.21.15.23:0/3712403014 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f34b80a29a0 0x7f34b80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:48.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.050+0000 7f34c2ffd700 1 -- 172.21.15.23:0/3712403014 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f34c80cf760 con 0x7f34cc104600 2024-03-28T05:21:48.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.050+0000 7f34d163d700 1 --2- 172.21.15.23:0/3712403014 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f34b80a29a0 0x7f34b80a4e60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f34cc10abf0 tx=0x7f34bc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:48.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.054+0000 7f34c2ffd700 1 -- 172.21.15.23:0/3712403014 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f34c8013070 con 0x7f34cc104600 2024-03-28T05:21:48.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.258+0000 7f34d40a2700 1 -- 172.21.15.23:0/3712403014 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34cc02da30 con 0x7f34cc104600 2024-03-28T05:21:48.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.260+0000 7f34c2ffd700 1 -- 172.21.15.23:0/3712403014 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24082 (secure 0 0 0) 0x7f34c809c4f0 con 0x7f34cc104600 2024-03-28T05:21:48.260 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:21:48.260 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":89,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","created":"2024-03-28T05:13:13.453653+0000","modified":"2024-03-28T05:21:12.641185+0000","last_up_change":"2024-03-28T05:20:20.221527+0000","last_in_change":"2024-03-28T05:20:00.365478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T05:16:32.986974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T05:21:06.103894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T05:21:06.827190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T05:21:10.507418+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"18150942-59a4-4c3b-83d4-c1fc3ccf1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6803","nonce":1219956881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6805","nonce":1219956881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6809","nonce":1219956881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6807","nonce":1219956881}]},"public_addr":"172.21.15.23:6803/1219956881","cluster_addr":"172.21.15.23:6805/1219956881","heartbeat_back_addr":"172.21.15.23:6809/1219956881","heartbeat_front_addr":"172.21.15.23:6807/1219956881","state":["exists","up"]},{"osd":1,"uuid":"ba6d1a95-5135-4670-95f9-7878c0e7bd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6801","nonce":555491234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6803","nonce":555491234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6807","nonce":555491234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6805","nonce":555491234}]},"public_addr":"172.21.15.98:6801/555491234","cluster_addr":"172.21.15.98:6803/555491234","heartbeat_back_addr":"172.21.15.98:6807/555491234","heartbeat_front_addr":"172.21.15.98:6805/555491234","state":["exists","up"]},{"osd":2,"uuid":"873bdd60-2a47-403b-b1f4-ee9226a95067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6801","nonce":1535856914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6803","nonce":1535856914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6807","nonce":1535856914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6805","nonce":1535856914}]},"public_addr":"172.21.15.113:6801/1535856914","cluster_addr":"172.21.15.113:6803/1535856914","heartbeat_back_addr":"172.21.15.113:6807/1535856914","heartbeat_front_addr":"172.21.15.113:6805/1535856914","state":["exists","up"]},{"osd":3,"uuid":"83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6811","nonce":2949415020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6813","nonce":2949415020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6817","nonce":2949415020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6815","nonce":2949415020}]},"public_addr":"172.21.15.23:6811/2949415020","cluster_addr":"172.21.15.23:6813/2949415020","heartbeat_back_addr":"172.21.15.23:6817/2949415020","heartbeat_front_addr":"172.21.15.23:6815/2949415020","state":["exists","up"]},{"osd":4,"uuid":"76a90bac-b221-40a3-8238-40e8049cec36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6809","nonce":2615902819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6811","nonce":2615902819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6815","nonce":2615902819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6813","nonce":2615902819}]},"public_addr":"172.21.15.98:6809/2615902819","cluster_addr":"172.21.15.98:6811/2615902819","heartbeat_back_addr":"172.21.15.98:6815/2615902819","heartbeat_front_addr":"172.21.15.98:6813/2615902819","state":["exists","up"]},{"osd":5,"uuid":"b53fdb55-a8df-40a8-bd1b-8bcef634e26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6809","nonce":625271528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6811","nonce":625271528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6815","nonce":625271528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6813","nonce":625271528}]},"public_addr":"172.21.15.113:6809/625271528","cluster_addr":"172.21.15.113:6811/625271528","heartbeat_back_addr":"172.21.15.113:6815/625271528","heartbeat_front_addr":"172.21.15.113:6813/625271528","state":["exists","up"]},{"osd":6,"uuid":"f00283d2-8d67-4e97-a575-9a9d3b55d5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6819","nonce":4188464176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6821","nonce":4188464176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6825","nonce":4188464176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6823","nonce":4188464176}]},"public_addr":"172.21.15.23:6819/4188464176","cluster_addr":"172.21.15.23:6821/4188464176","heartbeat_back_addr":"172.21.15.23:6825/4188464176","heartbeat_front_addr":"172.21.15.23:6823/4188464176","state":["exists","up"]},{"osd":7,"uuid":"edc008a7-b07b-40a8-88b7-1614ba38a775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6817","nonce":1985546028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6819","nonce":1985546028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6823","nonce":1985546028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6821","nonce":1985546028}]},"public_addr":"172.21.15.98:6817/1985546028","cluster_addr":"172.21.15.98:6819/1985546028","heartbeat_back_addr":"172.21.15.98:6823/1985546028","heartbeat_front_addr":"172.21.15.98:6821/1985546028","state":["exists","up"]},{"osd":8,"uuid":"56e7b2d1-81f0-47b2-8638-f29832bed8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6817","nonce":3534876081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6819","nonce":3534876081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6823","nonce":3534876081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6821","nonce":3534876081}]},"public_addr":"172.21.15.113:6817/3534876081","cluster_addr":"172.21.15.113:6819/3534876081","heartbeat_back_addr":"172.21.15.113:6823/3534876081","heartbeat_front_addr":"172.21.15.113:6821/3534876081","state":["exists","up"]},{"osd":9,"uuid":"a2e90caa-6e9c-42bf-9f18-81f7f47880cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6827","nonce":672264207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6829","nonce":672264207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6833","nonce":672264207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6831","nonce":672264207}]},"public_addr":"172.21.15.23:6827/672264207","cluster_addr":"172.21.15.23:6829/672264207","heartbeat_back_addr":"172.21.15.23:6833/672264207","heartbeat_front_addr":"172.21.15.23:6831/672264207","state":["exists","up"]},{"osd":10,"uuid":"0eda917f-020c-437f-95ae-b26203f6c68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6825","nonce":2699920595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6827","nonce":2699920595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6831","nonce":2699920595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6829","nonce":2699920595}]},"public_addr":"172.21.15.98:6825/2699920595","cluster_addr":"172.21.15.98:6827/2699920595","heartbeat_back_addr":"172.21.15.98:6831/2699920595","heartbeat_front_addr":"172.21.15.98:6829/2699920595","state":["exists","up"]},{"osd":11,"uuid":"aae78501-78a8-469a-be20-3006f5ec5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6825","nonce":2113644020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6827","nonce":2113644020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6831","nonce":2113644020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6829","nonce":2113644020}]},"public_addr":"172.21.15.113:6825/2113644020","cluster_addr":"172.21.15.113:6827/2113644020","heartbeat_back_addr":"172.21.15.113:6831/2113644020","heartbeat_front_addr":"172.21.15.113:6829/2113644020","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:15:43.527475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:07.006538+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:30.594176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:55.810999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:20.595811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:45.474138+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:10.500409+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:35.787638+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:01.001188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:26.761115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:52.952752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:20:18.559832+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:6800/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1876451381":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2253559862":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2653562762":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/3337463830":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6800/1735221490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/2784771635":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2389902490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6801/1735221490":"2024-03-29T05:13:42.579607+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T05:21:48.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.262+0000 7f34d40a2700 1 -- 172.21.15.23:0/3712403014 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f34b80a29a0 msgr2=0x7f34b80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:48.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.262+0000 7f34d40a2700 1 --2- 172.21.15.23:0/3712403014 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f34b80a29a0 0x7f34b80a4e60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f34cc10abf0 tx=0x7f34bc009260 comp rx=0 tx=0).stop 2024-03-28T05:21:48.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.263+0000 7f34d40a2700 1 -- 172.21.15.23:0/3712403014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34cc104600 msgr2=0x7f34cc1a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:48.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.263+0000 7f34d40a2700 1 --2- 172.21.15.23:0/3712403014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34cc104600 0x7f34cc1a0780 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f34c80098e0 tx=0x7f34c8005f10 comp rx=0 tx=0).stop 2024-03-28T05:21:48.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.263+0000 7f34d40a2700 1 -- 172.21.15.23:0/3712403014 shutdown_connections 2024-03-28T05:21:48.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.263+0000 7f34d40a2700 1 --2- 172.21.15.23:0/3712403014 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f34b80a29a0 0x7f34b80a4e60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:48.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.263+0000 7f34d40a2700 1 --2- 172.21.15.23:0/3712403014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34cc1096e0 0x7f34cc19a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:48.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.263+0000 7f34d40a2700 1 --2- 172.21.15.23:0/3712403014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f34cc104600 0x7f34cc1a0780 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:48.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.263+0000 7f34d40a2700 1 --2- 172.21.15.23:0/3712403014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34cc104fb0 0x7f34cc1a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:48.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.263+0000 7f34d40a2700 1 -- 172.21.15.23:0/3712403014 >> 172.21.15.23:0/3712403014 conn(0x7f34cc0ffe90 msgr2=0x7f34cc102190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:48.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.263+0000 7f34d40a2700 1 -- 172.21.15.23:0/3712403014 shutdown_connections 2024-03-28T05:21:48.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:48.263+0000 7f34d40a2700 1 -- 172.21.15.23:0/3712403014 wait complete. 2024-03-28T05:21:48.633 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:48 smithi023 ceph-mon[31954]: pgmap v308: 129 pgs: 129 active+clean; 634 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 3.7 KiB/s wr, 3 op/s 2024-03-28T05:21:48.633 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:48 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3712403014' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:48.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:48 smithi113 ceph-mon[37164]: pgmap v308: 129 pgs: 129 active+clean; 634 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 3.7 KiB/s wr, 3 op/s 2024-03-28T05:21:48.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:48 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3712403014' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:48.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:48 smithi098 ceph-mon[36087]: pgmap v308: 129 pgs: 129 active+clean; 634 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 3.7 KiB/s wr, 3 op/s 2024-03-28T05:21:48.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:48 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3712403014' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:21:48.956 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-28T05:21:48.956 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-28T05:21:48.957 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-28T05:21:48.957 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-28T05:21:48.958 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-28T05:21:48.958 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-28T05:21:48.958 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-28T05:21:48.959 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-28T05:21:48.959 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-28T05:21:48.960 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-28T05:21:48.960 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-28T05:21:48.961 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-28T05:21:49.740 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:21:49.742 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:21:49.742 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:21:49.745 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:21:49.758 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:21:49.762 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:21:49.766 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:21:49.766 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:21:49.770 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:21:49.772 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:21:49.783 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:21:49.830 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:21:50.606 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:50 smithi023 ceph-mon[31954]: pgmap v309: 129 pgs: 129 active+clean; 634 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 3.7 KiB/s wr, 3 op/s 2024-03-28T05:21:50.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:50 smithi113 ceph-mon[37164]: pgmap v309: 129 pgs: 129 active+clean; 634 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 3.7 KiB/s wr, 3 op/s 2024-03-28T05:21:50.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:50 smithi098 ceph-mon[36087]: pgmap v309: 129 pgs: 129 active+clean; 634 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 3.7 KiB/s wr, 3 op/s 2024-03-28T05:21:52.734 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:52 smithi023 ceph-mon[31954]: pgmap v310: 129 pgs: 129 active+clean; 638 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 4.0 KiB/s wr, 4 op/s 2024-03-28T05:21:52.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:52 smithi113 ceph-mon[37164]: pgmap v310: 129 pgs: 129 active+clean; 638 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 4.0 KiB/s wr, 4 op/s 2024-03-28T05:21:52.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:52 smithi098 ceph-mon[36087]: pgmap v310: 129 pgs: 129 active+clean; 638 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 4.0 KiB/s wr, 4 op/s 2024-03-28T05:21:54.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:54 smithi113 ceph-mon[37164]: pgmap v311: 129 pgs: 129 active+clean; 638 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-28T05:21:54.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:54 smithi023 ceph-mon[31954]: pgmap v311: 129 pgs: 129 active+clean; 638 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-28T05:21:54.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:54 smithi098 ceph-mon[36087]: pgmap v311: 129 pgs: 129 active+clean; 638 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-28T05:21:56.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.119+0000 7f1fdca94700 1 -- 172.21.15.23:0/906920286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1fd8100780 msgr2=0x7f1fd8100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:56.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.119+0000 7f1fdca94700 1 --2- 172.21.15.23:0/906920286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1fd8100780 0x7f1fd8100c00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc009990 tx=0x7f1fcc009ca0 comp rx=0 tx=0).stop 2024-03-28T05:21:56.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.119+0000 7f1fdca94700 1 -- 172.21.15.23:0/906920286 shutdown_connections 2024-03-28T05:21:56.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.119+0000 7f1fdca94700 1 --2- 172.21.15.23:0/906920286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fd8101140 0x7f1fd810f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:56.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.119+0000 7f1fdca94700 1 --2- 172.21.15.23:0/906920286 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fd8107200 0x7f1fd81075e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:56.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.119+0000 7f1fdca94700 1 --2- 172.21.15.23:0/906920286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1fd8100780 0x7f1fd8100c00 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:56.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.119+0000 7f1fdca94700 1 -- 172.21.15.23:0/906920286 >> 172.21.15.23:0/906920286 conn(0x7f1fd8077690 msgr2=0x7f1fd80feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:56.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.119+0000 7f1fdca94700 1 -- 172.21.15.23:0/906920286 shutdown_connections 2024-03-28T05:21:56.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.119+0000 7f1fdca94700 1 -- 172.21.15.23:0/906920286 wait complete. 2024-03-28T05:21:56.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.120+0000 7f1fdca94700 1 Processor -- start 2024-03-28T05:21:56.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.120+0000 7f1fdca94700 1 -- start start 2024-03-28T05:21:56.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.120+0000 7f1fdca94700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1fd8100780 0x7f1fd81a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:56.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.121+0000 7f1fdca94700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fd8101140 0x7f1fd81a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:56.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.121+0000 7f1fdca94700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fd8107200 0x7f1fd819aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.121+0000 7f1fdca94700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd81125e0 con 0x7f1fd8100780 2024-03-28T05:21:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.121+0000 7f1fdca94700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd8112460 con 0x7f1fd8101140 2024-03-28T05:21:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.121+0000 7f1fdca94700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd8112760 con 0x7f1fd8107200 2024-03-28T05:21:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.121+0000 7f1fd6d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fd8107200 0x7f1fd819aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.121+0000 7f1fd6d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fd8107200 0x7f1fd819aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:49814/0 (socket says 172.21.15.23:49814) 2024-03-28T05:21:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.121+0000 7f1fd6d9d700 1 -- 172.21.15.23:0/2903802613 learned_addr learned my addr 172.21.15.23:0/2903802613 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:21:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.121+0000 7f1fd659c700 1 --2- 172.21.15.23:0/2903802613 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1fd8100780 0x7f1fd81a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.121+0000 7f1fd6d9d700 1 -- 172.21.15.23:0/2903802613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fd8101140 msgr2=0x7f1fd81a0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.121+0000 7f1fd6d9d700 1 --2- 172.21.15.23:0/2903802613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1fd8101140 0x7f1fd81a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.121+0000 7f1fd6d9d700 1 -- 172.21.15.23:0/2903802613 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1fd8100780 msgr2=0x7f1fd81a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.121+0000 7f1fd6d9d700 1 --2- 172.21.15.23:0/2903802613 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1fd8100780 0x7f1fd81a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.121+0000 7f1fd6d9d700 1 -- 172.21.15.23:0/2903802613 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd819b240 con 0x7f1fd8107200 2024-03-28T05:21:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.121+0000 7f1fd659c700 1 --2- 172.21.15.23:0/2903802613 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1fd8100780 0x7f1fd81a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:21:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.122+0000 7f1fd6d9d700 1 --2- 172.21.15.23:0/2903802613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fd8107200 0x7f1fd819aae0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000ef90 tx=0x7f1fd000c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.122+0000 7f1fc77fe700 1 -- 172.21.15.23:0/2903802613 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd00092e0 con 0x7f1fd8107200 2024-03-28T05:21:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.122+0000 7f1fc77fe700 1 -- 172.21.15.23:0/2903802613 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1fd0004bb0 con 0x7f1fd8107200 2024-03-28T05:21:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.122+0000 7f1fdca94700 1 -- 172.21.15.23:0/2903802613 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd819b4d0 con 0x7f1fd8107200 2024-03-28T05:21:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.122+0000 7f1fc77fe700 1 -- 172.21.15.23:0/2903802613 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0004280 con 0x7f1fd8107200 2024-03-28T05:21:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.122+0000 7f1fdca94700 1 -- 172.21.15.23:0/2903802613 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd81a77f0 con 0x7f1fd8107200 2024-03-28T05:21:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.123+0000 7f1fdca94700 1 -- 172.21.15.23:0/2903802613 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1fd810e740 con 0x7f1fd8107200 2024-03-28T05:21:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.124+0000 7f1fc77fe700 1 -- 172.21.15.23:0/2903802613 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f1fd001b030 con 0x7f1fd8107200 2024-03-28T05:21:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.124+0000 7f1fc77fe700 1 --2- 172.21.15.23:0/2903802613 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1fc00a29a0 0x7f1fc00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.125+0000 7f1fd659c700 1 --2- 172.21.15.23:0/2903802613 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1fc00a29a0 0x7f1fc00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.125+0000 7f1fc77fe700 1 -- 172.21.15.23:0/2903802613 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f1fd00ceb60 con 0x7f1fd8107200 2024-03-28T05:21:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.125+0000 7f1fc77fe700 1 --2- 172.21.15.23:0/2903802613 >> [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020] conn(0x7f1fc00ad660 0x7f1fc00afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:21:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.125+0000 7f1fc77fe700 1 -- 172.21.15.23:0/2903802613 --> [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1fc00b0170 con 0x7f1fc00ad660 2024-03-28T05:21:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.125+0000 7f1fc77fe700 1 -- 172.21.15.23:0/2903802613 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1fd00cef10 con 0x7f1fd8107200 2024-03-28T05:21:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.125+0000 7f1fd5d9b700 1 --2- 172.21.15.23:0/2903802613 >> [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020] conn(0x7f1fc00ad660 0x7f1fc00afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:21:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.125+0000 7f1fd659c700 1 --2- 172.21.15.23:0/2903802613 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1fc00a29a0 0x7f1fc00a4e60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8005fd0 tx=0x7f1fc8005db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.125+0000 7f1fd5d9b700 1 --2- 172.21.15.23:0/2903802613 >> [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020] conn(0x7f1fc00ad660 0x7f1fc00afaa0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:21:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.126+0000 7f1fc77fe700 1 -- 172.21.15.23:0/2903802613 <== osd.3 v2:172.21.15.23:6810/2949415020 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1fc00b0170 con 0x7f1fc00ad660 2024-03-28T05:21:56.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.145+0000 7f1fdca94700 1 -- 172.21.15.23:0/2903802613 --> [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1fd8103aa0 con 0x7f1fc00ad660 2024-03-28T05:21:56.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.145+0000 7f1fc77fe700 1 -- 172.21.15.23:0/2903802613 <== osd.3 v2:172.21.15.23:6810/2949415020 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1fd8103aa0 con 0x7f1fc00ad660 2024-03-28T05:21:56.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.146+0000 7f1fdca94700 1 -- 172.21.15.23:0/2903802613 >> [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020] conn(0x7f1fc00ad660 msgr2=0x7f1fc00afaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:56.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.146+0000 7f1fdca94700 1 --2- 172.21.15.23:0/2903802613 >> [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020] conn(0x7f1fc00ad660 0x7f1fc00afaa0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:21:56.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.146+0000 7f1fdca94700 1 -- 172.21.15.23:0/2903802613 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1fc00a29a0 msgr2=0x7f1fc00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:56.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.146+0000 7f1fdca94700 1 --2- 172.21.15.23:0/2903802613 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1fc00a29a0 0x7f1fc00a4e60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8005fd0 tx=0x7f1fc8005db0 comp rx=0 tx=0).stop 2024-03-28T05:21:56.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.146+0000 7f1fdca94700 1 -- 172.21.15.23:0/2903802613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fd8107200 msgr2=0x7f1fd819aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:21:56.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.146+0000 7f1fdca94700 1 --2- 172.21.15.23:0/2903802613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1fd8107200 0x7f1fd819aae0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000ef90 tx=0x7f1fd000c450 comp rx=0 tx=0).stop 2024-03-28T05:21:56.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.146+0000 7f1fd6d9d700 1 -- 172.21.15.23:0/2903802613 reap_dead start 2024-03-28T05:21:56.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.146+0000 7f1fdca94700 1 -- 172.21.15.23:0/2903802613 shutdown_connections 2024-03-28T05:21:56.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.146+0000 7f1fdca94700 1 -- 172.21.15.23:0/2903802613 >> 172.21.15.23:0/2903802613 conn(0x7f1fd8077690 msgr2=0x7f1fd8105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:21:56.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.146+0000 7f1fdca94700 1 -- 172.21.15.23:0/2903802613 shutdown_connections 2024-03-28T05:21:56.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:21:56.146+0000 7f1fdca94700 1 -- 172.21.15.23:0/2903802613 wait complete. 2024-03-28T05:21:56.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:56 smithi113 ceph-mon[37164]: pgmap v312: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.9 KiB/s wr, 3 op/s 2024-03-28T05:21:56.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:56 smithi023 ceph-mon[31954]: pgmap v312: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.9 KiB/s wr, 3 op/s 2024-03-28T05:21:56.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:56 smithi098 ceph-mon[36087]: pgmap v312: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.9 KiB/s wr, 3 op/s 2024-03-28T05:21:58.650 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:21:58 smithi023 ceph-mon[31954]: pgmap v313: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 597 B/s wr, 1 op/s 2024-03-28T05:21:58.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:21:58 smithi113 ceph-mon[37164]: pgmap v313: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 597 B/s wr, 1 op/s 2024-03-28T05:21:58.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:21:58 smithi098 ceph-mon[36087]: pgmap v313: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 597 B/s wr, 1 op/s 2024-03-28T05:22:00.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:00 smithi023 ceph-mon[31954]: pgmap v314: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 597 B/s wr, 1 op/s 2024-03-28T05:22:00.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:00 smithi113 ceph-mon[37164]: pgmap v314: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 597 B/s wr, 1 op/s 2024-03-28T05:22:00.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:00 smithi098 ceph-mon[36087]: pgmap v314: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 597 B/s wr, 1 op/s 2024-03-28T05:22:01.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.636+0000 7fe015807700 1 -- 172.21.15.23:0/3170049136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe010101a30 msgr2=0x7fe01010df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:01.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.636+0000 7fe015807700 1 --2- 172.21.15.23:0/3170049136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe010101a30 0x7fe01010df10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe00800b0a0 tx=0x7fe00800b3b0 comp rx=0 tx=0).stop 2024-03-28T05:22:01.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.636+0000 7fe015807700 1 -- 172.21.15.23:0/3170049136 shutdown_connections 2024-03-28T05:22:01.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.636+0000 7fe015807700 1 --2- 172.21.15.23:0/3170049136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe010101a30 0x7fe01010df10 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:01.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.636+0000 7fe015807700 1 --2- 172.21.15.23:0/3170049136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0101070d0 0x7fe0101074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:01.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.636+0000 7fe015807700 1 --2- 172.21.15.23:0/3170049136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe010101090 0x7fe0101014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:01.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.636+0000 7fe015807700 1 -- 172.21.15.23:0/3170049136 >> 172.21.15.23:0/3170049136 conn(0x7fe010077fc0 msgr2=0x7fe0100783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:01.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.636+0000 7fe015807700 1 -- 172.21.15.23:0/3170049136 shutdown_connections 2024-03-28T05:22:01.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.636+0000 7fe015807700 1 -- 172.21.15.23:0/3170049136 wait complete. 2024-03-28T05:22:01.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.637+0000 7fe015807700 1 Processor -- start 2024-03-28T05:22:01.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.637+0000 7fe015807700 1 -- start start 2024-03-28T05:22:01.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.637+0000 7fe015807700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe010101090 0x7fe01019e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:01.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.637+0000 7fe015807700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe010101a30 0x7fe01019eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:01.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.637+0000 7fe015807700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0101070d0 0x7fe0101986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:01.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.637+0000 7fe015807700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe010111350 con 0x7fe0101070d0 2024-03-28T05:22:01.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.637+0000 7fe015807700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0101111d0 con 0x7fe010101090 2024-03-28T05:22:01.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.637+0000 7fe015807700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0101114d0 con 0x7fe010101a30 2024-03-28T05:22:01.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.637+0000 7fe00f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0101070d0 0x7fe0101986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:01.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.637+0000 7fe00f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0101070d0 0x7fe0101986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34020/0 (socket says 172.21.15.23:34020) 2024-03-28T05:22:01.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.637+0000 7fe00f7fe700 1 -- 172.21.15.23:0/483857030 learned_addr learned my addr 172.21.15.23:0/483857030 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:01.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.637+0000 7fe00e7fc700 1 --2- 172.21.15.23:0/483857030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe010101a30 0x7fe01019eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:01.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.637+0000 7fe00effd700 1 --2- 172.21.15.23:0/483857030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe010101090 0x7fe01019e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:01.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.637+0000 7fe00f7fe700 1 -- 172.21.15.23:0/483857030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe010101a30 msgr2=0x7fe01019eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:01.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.637+0000 7fe00f7fe700 1 --2- 172.21.15.23:0/483857030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe010101a30 0x7fe01019eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:01.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.637+0000 7fe00f7fe700 1 -- 172.21.15.23:0/483857030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe010101090 msgr2=0x7fe01019e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:01.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.638+0000 7fe00f7fe700 1 --2- 172.21.15.23:0/483857030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe010101090 0x7fe01019e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:01.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.638+0000 7fe00f7fe700 1 -- 172.21.15.23:0/483857030 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe010198f90 con 0x7fe0101070d0 2024-03-28T05:22:01.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.638+0000 7fe00effd700 1 --2- 172.21.15.23:0/483857030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe010101090 0x7fe01019e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:22:01.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.638+0000 7fe00e7fc700 1 --2- 172.21.15.23:0/483857030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe010101a30 0x7fe01019eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:22:01.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.638+0000 7fe00f7fe700 1 --2- 172.21.15.23:0/483857030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0101070d0 0x7fe0101986b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe00800b780 tx=0x7fe008004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:01.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.638+0000 7fe014805700 1 -- 172.21.15.23:0/483857030 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe008021070 con 0x7fe0101070d0 2024-03-28T05:22:01.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.638+0000 7fe015807700 1 -- 172.21.15.23:0/483857030 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe010199220 con 0x7fe0101070d0 2024-03-28T05:22:01.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.638+0000 7fe014805700 1 -- 172.21.15.23:0/483857030 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe008007b80 con 0x7fe0101070d0 2024-03-28T05:22:01.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.638+0000 7fe014805700 1 -- 172.21.15.23:0/483857030 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0080052a0 con 0x7fe0101070d0 2024-03-28T05:22:01.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.638+0000 7fe015807700 1 -- 172.21.15.23:0/483857030 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0101a5560 con 0x7fe0101070d0 2024-03-28T05:22:01.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.639+0000 7fe015807700 1 -- 172.21.15.23:0/483857030 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe010105190 con 0x7fe0101070d0 2024-03-28T05:22:01.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.640+0000 7fe014805700 1 -- 172.21.15.23:0/483857030 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fe008024070 con 0x7fe0101070d0 2024-03-28T05:22:01.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.640+0000 7fe014805700 1 --2- 172.21.15.23:0/483857030 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fdffc0a2920 0x7fdffc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:01.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.641+0000 7fe00effd700 1 --2- 172.21.15.23:0/483857030 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fdffc0a2920 0x7fdffc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:01.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.641+0000 7fe014805700 1 -- 172.21.15.23:0/483857030 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fe0080d02b0 con 0x7fe0101070d0 2024-03-28T05:22:01.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.641+0000 7fe014805700 1 --2- 172.21.15.23:0/483857030 >> [v2:172.21.15.113:6808/625271528,v1:172.21.15.113:6809/625271528] conn(0x7fdffc0ad4b0 0x7fdffc0af8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:01.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.641+0000 7fe014805700 1 -- 172.21.15.23:0/483857030 --> [v2:172.21.15.113:6808/625271528,v1:172.21.15.113:6809/625271528] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe0080020b0 con 0x7fdffc0ad4b0 2024-03-28T05:22:01.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.641+0000 7fe014805700 1 -- 172.21.15.23:0/483857030 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe0080d0660 con 0x7fe0101070d0 2024-03-28T05:22:01.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.641+0000 7fe00effd700 1 --2- 172.21.15.23:0/483857030 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fdffc0a2920 0x7fdffc0a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdff8009c90 tx=0x7fdff8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:01.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.641+0000 7fe00e7fc700 1 --2- 172.21.15.23:0/483857030 >> [v2:172.21.15.113:6808/625271528,v1:172.21.15.113:6809/625271528] conn(0x7fdffc0ad4b0 0x7fdffc0af8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:01.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.641+0000 7fe00e7fc700 1 --2- 172.21.15.23:0/483857030 >> [v2:172.21.15.113:6808/625271528,v1:172.21.15.113:6809/625271528] conn(0x7fdffc0ad4b0 0x7fdffc0af8f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:01.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.642+0000 7fe014805700 1 -- 172.21.15.23:0/483857030 <== osd.5 v2:172.21.15.113:6808/625271528 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe0080020b0 con 0x7fdffc0ad4b0 2024-03-28T05:22:01.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.658+0000 7fe015807700 1 -- 172.21.15.23:0/483857030 --> [v2:172.21.15.113:6808/625271528,v1:172.21.15.113:6809/625271528] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe010102780 con 0x7fdffc0ad4b0 2024-03-28T05:22:01.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.658+0000 7fe014805700 1 -- 172.21.15.23:0/483857030 <== osd.5 v2:172.21.15.113:6808/625271528 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe010102780 con 0x7fdffc0ad4b0 2024-03-28T05:22:01.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.659+0000 7fe015807700 1 -- 172.21.15.23:0/483857030 >> [v2:172.21.15.113:6808/625271528,v1:172.21.15.113:6809/625271528] conn(0x7fdffc0ad4b0 msgr2=0x7fdffc0af8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:01.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.659+0000 7fe015807700 1 --2- 172.21.15.23:0/483857030 >> [v2:172.21.15.113:6808/625271528,v1:172.21.15.113:6809/625271528] conn(0x7fdffc0ad4b0 0x7fdffc0af8f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:01.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.659+0000 7fe015807700 1 -- 172.21.15.23:0/483857030 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fdffc0a2920 msgr2=0x7fdffc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:01.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.659+0000 7fe015807700 1 --2- 172.21.15.23:0/483857030 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fdffc0a2920 0x7fdffc0a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdff8009c90 tx=0x7fdff8009260 comp rx=0 tx=0).stop 2024-03-28T05:22:01.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.659+0000 7fe015807700 1 -- 172.21.15.23:0/483857030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0101070d0 msgr2=0x7fe0101986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:01.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.659+0000 7fe015807700 1 --2- 172.21.15.23:0/483857030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0101070d0 0x7fe0101986b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe00800b780 tx=0x7fe008004d00 comp rx=0 tx=0).stop 2024-03-28T05:22:01.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.659+0000 7fe00f7fe700 1 -- 172.21.15.23:0/483857030 reap_dead start 2024-03-28T05:22:01.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.659+0000 7fe015807700 1 -- 172.21.15.23:0/483857030 shutdown_connections 2024-03-28T05:22:01.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.659+0000 7fe015807700 1 -- 172.21.15.23:0/483857030 >> 172.21.15.23:0/483857030 conn(0x7fe010077fc0 msgr2=0x7fe010100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:01.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.659+0000 7fe015807700 1 -- 172.21.15.23:0/483857030 shutdown_connections 2024-03-28T05:22:01.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:01.659+0000 7fe015807700 1 -- 172.21.15.23:0/483857030 wait complete. 2024-03-28T05:22:02.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:02 smithi113 ceph-mon[37164]: pgmap v315: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 597 B/s wr, 2 op/s 2024-03-28T05:22:02.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:02 smithi023 ceph-mon[31954]: pgmap v315: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 597 B/s wr, 2 op/s 2024-03-28T05:22:02.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:02 smithi098 ceph-mon[36087]: pgmap v315: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 597 B/s wr, 2 op/s 2024-03-28T05:22:03.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.548+0000 7fa0aeeff700 1 -- 172.21.15.23:0/1053114922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0a8107120 msgr2=0x7fa0a8107500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:03.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.548+0000 7fa0aeeff700 1 --2- 172.21.15.23:0/1053114922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0a8107120 0x7fa0a8107500 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa098009990 tx=0x7fa098009ca0 comp rx=0 tx=0).stop 2024-03-28T05:22:03.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.548+0000 7fa0aeeff700 1 -- 172.21.15.23:0/1053114922 shutdown_connections 2024-03-28T05:22:03.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.548+0000 7fa0aeeff700 1 --2- 172.21.15.23:0/1053114922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0a8101a30 0x7fa0a810ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:03.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.548+0000 7fa0aeeff700 1 --2- 172.21.15.23:0/1053114922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0a8107120 0x7fa0a8107500 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:03.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.548+0000 7fa0aeeff700 1 --2- 172.21.15.23:0/1053114922 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0a8101070 0x7fa0a81014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:03.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.548+0000 7fa0aeeff700 1 -- 172.21.15.23:0/1053114922 >> 172.21.15.23:0/1053114922 conn(0x7fa0a80780a0 msgr2=0x7fa0a80784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:03.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.548+0000 7fa0aeeff700 1 -- 172.21.15.23:0/1053114922 shutdown_connections 2024-03-28T05:22:03.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.548+0000 7fa0aeeff700 1 -- 172.21.15.23:0/1053114922 wait complete. 2024-03-28T05:22:03.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.548+0000 7fa0aeeff700 1 Processor -- start 2024-03-28T05:22:03.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.549+0000 7fa0aeeff700 1 -- start start 2024-03-28T05:22:03.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.549+0000 7fa0aeeff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0a8101070 0x7fa0a819e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:03.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.549+0000 7fa0aeeff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0a8101a30 0x7fa0a819e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:03.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.549+0000 7fa0aeeff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0a8107120 0x7fa0a8198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:03.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.549+0000 7fa0aeeff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a8111330 con 0x7fa0a8101070 2024-03-28T05:22:03.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.549+0000 7fa0aeeff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a81111b0 con 0x7fa0a8101a30 2024-03-28T05:22:03.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.549+0000 7fa0aeeff700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a81114b0 con 0x7fa0a8107120 2024-03-28T05:22:03.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.549+0000 7fa0acc9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0a8101070 0x7fa0a819e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:03.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.549+0000 7fa0acc9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0a8101070 0x7fa0a819e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36086/0 (socket says 172.21.15.23:36086) 2024-03-28T05:22:03.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.549+0000 7fa0acc9b700 1 -- 172.21.15.23:0/1410832410 learned_addr learned my addr 172.21.15.23:0/1410832410 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:03.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.549+0000 7fa0ad49c700 1 --2- 172.21.15.23:0/1410832410 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0a8107120 0x7fa0a8198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:03.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.549+0000 7fa0a7fff700 1 --2- 172.21.15.23:0/1410832410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0a8101a30 0x7fa0a819e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:03.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.549+0000 7fa0acc9b700 1 -- 172.21.15.23:0/1410832410 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0a8107120 msgr2=0x7fa0a8198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:03.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.549+0000 7fa0acc9b700 1 --2- 172.21.15.23:0/1410832410 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0a8107120 0x7fa0a8198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:03.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.549+0000 7fa0acc9b700 1 -- 172.21.15.23:0/1410832410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0a8101a30 msgr2=0x7fa0a819e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:03.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.549+0000 7fa0acc9b700 1 --2- 172.21.15.23:0/1410832410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa0a8101a30 0x7fa0a819e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:03.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.549+0000 7fa0acc9b700 1 -- 172.21.15.23:0/1410832410 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a8198d80 con 0x7fa0a8101070 2024-03-28T05:22:03.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.549+0000 7fa0acc9b700 1 --2- 172.21.15.23:0/1410832410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0a8101070 0x7fa0a819e460 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa09800b2e0 tx=0x7fa09800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:03.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.550+0000 7fa0a5ffb700 1 -- 172.21.15.23:0/1410832410 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098026070 con 0x7fa0a8101070 2024-03-28T05:22:03.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.550+0000 7fa0a5ffb700 1 -- 172.21.15.23:0/1410832410 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa098021410 con 0x7fa0a8101070 2024-03-28T05:22:03.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.550+0000 7fa0aeeff700 1 -- 172.21.15.23:0/1410832410 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a8199010 con 0x7fa0a8101070 2024-03-28T05:22:03.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.550+0000 7fa0a5ffb700 1 -- 172.21.15.23:0/1410832410 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098017650 con 0x7fa0a8101070 2024-03-28T05:22:03.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.550+0000 7fa0aeeff700 1 -- 172.21.15.23:0/1410832410 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a81a5330 con 0x7fa0a8101070 2024-03-28T05:22:03.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.551+0000 7fa0aeeff700 1 -- 172.21.15.23:0/1410832410 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa0a81051b0 con 0x7fa0a8101070 2024-03-28T05:22:03.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.552+0000 7fa0a5ffb700 1 -- 172.21.15.23:0/1410832410 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fa098029070 con 0x7fa0a8101070 2024-03-28T05:22:03.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.552+0000 7fa0a5ffb700 1 --2- 172.21.15.23:0/1410832410 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa0900a2920 0x7fa0900a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:03.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.553+0000 7fa0a7fff700 1 --2- 172.21.15.23:0/1410832410 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa0900a2920 0x7fa0900a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:03.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.553+0000 7fa0a5ffb700 1 -- 172.21.15.23:0/1410832410 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fa0980d03b0 con 0x7fa0a8101070 2024-03-28T05:22:03.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.553+0000 7fa0a5ffb700 1 --2- 172.21.15.23:0/1410832410 >> [v2:172.21.15.113:6824/2113644020,v1:172.21.15.113:6825/2113644020] conn(0x7fa0900ad560 0x7fa0900af9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:03.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.553+0000 7fa0a5ffb700 1 -- 172.21.15.23:0/1410832410 --> [v2:172.21.15.113:6824/2113644020,v1:172.21.15.113:6825/2113644020] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa098018e50 con 0x7fa0900ad560 2024-03-28T05:22:03.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.553+0000 7fa0a5ffb700 1 -- 172.21.15.23:0/1410832410 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0980d0800 con 0x7fa0a8101070 2024-03-28T05:22:03.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.553+0000 7fa0ad49c700 1 --2- 172.21.15.23:0/1410832410 >> [v2:172.21.15.113:6824/2113644020,v1:172.21.15.113:6825/2113644020] conn(0x7fa0900ad560 0x7fa0900af9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:03.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.553+0000 7fa0a7fff700 1 --2- 172.21.15.23:0/1410832410 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa0900a2920 0x7fa0900a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8078680 tx=0x7fa09c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:03.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.553+0000 7fa0ad49c700 1 --2- 172.21.15.23:0/1410832410 >> [v2:172.21.15.113:6824/2113644020,v1:172.21.15.113:6825/2113644020] conn(0x7fa0900ad560 0x7fa0900af9a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:03.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.554+0000 7fa0a5ffb700 1 -- 172.21.15.23:0/1410832410 <== osd.11 v2:172.21.15.113:6824/2113644020 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa098018e50 con 0x7fa0900ad560 2024-03-28T05:22:03.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.571+0000 7fa0aeeff700 1 -- 172.21.15.23:0/1410832410 --> [v2:172.21.15.113:6824/2113644020,v1:172.21.15.113:6825/2113644020] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa0a8102760 con 0x7fa0900ad560 2024-03-28T05:22:03.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.571+0000 7fa0a5ffb700 1 -- 172.21.15.23:0/1410832410 <== osd.11 v2:172.21.15.113:6824/2113644020 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa0a8102760 con 0x7fa0900ad560 2024-03-28T05:22:03.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.571+0000 7fa0aeeff700 1 -- 172.21.15.23:0/1410832410 >> [v2:172.21.15.113:6824/2113644020,v1:172.21.15.113:6825/2113644020] conn(0x7fa0900ad560 msgr2=0x7fa0900af9a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:03.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.571+0000 7fa0aeeff700 1 --2- 172.21.15.23:0/1410832410 >> [v2:172.21.15.113:6824/2113644020,v1:172.21.15.113:6825/2113644020] conn(0x7fa0900ad560 0x7fa0900af9a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:03.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.571+0000 7fa0aeeff700 1 -- 172.21.15.23:0/1410832410 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa0900a2920 msgr2=0x7fa0900a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:03.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.571+0000 7fa0aeeff700 1 --2- 172.21.15.23:0/1410832410 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa0900a2920 0x7fa0900a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8078680 tx=0x7fa09c009260 comp rx=0 tx=0).stop 2024-03-28T05:22:03.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.571+0000 7fa0aeeff700 1 -- 172.21.15.23:0/1410832410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0a8101070 msgr2=0x7fa0a819e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:03.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.571+0000 7fa0aeeff700 1 --2- 172.21.15.23:0/1410832410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0a8101070 0x7fa0a819e460 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa09800b2e0 tx=0x7fa09800b840 comp rx=0 tx=0).stop 2024-03-28T05:22:03.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.571+0000 7fa0ad49c700 1 -- 172.21.15.23:0/1410832410 reap_dead start 2024-03-28T05:22:03.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.572+0000 7fa0aeeff700 1 -- 172.21.15.23:0/1410832410 shutdown_connections 2024-03-28T05:22:03.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.572+0000 7fa0aeeff700 1 -- 172.21.15.23:0/1410832410 >> 172.21.15.23:0/1410832410 conn(0x7fa0a80780a0 msgr2=0x7fa0a8100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:03.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.572+0000 7fa0aeeff700 1 -- 172.21.15.23:0/1410832410 shutdown_connections 2024-03-28T05:22:03.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.572+0000 7fa0aeeff700 1 -- 172.21.15.23:0/1410832410 wait complete. 2024-03-28T05:22:03.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.737+0000 7fdf8786f700 1 -- 172.21.15.23:0/2347197886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf801074a0 msgr2=0x7fdf8010f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:03.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.737+0000 7fdf8786f700 1 --2- 172.21.15.23:0/2347197886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf801074a0 0x7fdf8010f070 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009990 tx=0x7fdf78009ca0 comp rx=0 tx=0).stop 2024-03-28T05:22:03.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.737+0000 7fdf8786f700 1 -- 172.21.15.23:0/2347197886 shutdown_connections 2024-03-28T05:22:03.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.737+0000 7fdf8786f700 1 --2- 172.21.15.23:0/2347197886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf801074a0 0x7fdf8010f070 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:03.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.737+0000 7fdf8786f700 1 --2- 172.21.15.23:0/2347197886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf80102470 0x7fdf80102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:03.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.737+0000 7fdf8786f700 1 --2- 172.21.15.23:0/2347197886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf80102e20 0x7fdf80106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:03.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.737+0000 7fdf8786f700 1 -- 172.21.15.23:0/2347197886 >> 172.21.15.23:0/2347197886 conn(0x7fdf80079220 msgr2=0x7fdf80079630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:03.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.737+0000 7fdf8786f700 1 -- 172.21.15.23:0/2347197886 shutdown_connections 2024-03-28T05:22:03.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.737+0000 7fdf8786f700 1 -- 172.21.15.23:0/2347197886 wait complete. 2024-03-28T05:22:03.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.738+0000 7fdf8786f700 1 Processor -- start 2024-03-28T05:22:03.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.738+0000 7fdf8786f700 1 -- start start 2024-03-28T05:22:03.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.738+0000 7fdf8786f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf80102470 0x7fdf8019e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:03.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.738+0000 7fdf8786f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf80102e20 0x7fdf8019eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:03.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.738+0000 7fdf8786f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf801074a0 0x7fdf801986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:03.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.738+0000 7fdf8786f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf80112480 con 0x7fdf80102e20 2024-03-28T05:22:03.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.738+0000 7fdf8786f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf80112300 con 0x7fdf801074a0 2024-03-28T05:22:03.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.738+0000 7fdf8786f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf80112600 con 0x7fdf80102470 2024-03-28T05:22:03.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.738+0000 7fdf84e0a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf80102e20 0x7fdf8019eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:03.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.738+0000 7fdf84e0a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf80102e20 0x7fdf8019eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36104/0 (socket says 172.21.15.23:36104) 2024-03-28T05:22:03.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.738+0000 7fdf84e0a700 1 -- 172.21.15.23:0/3376812892 learned_addr learned my addr 172.21.15.23:0/3376812892 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:03.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.738+0000 7fdf8560b700 1 --2- 172.21.15.23:0/3376812892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf80102470 0x7fdf8019e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:03.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.738+0000 7fdf85e0c700 1 --2- 172.21.15.23:0/3376812892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf801074a0 0x7fdf801986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:03.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.738+0000 7fdf84e0a700 1 -- 172.21.15.23:0/3376812892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf80102470 msgr2=0x7fdf8019e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:03.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.738+0000 7fdf84e0a700 1 --2- 172.21.15.23:0/3376812892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf80102470 0x7fdf8019e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:03.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.738+0000 7fdf84e0a700 1 -- 172.21.15.23:0/3376812892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf801074a0 msgr2=0x7fdf801986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:03.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.738+0000 7fdf84e0a700 1 --2- 172.21.15.23:0/3376812892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf801074a0 0x7fdf801986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:03.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.738+0000 7fdf84e0a700 1 -- 172.21.15.23:0/3376812892 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf80198f80 con 0x7fdf80102e20 2024-03-28T05:22:03.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.738+0000 7fdf85e0c700 1 --2- 172.21.15.23:0/3376812892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf801074a0 0x7fdf801986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:22:03.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.738+0000 7fdf8560b700 1 --2- 172.21.15.23:0/3376812892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf80102470 0x7fdf8019e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:22:03.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.738+0000 7fdf84e0a700 1 --2- 172.21.15.23:0/3376812892 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf80102e20 0x7fdf8019eb10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400e970 tx=0x7fdf7400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:03.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.739+0000 7fdf7e7fc700 1 -- 172.21.15.23:0/3376812892 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7400cbd0 con 0x7fdf80102e20 2024-03-28T05:22:03.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.739+0000 7fdf7e7fc700 1 -- 172.21.15.23:0/3376812892 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdf7400cd70 con 0x7fdf80102e20 2024-03-28T05:22:03.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.739+0000 7fdf8786f700 1 -- 172.21.15.23:0/3376812892 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf80199270 con 0x7fdf80102e20 2024-03-28T05:22:03.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.739+0000 7fdf7e7fc700 1 -- 172.21.15.23:0/3376812892 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf740056b0 con 0x7fdf80102e20 2024-03-28T05:22:03.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.739+0000 7fdf8786f700 1 -- 172.21.15.23:0/3376812892 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf801a5550 con 0x7fdf80102e20 2024-03-28T05:22:03.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.740+0000 7fdf8786f700 1 -- 172.21.15.23:0/3376812892 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdf800628e0 con 0x7fdf80102e20 2024-03-28T05:22:03.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.741+0000 7fdf7e7fc700 1 -- 172.21.15.23:0/3376812892 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fdf74004750 con 0x7fdf80102e20 2024-03-28T05:22:03.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.741+0000 7fdf7e7fc700 1 --2- 172.21.15.23:0/3376812892 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fdf680a29a0 0x7fdf680a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:03.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.742+0000 7fdf8560b700 1 --2- 172.21.15.23:0/3376812892 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fdf680a29a0 0x7fdf680a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:03.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.742+0000 7fdf7e7fc700 1 -- 172.21.15.23:0/3376812892 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fdf740cf750 con 0x7fdf80102e20 2024-03-28T05:22:03.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.742+0000 7fdf7e7fc700 1 --2- 172.21.15.23:0/3376812892 >> [v2:172.21.15.113:6816/3534876081,v1:172.21.15.113:6817/3534876081] conn(0x7fdf680ad660 0x7fdf680afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:03.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.742+0000 7fdf7e7fc700 1 -- 172.21.15.23:0/3376812892 --> [v2:172.21.15.113:6816/3534876081,v1:172.21.15.113:6817/3534876081] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdf74017e50 con 0x7fdf680ad660 2024-03-28T05:22:03.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.742+0000 7fdf7e7fc700 1 -- 172.21.15.23:0/3376812892 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdf740cfba0 con 0x7fdf80102e20 2024-03-28T05:22:03.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.742+0000 7fdf8560b700 1 --2- 172.21.15.23:0/3376812892 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fdf680a29a0 0x7fdf680a4e60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdf70007fd0 tx=0x7fdf7000c040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:03.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.742+0000 7fdf85e0c700 1 --2- 172.21.15.23:0/3376812892 >> [v2:172.21.15.113:6816/3534876081,v1:172.21.15.113:6817/3534876081] conn(0x7fdf680ad660 0x7fdf680afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:03.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.742+0000 7fdf85e0c700 1 --2- 172.21.15.23:0/3376812892 >> [v2:172.21.15.113:6816/3534876081,v1:172.21.15.113:6817/3534876081] conn(0x7fdf680ad660 0x7fdf680afaa0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:03.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.743+0000 7fdf7e7fc700 1 -- 172.21.15.23:0/3376812892 <== osd.8 v2:172.21.15.113:6816/3534876081 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdf74017e50 con 0x7fdf680ad660 2024-03-28T05:22:03.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.759+0000 7fdf8786f700 1 -- 172.21.15.23:0/3376812892 --> [v2:172.21.15.113:6816/3534876081,v1:172.21.15.113:6817/3534876081] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdf80062ce0 con 0x7fdf680ad660 2024-03-28T05:22:03.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.760+0000 7fdf7e7fc700 1 -- 172.21.15.23:0/3376812892 <== osd.8 v2:172.21.15.113:6816/3534876081 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fdf80062ce0 con 0x7fdf680ad660 2024-03-28T05:22:03.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.760+0000 7fdf8786f700 1 -- 172.21.15.23:0/3376812892 >> [v2:172.21.15.113:6816/3534876081,v1:172.21.15.113:6817/3534876081] conn(0x7fdf680ad660 msgr2=0x7fdf680afaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:03.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.760+0000 7fdf8786f700 1 --2- 172.21.15.23:0/3376812892 >> [v2:172.21.15.113:6816/3534876081,v1:172.21.15.113:6817/3534876081] conn(0x7fdf680ad660 0x7fdf680afaa0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:03.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.760+0000 7fdf8786f700 1 -- 172.21.15.23:0/3376812892 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fdf680a29a0 msgr2=0x7fdf680a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:03.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.760+0000 7fdf8786f700 1 --2- 172.21.15.23:0/3376812892 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fdf680a29a0 0x7fdf680a4e60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdf70007fd0 tx=0x7fdf7000c040 comp rx=0 tx=0).stop 2024-03-28T05:22:03.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.760+0000 7fdf8786f700 1 -- 172.21.15.23:0/3376812892 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf80102e20 msgr2=0x7fdf8019eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:03.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.760+0000 7fdf8786f700 1 --2- 172.21.15.23:0/3376812892 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf80102e20 0x7fdf8019eb10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400e970 tx=0x7fdf7400ec80 comp rx=0 tx=0).stop 2024-03-28T05:22:03.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.760+0000 7fdf85e0c700 1 -- 172.21.15.23:0/3376812892 reap_dead start 2024-03-28T05:22:03.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.760+0000 7fdf8786f700 1 -- 172.21.15.23:0/3376812892 shutdown_connections 2024-03-28T05:22:03.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.760+0000 7fdf8786f700 1 -- 172.21.15.23:0/3376812892 >> 172.21.15.23:0/3376812892 conn(0x7fdf80079220 msgr2=0x7fdf801002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:03.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.760+0000 7fdf8786f700 1 -- 172.21.15.23:0/3376812892 shutdown_connections 2024-03-28T05:22:03.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:03.760+0000 7fdf8786f700 1 -- 172.21.15.23:0/3376812892 wait complete. 2024-03-28T05:22:04.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:04 smithi113 ceph-mon[37164]: pgmap v316: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-28T05:22:04.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:04 smithi023 ceph-mon[31954]: pgmap v316: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-28T05:22:04.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:04 smithi098 ceph-mon[36087]: pgmap v316: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-28T05:22:05.276 INFO:teuthology.orchestra.run.smithi023.stdout:163208757301 2024-03-28T05:22:05.277 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-28T05:22:05.675 INFO:teuthology.orchestra.run.smithi023.stdout:343597383703 2024-03-28T05:22:05.676 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-28T05:22:05.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.863+0000 7f03529a5700 1 -- 172.21.15.23:0/772581180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f034c1021e0 msgr2=0x7f034c102660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:05.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.863+0000 7f03529a5700 1 --2- 172.21.15.23:0/772581180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f034c1021e0 0x7f034c102660 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f03400099c0 tx=0x7f0340009cd0 comp rx=0 tx=0).stop 2024-03-28T05:22:05.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.863+0000 7f03529a5700 1 -- 172.21.15.23:0/772581180 shutdown_connections 2024-03-28T05:22:05.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.863+0000 7f03529a5700 1 --2- 172.21.15.23:0/772581180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f034c102ba0 0x7f034c10efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:05.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.863+0000 7f03529a5700 1 --2- 172.21.15.23:0/772581180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f034c1081c0 0x7f034c1085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:05.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.863+0000 7f03529a5700 1 --2- 172.21.15.23:0/772581180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f034c1021e0 0x7f034c102660 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:05.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.863+0000 7f03529a5700 1 -- 172.21.15.23:0/772581180 >> 172.21.15.23:0/772581180 conn(0x7f034c0792c0 msgr2=0x7f034c0796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:05.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.863+0000 7f03529a5700 1 -- 172.21.15.23:0/772581180 shutdown_connections 2024-03-28T05:22:05.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.863+0000 7f03529a5700 1 -- 172.21.15.23:0/772581180 wait complete. 2024-03-28T05:22:05.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.864+0000 7f03529a5700 1 Processor -- start 2024-03-28T05:22:05.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.864+0000 7f03529a5700 1 -- start start 2024-03-28T05:22:05.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.864+0000 7f03529a5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f034c1021e0 0x7f034c1a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:05.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.864+0000 7f03529a5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f034c102ba0 0x7f034c1a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:05.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.864+0000 7f03529a5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f034c1081c0 0x7f034c19a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:05.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.864+0000 7f03529a5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034c112440 con 0x7f034c1081c0 2024-03-28T05:22:05.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.864+0000 7f03529a5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034c1122c0 con 0x7f034c102ba0 2024-03-28T05:22:05.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.864+0000 7f03529a5700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034c1125c0 con 0x7f034c1021e0 2024-03-28T05:22:05.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.864+0000 7f0350f42700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f034c1081c0 0x7f034c19a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:05.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.864+0000 7f0350f42700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f034c1081c0 0x7f034c19a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36126/0 (socket says 172.21.15.23:36126) 2024-03-28T05:22:05.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.864+0000 7f034bfff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f034c1021e0 0x7f034c1a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:05.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.864+0000 7f0350f42700 1 -- 172.21.15.23:0/1364234225 learned_addr learned my addr 172.21.15.23:0/1364234225 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:05.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.864+0000 7f034b7fe700 1 --2- 172.21.15.23:0/1364234225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f034c102ba0 0x7f034c1a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:05.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.865+0000 7f0350f42700 1 -- 172.21.15.23:0/1364234225 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f034c1021e0 msgr2=0x7f034c1a0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:05.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.865+0000 7f0350f42700 1 --2- 172.21.15.23:0/1364234225 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f034c1021e0 0x7f034c1a0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:05.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.865+0000 7f0350f42700 1 -- 172.21.15.23:0/1364234225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f034c102ba0 msgr2=0x7f034c1a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:05.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.865+0000 7f0350f42700 1 --2- 172.21.15.23:0/1364234225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f034c102ba0 0x7f034c1a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:05.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.865+0000 7f0350f42700 1 -- 172.21.15.23:0/1364234225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034c19b060 con 0x7f034c1081c0 2024-03-28T05:22:05.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.865+0000 7f0350f42700 1 --2- 172.21.15.23:0/1364234225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f034c1081c0 0x7f034c19a900 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0344007ee0 tx=0x7f034400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:05.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.865+0000 7f03497fa700 1 -- 172.21.15.23:0/1364234225 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034400dd20 con 0x7f034c1081c0 2024-03-28T05:22:05.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.865+0000 7f03529a5700 1 -- 172.21.15.23:0/1364234225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034c19b350 con 0x7f034c1081c0 2024-03-28T05:22:05.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.865+0000 7f03497fa700 1 -- 172.21.15.23:0/1364234225 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0344004020 con 0x7f034c1081c0 2024-03-28T05:22:05.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.865+0000 7f03497fa700 1 -- 172.21.15.23:0/1364234225 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0344015660 con 0x7f034c1081c0 2024-03-28T05:22:05.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.865+0000 7f03529a5700 1 -- 172.21.15.23:0/1364234225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034c1a7610 con 0x7f034c1081c0 2024-03-28T05:22:05.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.866+0000 7f03529a5700 1 -- 172.21.15.23:0/1364234225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f034c076060 con 0x7f034c1081c0 2024-03-28T05:22:05.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.867+0000 7f03497fa700 1 -- 172.21.15.23:0/1364234225 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f03440041d0 con 0x7f034c1081c0 2024-03-28T05:22:05.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.868+0000 7f03497fa700 1 --2- 172.21.15.23:0/1364234225 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f03340a2920 0x7f03340a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:05.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.868+0000 7f034bfff700 1 --2- 172.21.15.23:0/1364234225 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f03340a2920 0x7f03340a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:05.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.868+0000 7f03497fa700 1 -- 172.21.15.23:0/1364234225 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f03440ce820 con 0x7f034c1081c0 2024-03-28T05:22:05.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.868+0000 7f03497fa700 1 --2- 172.21.15.23:0/1364234225 >> [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881] conn(0x7f03340ad5e0 0x7f03340afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:05.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.868+0000 7f03497fa700 1 -- 172.21.15.23:0/1364234225 --> [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f03340b00f0 con 0x7f03340ad5e0 2024-03-28T05:22:05.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.868+0000 7f03497fa700 1 -- 172.21.15.23:0/1364234225 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03440d32c0 con 0x7f034c1081c0 2024-03-28T05:22:05.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.868+0000 7f034b7fe700 1 --2- 172.21.15.23:0/1364234225 >> [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881] conn(0x7f03340ad5e0 0x7f03340afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:05.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.868+0000 7f034bfff700 1 --2- 172.21.15.23:0/1364234225 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f03340a2920 0x7f03340a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f034c19ba40 tx=0x7f033c00a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:05.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.868+0000 7f034b7fe700 1 --2- 172.21.15.23:0/1364234225 >> [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881] conn(0x7f03340ad5e0 0x7f03340afa20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:05.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.869+0000 7f03497fa700 1 -- 172.21.15.23:0/1364234225 <== osd.0 v2:172.21.15.23:6802/1219956881 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f03340b00f0 con 0x7f03340ad5e0 2024-03-28T05:22:05.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.885+0000 7f03529a5700 1 -- 172.21.15.23:0/1364234225 --> [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f034c103870 con 0x7f03340ad5e0 2024-03-28T05:22:05.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.886+0000 7f03497fa700 1 -- 172.21.15.23:0/1364234225 <== osd.0 v2:172.21.15.23:6802/1219956881 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f034c103870 con 0x7f03340ad5e0 2024-03-28T05:22:05.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.886+0000 7f03529a5700 1 -- 172.21.15.23:0/1364234225 >> [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881] conn(0x7f03340ad5e0 msgr2=0x7f03340afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:05.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.886+0000 7f03529a5700 1 --2- 172.21.15.23:0/1364234225 >> [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881] conn(0x7f03340ad5e0 0x7f03340afa20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:05.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.886+0000 7f03529a5700 1 -- 172.21.15.23:0/1364234225 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f03340a2920 msgr2=0x7f03340a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:05.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.886+0000 7f03529a5700 1 --2- 172.21.15.23:0/1364234225 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f03340a2920 0x7f03340a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f034c19ba40 tx=0x7f033c00a380 comp rx=0 tx=0).stop 2024-03-28T05:22:05.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.886+0000 7f03529a5700 1 -- 172.21.15.23:0/1364234225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f034c1081c0 msgr2=0x7f034c19a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:05.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.886+0000 7f03529a5700 1 --2- 172.21.15.23:0/1364234225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f034c1081c0 0x7f034c19a900 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0344007ee0 tx=0x7f034400d3b0 comp rx=0 tx=0).stop 2024-03-28T05:22:05.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.886+0000 7f0350f42700 1 -- 172.21.15.23:0/1364234225 reap_dead start 2024-03-28T05:22:05.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.886+0000 7f03529a5700 1 -- 172.21.15.23:0/1364234225 shutdown_connections 2024-03-28T05:22:05.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.886+0000 7f03529a5700 1 -- 172.21.15.23:0/1364234225 >> 172.21.15.23:0/1364234225 conn(0x7f034c0792c0 msgr2=0x7f034c0745f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:05.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.886+0000 7f03529a5700 1 -- 172.21.15.23:0/1364234225 shutdown_connections 2024-03-28T05:22:05.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:05.886+0000 7f03529a5700 1 -- 172.21.15.23:0/1364234225 wait complete. 2024-03-28T05:22:06.637 INFO:teuthology.orchestra.run.smithi023.stdout:253403070503 2024-03-28T05:22:06.657 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-28T05:22:06.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:06 smithi113 ceph-mon[37164]: pgmap v317: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-28T05:22:06.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:06 smithi023 ceph-mon[31954]: pgmap v317: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-28T05:22:06.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:06 smithi098 ceph-mon[36087]: pgmap v317: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-28T05:22:07.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.522+0000 7f40f1105700 1 -- 172.21.15.23:0/4210123616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40ec1074a0 msgr2=0x7f40ec10f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:07.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.522+0000 7f40f1105700 1 --2- 172.21.15.23:0/4210123616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40ec1074a0 0x7f40ec10f070 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f40e400b0a0 tx=0x7f40e400b3b0 comp rx=0 tx=0).stop 2024-03-28T05:22:07.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.522+0000 7f40f1105700 1 -- 172.21.15.23:0/4210123616 shutdown_connections 2024-03-28T05:22:07.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.522+0000 7f40f1105700 1 --2- 172.21.15.23:0/4210123616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40ec1074a0 0x7f40ec10f070 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:07.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.522+0000 7f40f1105700 1 --2- 172.21.15.23:0/4210123616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40ec102470 0x7f40ec102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:07.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.522+0000 7f40f1105700 1 --2- 172.21.15.23:0/4210123616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40ec102e20 0x7f40ec106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:07.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.522+0000 7f40f1105700 1 -- 172.21.15.23:0/4210123616 >> 172.21.15.23:0/4210123616 conn(0x7f40ec079220 msgr2=0x7f40ec079630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:07.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.522+0000 7f40f1105700 1 -- 172.21.15.23:0/4210123616 shutdown_connections 2024-03-28T05:22:07.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.522+0000 7f40f1105700 1 -- 172.21.15.23:0/4210123616 wait complete. 2024-03-28T05:22:07.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.522+0000 7f40f1105700 1 Processor -- start 2024-03-28T05:22:07.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.523+0000 7f40f1105700 1 -- start start 2024-03-28T05:22:07.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.523+0000 7f40f1105700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40ec102470 0x7f40ec19e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:07.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.523+0000 7f40f1105700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40ec102e20 0x7f40ec19eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:07.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.523+0000 7f40f1105700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40ec1074a0 0x7f40ec198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:07.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.523+0000 7f40f1105700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ec112480 con 0x7f40ec1074a0 2024-03-28T05:22:07.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.523+0000 7f40f1105700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ec112300 con 0x7f40ec102470 2024-03-28T05:22:07.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.523+0000 7f40f1105700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ec112600 con 0x7f40ec102e20 2024-03-28T05:22:07.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.523+0000 7f40eb59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40ec1074a0 0x7f40ec198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:07.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.523+0000 7f40eb59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40ec1074a0 0x7f40ec198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36150/0 (socket says 172.21.15.23:36150) 2024-03-28T05:22:07.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.523+0000 7f40eb59e700 1 -- 172.21.15.23:0/2274960637 learned_addr learned my addr 172.21.15.23:0/2274960637 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:07.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.523+0000 7f40e3fff700 1 --2- 172.21.15.23:0/2274960637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40ec102e20 0x7f40ec19eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:07.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.523+0000 7f40ead9d700 1 --2- 172.21.15.23:0/2274960637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40ec102470 0x7f40ec19e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:07.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.523+0000 7f40eb59e700 1 -- 172.21.15.23:0/2274960637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40ec102e20 msgr2=0x7f40ec19eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:07.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.523+0000 7f40eb59e700 1 --2- 172.21.15.23:0/2274960637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40ec102e20 0x7f40ec19eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:07.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.523+0000 7f40eb59e700 1 -- 172.21.15.23:0/2274960637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40ec102470 msgr2=0x7f40ec19e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:07.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.523+0000 7f40eb59e700 1 --2- 172.21.15.23:0/2274960637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40ec102470 0x7f40ec19e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:07.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.523+0000 7f40eb59e700 1 -- 172.21.15.23:0/2274960637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ec199000 con 0x7f40ec1074a0 2024-03-28T05:22:07.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.523+0000 7f40ead9d700 1 --2- 172.21.15.23:0/2274960637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40ec102470 0x7f40ec19e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:22:07.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.523+0000 7f40eb59e700 1 --2- 172.21.15.23:0/2274960637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40ec1074a0 0x7f40ec198720 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f40e400b780 tx=0x7f40e4004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:07.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.524+0000 7f40e8d99700 1 -- 172.21.15.23:0/2274960637 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e4021070 con 0x7f40ec1074a0 2024-03-28T05:22:07.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.524+0000 7f40e8d99700 1 -- 172.21.15.23:0/2274960637 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40e4007b80 con 0x7f40ec1074a0 2024-03-28T05:22:07.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.524+0000 7f40f1105700 1 -- 172.21.15.23:0/2274960637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40ec199290 con 0x7f40ec1074a0 2024-03-28T05:22:07.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.524+0000 7f40e8d99700 1 -- 172.21.15.23:0/2274960637 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e4005370 con 0x7f40ec1074a0 2024-03-28T05:22:07.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.524+0000 7f40f1105700 1 -- 172.21.15.23:0/2274960637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40ec074590 con 0x7f40ec1074a0 2024-03-28T05:22:07.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.524+0000 7f40f1105700 1 -- 172.21.15.23:0/2274960637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f40ec062920 con 0x7f40ec1074a0 2024-03-28T05:22:07.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.526+0000 7f40e8d99700 1 -- 172.21.15.23:0/2274960637 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f40e4024070 con 0x7f40ec1074a0 2024-03-28T05:22:07.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.526+0000 7f40e8d99700 1 --2- 172.21.15.23:0/2274960637 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f40d80a2920 0x7f40d80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:07.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.526+0000 7f40ead9d700 1 --2- 172.21.15.23:0/2274960637 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f40d80a2920 0x7f40d80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:07.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.526+0000 7f40e8d99700 1 -- 172.21.15.23:0/2274960637 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f40e400f020 con 0x7f40ec1074a0 2024-03-28T05:22:07.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.527+0000 7f40e8d99700 1 --2- 172.21.15.23:0/2274960637 >> [v2:172.21.15.98:6816/1985546028,v1:172.21.15.98:6817/1985546028] conn(0x7f40d80ad5e0 0x7f40d80afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:07.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.527+0000 7f40e8d99700 1 -- 172.21.15.23:0/2274960637 --> [v2:172.21.15.98:6816/1985546028,v1:172.21.15.98:6817/1985546028] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f40e4028e50 con 0x7f40d80ad5e0 2024-03-28T05:22:07.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.527+0000 7f40e8d99700 1 -- 172.21.15.23:0/2274960637 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40e40d0a10 con 0x7f40ec1074a0 2024-03-28T05:22:07.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.527+0000 7f40ead9d700 1 --2- 172.21.15.23:0/2274960637 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f40d80a2920 0x7f40d80a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f40dc005d90 tx=0x7f40dc005d00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:07.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.527+0000 7f40e3fff700 1 --2- 172.21.15.23:0/2274960637 >> [v2:172.21.15.98:6816/1985546028,v1:172.21.15.98:6817/1985546028] conn(0x7f40d80ad5e0 0x7f40d80afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:07.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.527+0000 7f40e3fff700 1 --2- 172.21.15.23:0/2274960637 >> [v2:172.21.15.98:6816/1985546028,v1:172.21.15.98:6817/1985546028] conn(0x7f40d80ad5e0 0x7f40d80afa20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:07.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.528+0000 7f40e8d99700 1 -- 172.21.15.23:0/2274960637 <== osd.7 v2:172.21.15.98:6816/1985546028 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f40e4028e50 con 0x7f40d80ad5e0 2024-03-28T05:22:07.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.545+0000 7f40f1105700 1 -- 172.21.15.23:0/2274960637 --> [v2:172.21.15.98:6816/1985546028,v1:172.21.15.98:6817/1985546028] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f40ec062da0 con 0x7f40d80ad5e0 2024-03-28T05:22:07.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.545+0000 7f40e8d99700 1 -- 172.21.15.23:0/2274960637 <== osd.7 v2:172.21.15.98:6816/1985546028 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f40ec062da0 con 0x7f40d80ad5e0 2024-03-28T05:22:07.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.545+0000 7f40f1105700 1 -- 172.21.15.23:0/2274960637 >> [v2:172.21.15.98:6816/1985546028,v1:172.21.15.98:6817/1985546028] conn(0x7f40d80ad5e0 msgr2=0x7f40d80afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:07.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.545+0000 7f40f1105700 1 --2- 172.21.15.23:0/2274960637 >> [v2:172.21.15.98:6816/1985546028,v1:172.21.15.98:6817/1985546028] conn(0x7f40d80ad5e0 0x7f40d80afa20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:07.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.546+0000 7f40f1105700 1 -- 172.21.15.23:0/2274960637 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f40d80a2920 msgr2=0x7f40d80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:07.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.546+0000 7f40f1105700 1 --2- 172.21.15.23:0/2274960637 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f40d80a2920 0x7f40d80a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f40dc005d90 tx=0x7f40dc005d00 comp rx=0 tx=0).stop 2024-03-28T05:22:07.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.546+0000 7f40f1105700 1 -- 172.21.15.23:0/2274960637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40ec1074a0 msgr2=0x7f40ec198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:07.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.546+0000 7f40f1105700 1 --2- 172.21.15.23:0/2274960637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40ec1074a0 0x7f40ec198720 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f40e400b780 tx=0x7f40e4004d00 comp rx=0 tx=0).stop 2024-03-28T05:22:07.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.546+0000 7f40eb59e700 1 -- 172.21.15.23:0/2274960637 reap_dead start 2024-03-28T05:22:07.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.546+0000 7f40f1105700 1 -- 172.21.15.23:0/2274960637 shutdown_connections 2024-03-28T05:22:07.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.546+0000 7f40f1105700 1 -- 172.21.15.23:0/2274960637 >> 172.21.15.23:0/2274960637 conn(0x7f40ec079220 msgr2=0x7f40ec100260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:07.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.546+0000 7f40f1105700 1 -- 172.21.15.23:0/2274960637 shutdown_connections 2024-03-28T05:22:07.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:07.546+0000 7f40f1105700 1 -- 172.21.15.23:0/2274960637 wait complete. 2024-03-28T05:22:08.184 INFO:teuthology.orchestra.run.smithi023.stdout:34359738447 2024-03-28T05:22:08.184 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-28T05:22:08.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.613+0000 7f74c3e94700 1 -- 172.21.15.23:0/3077293272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74bc1074b0 msgr2=0x7f74bc10f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:08.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.613+0000 7f74c3e94700 1 --2- 172.21.15.23:0/3077293272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74bc1074b0 0x7f74bc10f080 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f74b400b0d0 tx=0x7f74b400b3e0 comp rx=0 tx=0).stop 2024-03-28T05:22:08.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.613+0000 7f74c3e94700 1 -- 172.21.15.23:0/3077293272 shutdown_connections 2024-03-28T05:22:08.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.613+0000 7f74c3e94700 1 --2- 172.21.15.23:0/3077293272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74bc1074b0 0x7f74bc10f080 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:08.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.613+0000 7f74c3e94700 1 --2- 172.21.15.23:0/3077293272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74bc1023d0 0x7f74bc1027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:08.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.613+0000 7f74c3e94700 1 --2- 172.21.15.23:0/3077293272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f74bc102d80 0x7f74bc106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:08.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.613+0000 7f74c3e94700 1 -- 172.21.15.23:0/3077293272 >> 172.21.15.23:0/3077293272 conn(0x7f74bc0fdc40 msgr2=0x7f74bc100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:08.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.613+0000 7f74c3e94700 1 -- 172.21.15.23:0/3077293272 shutdown_connections 2024-03-28T05:22:08.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.613+0000 7f74c3e94700 1 -- 172.21.15.23:0/3077293272 wait complete. 2024-03-28T05:22:08.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.614+0000 7f74c3e94700 1 Processor -- start 2024-03-28T05:22:08.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.614+0000 7f74c3e94700 1 -- start start 2024-03-28T05:22:08.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.614+0000 7f74c3e94700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f74bc1023d0 0x7f74bc074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:08.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.614+0000 7f74c3e94700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74bc102d80 0x7f74bc0749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:08.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.614+0000 7f74c3e94700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74bc1074b0 0x7f74bc074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:08.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.614+0000 7f74c3e94700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc1124a0 con 0x7f74bc102d80 2024-03-28T05:22:08.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.614+0000 7f74c3e94700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc112320 con 0x7f74bc1074b0 2024-03-28T05:22:08.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.614+0000 7f74c3e94700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc112620 con 0x7f74bc1023d0 2024-03-28T05:22:08.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.614+0000 7f74c1c30700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f74bc1023d0 0x7f74bc074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:08.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.614+0000 7f74c1c30700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f74bc1023d0 0x7f74bc074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:43492/0 (socket says 172.21.15.23:43492) 2024-03-28T05:22:08.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.614+0000 7f74c1c30700 1 -- 172.21.15.23:0/2501733764 learned_addr learned my addr 172.21.15.23:0/2501733764 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:08.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.614+0000 7f74c2431700 1 --2- 172.21.15.23:0/2501733764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74bc1074b0 0x7f74bc074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:08.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.615+0000 7f74c142f700 1 --2- 172.21.15.23:0/2501733764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74bc102d80 0x7f74bc0749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:08.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.615+0000 7f74c1c30700 1 -- 172.21.15.23:0/2501733764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74bc1074b0 msgr2=0x7f74bc074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:08.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.615+0000 7f74c1c30700 1 --2- 172.21.15.23:0/2501733764 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74bc1074b0 0x7f74bc074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:08.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.615+0000 7f74c1c30700 1 -- 172.21.15.23:0/2501733764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74bc102d80 msgr2=0x7f74bc0749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:08.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.615+0000 7f74c1c30700 1 --2- 172.21.15.23:0/2501733764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74bc102d80 0x7f74bc0749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:08.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.615+0000 7f74c1c30700 1 -- 172.21.15.23:0/2501733764 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74bc1a6dd0 con 0x7f74bc1023d0 2024-03-28T05:22:08.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.615+0000 7f74c1c30700 1 --2- 172.21.15.23:0/2501733764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f74bc1023d0 0x7f74bc074490 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f74b800d950 tx=0x7f74b800dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:08.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.615+0000 7f74b2ffd700 1 -- 172.21.15.23:0/2501733764 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b8016070 con 0x7f74bc1023d0 2024-03-28T05:22:08.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.615+0000 7f74c3e94700 1 -- 172.21.15.23:0/2501733764 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74bc1a70c0 con 0x7f74bc1023d0 2024-03-28T05:22:08.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.615+0000 7f74b2ffd700 1 -- 172.21.15.23:0/2501733764 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f74b8009e20 con 0x7f74bc1023d0 2024-03-28T05:22:08.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.615+0000 7f74b2ffd700 1 -- 172.21.15.23:0/2501733764 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b800f620 con 0x7f74bc1023d0 2024-03-28T05:22:08.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.615+0000 7f74c3e94700 1 -- 172.21.15.23:0/2501733764 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74bc1a7600 con 0x7f74bc1023d0 2024-03-28T05:22:08.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.616+0000 7f74c3e94700 1 -- 172.21.15.23:0/2501733764 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f74bc0ff080 con 0x7f74bc1023d0 2024-03-28T05:22:08.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.618+0000 7f74b2ffd700 1 -- 172.21.15.23:0/2501733764 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f74b8019070 con 0x7f74bc1023d0 2024-03-28T05:22:08.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.618+0000 7f74b2ffd700 1 --2- 172.21.15.23:0/2501733764 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f74a80a2920 0x7f74a80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:08.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.618+0000 7f74c142f700 1 --2- 172.21.15.23:0/2501733764 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f74a80a2920 0x7f74a80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:08.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.619+0000 7f74b2ffd700 1 -- 172.21.15.23:0/2501733764 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f74b80cef20 con 0x7f74bc1023d0 2024-03-28T05:22:08.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.619+0000 7f74b2ffd700 1 --2- 172.21.15.23:0/2501733764 >> [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207] conn(0x7f74a80ad560 0x7f74a80af9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:08.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.619+0000 7f74b2ffd700 1 -- 172.21.15.23:0/2501733764 --> [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f74b801ee50 con 0x7f74a80ad560 2024-03-28T05:22:08.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.619+0000 7f74b2ffd700 1 -- 172.21.15.23:0/2501733764 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74b80cf370 con 0x7f74bc1023d0 2024-03-28T05:22:08.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.619+0000 7f74c2431700 1 --2- 172.21.15.23:0/2501733764 >> [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207] conn(0x7f74a80ad560 0x7f74a80af9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:08.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.619+0000 7f74c142f700 1 --2- 172.21.15.23:0/2501733764 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f74a80a2920 0x7f74a80a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f74ac005950 tx=0x7f74ac0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:08.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.619+0000 7f74c2431700 1 --2- 172.21.15.23:0/2501733764 >> [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207] conn(0x7f74a80ad560 0x7f74a80af9a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:08.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.620+0000 7f74b2ffd700 1 -- 172.21.15.23:0/2501733764 <== osd.9 v2:172.21.15.23:6826/672264207 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f74b801ee50 con 0x7f74a80ad560 2024-03-28T05:22:08.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.638+0000 7f74c3e94700 1 -- 172.21.15.23:0/2501733764 --> [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f74bc1038b0 con 0x7f74a80ad560 2024-03-28T05:22:08.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.638+0000 7f74b2ffd700 1 -- 172.21.15.23:0/2501733764 <== osd.9 v2:172.21.15.23:6826/672264207 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f74bc1038b0 con 0x7f74a80ad560 2024-03-28T05:22:08.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.639+0000 7f74c3e94700 1 -- 172.21.15.23:0/2501733764 >> [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207] conn(0x7f74a80ad560 msgr2=0x7f74a80af9a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:08.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.639+0000 7f74c3e94700 1 --2- 172.21.15.23:0/2501733764 >> [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207] conn(0x7f74a80ad560 0x7f74a80af9a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:08.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.639+0000 7f74c3e94700 1 -- 172.21.15.23:0/2501733764 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f74a80a2920 msgr2=0x7f74a80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:08.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.639+0000 7f74c3e94700 1 --2- 172.21.15.23:0/2501733764 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f74a80a2920 0x7f74a80a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f74ac005950 tx=0x7f74ac0058e0 comp rx=0 tx=0).stop 2024-03-28T05:22:08.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.639+0000 7f74c3e94700 1 -- 172.21.15.23:0/2501733764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f74bc1023d0 msgr2=0x7f74bc074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:08.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.639+0000 7f74c3e94700 1 --2- 172.21.15.23:0/2501733764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f74bc1023d0 0x7f74bc074490 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f74b800d950 tx=0x7f74b800dc60 comp rx=0 tx=0).stop 2024-03-28T05:22:08.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.639+0000 7f74c2431700 1 -- 172.21.15.23:0/2501733764 reap_dead start 2024-03-28T05:22:08.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.639+0000 7f74c3e94700 1 -- 172.21.15.23:0/2501733764 shutdown_connections 2024-03-28T05:22:08.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.639+0000 7f74c3e94700 1 -- 172.21.15.23:0/2501733764 >> 172.21.15.23:0/2501733764 conn(0x7f74bc0fdc40 msgr2=0x7f74bc0fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:08.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.639+0000 7f74c3e94700 1 -- 172.21.15.23:0/2501733764 shutdown_connections 2024-03-28T05:22:08.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:08.639+0000 7f74c3e94700 1 -- 172.21.15.23:0/2501733764 wait complete. 2024-03-28T05:22:08.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:08 smithi113 ceph-mon[37164]: pgmap v318: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:08.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:08 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:22:08.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:08 smithi023 ceph-mon[31954]: pgmap v318: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:08.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:08 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:22:08.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:08 smithi098 ceph-mon[36087]: pgmap v318: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:08.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:08 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:22:09.268 INFO:teuthology.orchestra.run.smithi023.stdout:219043332141 2024-03-28T05:22:09.268 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-28T05:22:09.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-03-28T05:22:09.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-03-28T05:22:09.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-03-28T05:22:09.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 7]}]: dispatch 2024-03-28T05:22:09.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-28T05:22:09.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-03-28T05:22:09.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]: dispatch 2024-03-28T05:22:09.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-03-28T05:22:09.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-03-28T05:22:09.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-03-28T05:22:09.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-03-28T05:22:09.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-03-28T05:22:09.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-03-28T05:22:09.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 7]}]: dispatch 2024-03-28T05:22:09.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-28T05:22:09.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-03-28T05:22:09.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]: dispatch 2024-03-28T05:22:09.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-03-28T05:22:09.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-03-28T05:22:09.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-03-28T05:22:09.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-03-28T05:22:09.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-03-28T05:22:09.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-03-28T05:22:09.742 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 7]}]: dispatch 2024-03-28T05:22:09.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-28T05:22:09.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-03-28T05:22:09.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]: dispatch 2024-03-28T05:22:09.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-03-28T05:22:09.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-03-28T05:22:09.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-03-28T05:22:10.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.604+0000 7ff2cfa67700 1 -- 172.21.15.23:0/4135392692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2c81060c0 msgr2=0x7ff2c81064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:10.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.604+0000 7ff2cfa67700 1 --2- 172.21.15.23:0/4135392692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2c81060c0 0x7ff2c81064a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff2b800b0d0 tx=0x7ff2b800b3e0 comp rx=0 tx=0).stop 2024-03-28T05:22:10.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.604+0000 7ff2cfa67700 1 -- 172.21.15.23:0/4135392692 shutdown_connections 2024-03-28T05:22:10.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.604+0000 7ff2cfa67700 1 --2- 172.21.15.23:0/4135392692 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2c8102600 0x7ff2c810dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:10.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.604+0000 7ff2cfa67700 1 --2- 172.21.15.23:0/4135392692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2c81060c0 0x7ff2c81064a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:10.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.604+0000 7ff2cfa67700 1 --2- 172.21.15.23:0/4135392692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2c8101c40 0x7ff2c81020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:10.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.604+0000 7ff2cfa67700 1 -- 172.21.15.23:0/4135392692 >> 172.21.15.23:0/4135392692 conn(0x7ff2c8076410 msgr2=0x7ff2c8076820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:10.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.604+0000 7ff2cfa67700 1 -- 172.21.15.23:0/4135392692 shutdown_connections 2024-03-28T05:22:10.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.604+0000 7ff2cfa67700 1 -- 172.21.15.23:0/4135392692 wait complete. 2024-03-28T05:22:10.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.604+0000 7ff2cfa67700 1 Processor -- start 2024-03-28T05:22:10.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.604+0000 7ff2cfa67700 1 -- start start 2024-03-28T05:22:10.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.605+0000 7ff2cfa67700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2c8101c40 0x7ff2c8102dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:10.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.605+0000 7ff2cfa67700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2c8102600 0x7ff2c8103300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:10.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.605+0000 7ff2cfa67700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2c81060c0 0x7ff2c8103840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:10.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.605+0000 7ff2cfa67700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c8111430 con 0x7ff2c8102600 2024-03-28T05:22:10.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.605+0000 7ff2cfa67700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c81112b0 con 0x7ff2c8101c40 2024-03-28T05:22:10.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.605+0000 7ff2cfa67700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c81115b0 con 0x7ff2c81060c0 2024-03-28T05:22:10.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.605+0000 7ff2ce004700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2c81060c0 0x7ff2c8103840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:10.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.605+0000 7ff2ce004700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2c81060c0 0x7ff2c8103840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:43504/0 (socket says 172.21.15.23:43504) 2024-03-28T05:22:10.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.605+0000 7ff2ce004700 1 -- 172.21.15.23:0/1998094298 learned_addr learned my addr 172.21.15.23:0/1998094298 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:10.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.605+0000 7ff2cd803700 1 --2- 172.21.15.23:0/1998094298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2c8101c40 0x7ff2c8102dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:10.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.605+0000 7ff2cd002700 1 --2- 172.21.15.23:0/1998094298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2c8102600 0x7ff2c8103300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:10.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.605+0000 7ff2ce004700 1 -- 172.21.15.23:0/1998094298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2c8101c40 msgr2=0x7ff2c8102dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:10.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.605+0000 7ff2ce004700 1 --2- 172.21.15.23:0/1998094298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2c8101c40 0x7ff2c8102dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:10.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.605+0000 7ff2ce004700 1 -- 172.21.15.23:0/1998094298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2c8102600 msgr2=0x7ff2c8103300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:10.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.605+0000 7ff2ce004700 1 --2- 172.21.15.23:0/1998094298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff2c8102600 0x7ff2c8103300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:10.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.605+0000 7ff2ce004700 1 -- 172.21.15.23:0/1998094298 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c81a6eb0 con 0x7ff2c81060c0 2024-03-28T05:22:10.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.606+0000 7ff2ce004700 1 --2- 172.21.15.23:0/1998094298 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2c81060c0 0x7ff2c8103840 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff2c000e940 tx=0x7ff2c000ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:10.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.606+0000 7ff2c6ffd700 1 -- 172.21.15.23:0/1998094298 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c000cbd0 con 0x7ff2c81060c0 2024-03-28T05:22:10.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.606+0000 7ff2c6ffd700 1 -- 172.21.15.23:0/1998094298 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff2c000ee90 con 0x7ff2c81060c0 2024-03-28T05:22:10.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.606+0000 7ff2cfa67700 1 -- 172.21.15.23:0/1998094298 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2c81a71a0 con 0x7ff2c81060c0 2024-03-28T05:22:10.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.606+0000 7ff2c6ffd700 1 -- 172.21.15.23:0/1998094298 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c00056c0 con 0x7ff2c81060c0 2024-03-28T05:22:10.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.606+0000 7ff2cfa67700 1 -- 172.21.15.23:0/1998094298 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c81a7690 con 0x7ff2c81060c0 2024-03-28T05:22:10.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.607+0000 7ff2cfa67700 1 -- 172.21.15.23:0/1998094298 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff2c80628a0 con 0x7ff2c81060c0 2024-03-28T05:22:10.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.608+0000 7ff2c6ffd700 1 -- 172.21.15.23:0/1998094298 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7ff2c0004750 con 0x7ff2c81060c0 2024-03-28T05:22:10.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.609+0000 7ff2c6ffd700 1 --2- 172.21.15.23:0/1998094298 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff2b00a2920 0x7ff2b00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:10.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.609+0000 7ff2cd803700 1 --2- 172.21.15.23:0/1998094298 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff2b00a2920 0x7ff2b00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:10.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.609+0000 7ff2c6ffd700 1 -- 172.21.15.23:0/1998094298 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7ff2c00cf620 con 0x7ff2c81060c0 2024-03-28T05:22:10.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.609+0000 7ff2c6ffd700 1 --2- 172.21.15.23:0/1998094298 >> [v2:172.21.15.98:6824/2699920595,v1:172.21.15.98:6825/2699920595] conn(0x7ff2b00ada80 0x7ff2b00aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:10.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.609+0000 7ff2c6ffd700 1 -- 172.21.15.23:0/1998094298 --> [v2:172.21.15.98:6824/2699920595,v1:172.21.15.98:6825/2699920595] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff2c0012ec0 con 0x7ff2b00ada80 2024-03-28T05:22:10.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.609+0000 7ff2c6ffd700 1 -- 172.21.15.23:0/1998094298 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff2c00cf9d0 con 0x7ff2c81060c0 2024-03-28T05:22:10.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.609+0000 7ff2cd803700 1 --2- 172.21.15.23:0/1998094298 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff2b00a2920 0x7ff2b00a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8009fd0 tx=0x7ff2b800b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:10.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.609+0000 7ff2cd002700 1 --2- 172.21.15.23:0/1998094298 >> [v2:172.21.15.98:6824/2699920595,v1:172.21.15.98:6825/2699920595] conn(0x7ff2b00ada80 0x7ff2b00aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:10.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.610+0000 7ff2cd002700 1 --2- 172.21.15.23:0/1998094298 >> [v2:172.21.15.98:6824/2699920595,v1:172.21.15.98:6825/2699920595] conn(0x7ff2b00ada80 0x7ff2b00aff20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:10.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.611+0000 7ff2c6ffd700 1 -- 172.21.15.23:0/1998094298 <== osd.10 v2:172.21.15.98:6824/2699920595 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff2c0012ec0 con 0x7ff2b00ada80 2024-03-28T05:22:10.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.632+0000 7ff2cfa67700 1 -- 172.21.15.23:0/1998094298 --> [v2:172.21.15.98:6824/2699920595,v1:172.21.15.98:6825/2699920595] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff2c8062d20 con 0x7ff2b00ada80 2024-03-28T05:22:10.631 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:10 smithi023 ceph-mon[31954]: pgmap v319: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:10.631 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]': finished 2024-03-28T05:22:10.632 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-03-28T05:22:10.632 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-03-28T05:22:10.632 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 7]}]': finished 2024-03-28T05:22:10.632 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-28T05:22:10.632 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-03-28T05:22:10.632 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]': finished 2024-03-28T05:22:10.632 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-03-28T05:22:10.632 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-03-28T05:22:10.632 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-03-28T05:22:10.632 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:10 smithi023 ceph-mon[31954]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T05:22:10.632 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:22:10.632 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:22:10.633 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:22:10.633 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:22:10.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.632+0000 7ff2c6ffd700 1 -- 172.21.15.23:0/1998094298 <== osd.10 v2:172.21.15.98:6824/2699920595 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff2c8062d20 con 0x7ff2b00ada80 2024-03-28T05:22:10.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.632+0000 7ff2cfa67700 1 -- 172.21.15.23:0/1998094298 >> [v2:172.21.15.98:6824/2699920595,v1:172.21.15.98:6825/2699920595] conn(0x7ff2b00ada80 msgr2=0x7ff2b00aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:10.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.632+0000 7ff2cfa67700 1 --2- 172.21.15.23:0/1998094298 >> [v2:172.21.15.98:6824/2699920595,v1:172.21.15.98:6825/2699920595] conn(0x7ff2b00ada80 0x7ff2b00aff20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:10.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.633+0000 7ff2cfa67700 1 -- 172.21.15.23:0/1998094298 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff2b00a2920 msgr2=0x7ff2b00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:10.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.633+0000 7ff2cfa67700 1 --2- 172.21.15.23:0/1998094298 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff2b00a2920 0x7ff2b00a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8009fd0 tx=0x7ff2b800b040 comp rx=0 tx=0).stop 2024-03-28T05:22:10.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.633+0000 7ff2cfa67700 1 -- 172.21.15.23:0/1998094298 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2c81060c0 msgr2=0x7ff2c8103840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:10.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.633+0000 7ff2cfa67700 1 --2- 172.21.15.23:0/1998094298 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff2c81060c0 0x7ff2c8103840 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff2c000e940 tx=0x7ff2c000ec50 comp rx=0 tx=0).stop 2024-03-28T05:22:10.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.633+0000 7ff2ce004700 1 -- 172.21.15.23:0/1998094298 reap_dead start 2024-03-28T05:22:10.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.633+0000 7ff2cfa67700 1 -- 172.21.15.23:0/1998094298 shutdown_connections 2024-03-28T05:22:10.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.633+0000 7ff2cfa67700 1 -- 172.21.15.23:0/1998094298 >> 172.21.15.23:0/1998094298 conn(0x7ff2c8076410 msgr2=0x7ff2c8100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:10.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.633+0000 7ff2cfa67700 1 -- 172.21.15.23:0/1998094298 shutdown_connections 2024-03-28T05:22:10.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:10.633+0000 7ff2cfa67700 1 -- 172.21.15.23:0/1998094298 wait complete. 2024-03-28T05:22:10.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:10 smithi098 ceph-mon[36087]: pgmap v319: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:10.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]': finished 2024-03-28T05:22:10.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-03-28T05:22:10.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-03-28T05:22:10.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 7]}]': finished 2024-03-28T05:22:10.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-28T05:22:10.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-03-28T05:22:10.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]': finished 2024-03-28T05:22:10.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-03-28T05:22:10.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-03-28T05:22:10.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-03-28T05:22:10.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:10 smithi098 ceph-mon[36087]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T05:22:10.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:22:10.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:22:10.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:22:10.866 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:22:10.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:10 smithi113 ceph-mon[37164]: pgmap v319: 129 pgs: 129 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:10.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]': finished 2024-03-28T05:22:10.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-03-28T05:22:10.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-03-28T05:22:10.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 7]}]': finished 2024-03-28T05:22:10.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-28T05:22:10.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-03-28T05:22:10.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]': finished 2024-03-28T05:22:10.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-03-28T05:22:10.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-03-28T05:22:10.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-03-28T05:22:10.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:10 smithi113 ceph-mon[37164]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T05:22:10.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:22:10.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:22:10.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:22:10.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:22:11.071 INFO:teuthology.orchestra.run.smithi023.stdout:287762808867 2024-03-28T05:22:11.071 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-28T05:22:11.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:11 smithi023 ceph-mon[31954]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T05:22:11.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:11 smithi098 ceph-mon[36087]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T05:22:11.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:11 smithi113 ceph-mon[37164]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T05:22:12.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.369+0000 7fbd36047700 1 -- 172.21.15.23:0/1777222642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd30107510 msgr2=0x7fbd3010f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:12.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.369+0000 7fbd36047700 1 --2- 172.21.15.23:0/1777222642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd30107510 0x7fbd3010f090 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbd2800b0d0 tx=0x7fbd2800b3e0 comp rx=0 tx=0).stop 2024-03-28T05:22:12.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.369+0000 7fbd36047700 1 -- 172.21.15.23:0/1777222642 shutdown_connections 2024-03-28T05:22:12.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.369+0000 7fbd36047700 1 --2- 172.21.15.23:0/1777222642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd30107510 0x7fbd3010f090 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:12.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.369+0000 7fbd36047700 1 --2- 172.21.15.23:0/1777222642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd3006a930 0x7fbd3006ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:12.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.369+0000 7fbd36047700 1 --2- 172.21.15.23:0/1777222642 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbd3006b2e0 0x7fbd30106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:12.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.369+0000 7fbd36047700 1 -- 172.21.15.23:0/1777222642 >> 172.21.15.23:0/1777222642 conn(0x7fbd300770e0 msgr2=0x7fbd300774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:12.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.369+0000 7fbd36047700 1 -- 172.21.15.23:0/1777222642 shutdown_connections 2024-03-28T05:22:12.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.369+0000 7fbd36047700 1 -- 172.21.15.23:0/1777222642 wait complete. 2024-03-28T05:22:12.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.369+0000 7fbd36047700 1 Processor -- start 2024-03-28T05:22:12.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.370+0000 7fbd36047700 1 -- start start 2024-03-28T05:22:12.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.370+0000 7fbd36047700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbd3006a930 0x7fbd301966f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:12.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.370+0000 7fbd36047700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd3006b2e0 0x7fbd30196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:12.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.370+0000 7fbd36047700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd30107510 0x7fbd3019afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:12.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.370+0000 7fbd36047700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd30110400 con 0x7fbd3006b2e0 2024-03-28T05:22:12.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.370+0000 7fbd36047700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd30110280 con 0x7fbd30107510 2024-03-28T05:22:12.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.370+0000 7fbd36047700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd30110580 con 0x7fbd3006a930 2024-03-28T05:22:12.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.370+0000 7fbd2f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbd3006a930 0x7fbd301966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:12.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.370+0000 7fbd2effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd3006b2e0 0x7fbd30196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:12.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.370+0000 7fbd2f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbd3006a930 0x7fbd301966f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:43524/0 (socket says 172.21.15.23:43524) 2024-03-28T05:22:12.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.370+0000 7fbd2effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd3006b2e0 0x7fbd30196c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36188/0 (socket says 172.21.15.23:36188) 2024-03-28T05:22:12.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.370+0000 7fbd2f7fe700 1 -- 172.21.15.23:0/3670177651 learned_addr learned my addr 172.21.15.23:0/3670177651 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:12.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.370+0000 7fbd2ffff700 1 --2- 172.21.15.23:0/3670177651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd30107510 0x7fbd3019afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:12.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.370+0000 7fbd2f7fe700 1 -- 172.21.15.23:0/3670177651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd30107510 msgr2=0x7fbd3019afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:12.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.370+0000 7fbd2f7fe700 1 --2- 172.21.15.23:0/3670177651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd30107510 0x7fbd3019afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:12.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.370+0000 7fbd2f7fe700 1 -- 172.21.15.23:0/3670177651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd3006b2e0 msgr2=0x7fbd30196c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:12.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.370+0000 7fbd2f7fe700 1 --2- 172.21.15.23:0/3670177651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd3006b2e0 0x7fbd30196c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:12.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.370+0000 7fbd2f7fe700 1 -- 172.21.15.23:0/3670177651 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd3019b700 con 0x7fbd3006a930 2024-03-28T05:22:12.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.370+0000 7fbd2effd700 1 --2- 172.21.15.23:0/3670177651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd3006b2e0 0x7fbd30196c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:22:12.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.371+0000 7fbd2f7fe700 1 --2- 172.21.15.23:0/3670177651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbd3006a930 0x7fbd301966f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbd2000d950 tx=0x7fbd2000dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:12.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.371+0000 7fbd2cff9700 1 -- 172.21.15.23:0/3670177651 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd20016070 con 0x7fbd3006a930 2024-03-28T05:22:12.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.371+0000 7fbd36047700 1 -- 172.21.15.23:0/3670177651 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd3019b9f0 con 0x7fbd3006a930 2024-03-28T05:22:12.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.371+0000 7fbd2cff9700 1 -- 172.21.15.23:0/3670177651 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd20009e20 con 0x7fbd3006a930 2024-03-28T05:22:12.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.371+0000 7fbd2cff9700 1 -- 172.21.15.23:0/3670177651 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2000f620 con 0x7fbd3006a930 2024-03-28T05:22:12.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.371+0000 7fbd36047700 1 -- 172.21.15.23:0/3670177651 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd301a3270 con 0x7fbd3006a930 2024-03-28T05:22:12.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.372+0000 7fbd36047700 1 -- 172.21.15.23:0/3670177651 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbd30062920 con 0x7fbd3006a930 2024-03-28T05:22:12.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.373+0000 7fbd2cff9700 1 -- 172.21.15.23:0/3670177651 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fbd20019070 con 0x7fbd3006a930 2024-03-28T05:22:12.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.374+0000 7fbd2cff9700 1 --2- 172.21.15.23:0/3670177651 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fbd180a6d30 0x7fbd180a91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:12.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.374+0000 7fbd2effd700 1 --2- 172.21.15.23:0/3670177651 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fbd180a6d30 0x7fbd180a91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:12.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.374+0000 7fbd2cff9700 1 -- 172.21.15.23:0/3670177651 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fbd200cf090 con 0x7fbd3006a930 2024-03-28T05:22:12.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.374+0000 7fbd2cff9700 1 --2- 172.21.15.23:0/3670177651 >> [v2:172.21.15.113:6800/1535856914,v1:172.21.15.113:6801/1535856914] conn(0x7fbd180b1e90 0x7fbd180b4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:12.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.374+0000 7fbd2cff9700 1 -- 172.21.15.23:0/3670177651 --> [v2:172.21.15.113:6800/1535856914,v1:172.21.15.113:6801/1535856914] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd20005690 con 0x7fbd180b1e90 2024-03-28T05:22:12.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.374+0000 7fbd2cff9700 1 -- 172.21.15.23:0/3670177651 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd200cf440 con 0x7fbd3006a930 2024-03-28T05:22:12.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.374+0000 7fbd2ffff700 1 --2- 172.21.15.23:0/3670177651 >> [v2:172.21.15.113:6800/1535856914,v1:172.21.15.113:6801/1535856914] conn(0x7fbd180b1e90 0x7fbd180b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:12.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.374+0000 7fbd2effd700 1 --2- 172.21.15.23:0/3670177651 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fbd180a6d30 0x7fbd180a91f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbd30106fa0 tx=0x7fbd240073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:12.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.374+0000 7fbd2ffff700 1 --2- 172.21.15.23:0/3670177651 >> [v2:172.21.15.113:6800/1535856914,v1:172.21.15.113:6801/1535856914] conn(0x7fbd180b1e90 0x7fbd180b4330 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:12.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.375+0000 7fbd2cff9700 1 -- 172.21.15.23:0/3670177651 <== osd.2 v2:172.21.15.113:6800/1535856914 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbd20005690 con 0x7fbd180b1e90 2024-03-28T05:22:12.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.391+0000 7fbd36047700 1 -- 172.21.15.23:0/3670177651 --> [v2:172.21.15.113:6800/1535856914,v1:172.21.15.113:6801/1535856914] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbd30062d20 con 0x7fbd180b1e90 2024-03-28T05:22:12.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.392+0000 7fbd2cff9700 1 -- 172.21.15.23:0/3670177651 <== osd.2 v2:172.21.15.113:6800/1535856914 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbd30062d20 con 0x7fbd180b1e90 2024-03-28T05:22:12.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.392+0000 7fbd36047700 1 -- 172.21.15.23:0/3670177651 >> [v2:172.21.15.113:6800/1535856914,v1:172.21.15.113:6801/1535856914] conn(0x7fbd180b1e90 msgr2=0x7fbd180b4330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:12.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.392+0000 7fbd36047700 1 --2- 172.21.15.23:0/3670177651 >> [v2:172.21.15.113:6800/1535856914,v1:172.21.15.113:6801/1535856914] conn(0x7fbd180b1e90 0x7fbd180b4330 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:12.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.392+0000 7fbd36047700 1 -- 172.21.15.23:0/3670177651 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fbd180a6d30 msgr2=0x7fbd180a91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:12.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.392+0000 7fbd36047700 1 --2- 172.21.15.23:0/3670177651 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fbd180a6d30 0x7fbd180a91f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbd30106fa0 tx=0x7fbd240073d0 comp rx=0 tx=0).stop 2024-03-28T05:22:12.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.392+0000 7fbd36047700 1 -- 172.21.15.23:0/3670177651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbd3006a930 msgr2=0x7fbd301966f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:12.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.392+0000 7fbd36047700 1 --2- 172.21.15.23:0/3670177651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbd3006a930 0x7fbd301966f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbd2000d950 tx=0x7fbd2000dc60 comp rx=0 tx=0).stop 2024-03-28T05:22:12.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.392+0000 7fbd2ffff700 1 -- 172.21.15.23:0/3670177651 reap_dead start 2024-03-28T05:22:12.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.393+0000 7fbd36047700 1 -- 172.21.15.23:0/3670177651 shutdown_connections 2024-03-28T05:22:12.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.393+0000 7fbd36047700 1 -- 172.21.15.23:0/3670177651 >> 172.21.15.23:0/3670177651 conn(0x7fbd300770e0 msgr2=0x7fbd30108640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:12.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.393+0000 7fbd36047700 1 -- 172.21.15.23:0/3670177651 shutdown_connections 2024-03-28T05:22:12.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.393+0000 7fbd36047700 1 -- 172.21.15.23:0/3670177651 wait complete. 2024-03-28T05:22:12.658 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:12 smithi023 ceph-mon[31954]: pgmap v322: 129 pgs: 1 activating, 128 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-03-28T05:22:12.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:12 smithi098 ceph-mon[36087]: pgmap v322: 129 pgs: 1 activating, 128 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-03-28T05:22:12.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:12 smithi113 ceph-mon[37164]: pgmap v322: 129 pgs: 1 activating, 128 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2 op/s 2024-03-28T05:22:13.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.525+0000 7f3006325700 1 -- 172.21.15.23:0/2866551507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30001081c0 msgr2=0x7f30001085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:13.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.525+0000 7f3006325700 1 --2- 172.21.15.23:0/2866551507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30001081c0 0x7f30001085a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00099c0 tx=0x7f2ff0009cd0 comp rx=0 tx=0).stop 2024-03-28T05:22:13.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.525+0000 7f3006325700 1 -- 172.21.15.23:0/2866551507 shutdown_connections 2024-03-28T05:22:13.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.525+0000 7f3006325700 1 --2- 172.21.15.23:0/2866551507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3000102ba0 0x7f300010efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:13.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.525+0000 7f3006325700 1 --2- 172.21.15.23:0/2866551507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30001081c0 0x7f30001085a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:13.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.525+0000 7f3006325700 1 --2- 172.21.15.23:0/2866551507 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30001021e0 0x7f3000102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:13.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.525+0000 7f3006325700 1 -- 172.21.15.23:0/2866551507 >> 172.21.15.23:0/2866551507 conn(0x7f30000792c0 msgr2=0x7f30000796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:13.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.525+0000 7f3006325700 1 -- 172.21.15.23:0/2866551507 shutdown_connections 2024-03-28T05:22:13.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.525+0000 7f3006325700 1 -- 172.21.15.23:0/2866551507 wait complete. 2024-03-28T05:22:13.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.526+0000 7f3006325700 1 Processor -- start 2024-03-28T05:22:13.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.526+0000 7f3006325700 1 -- start start 2024-03-28T05:22:13.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.527+0000 7f3006325700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30001021e0 0x7f300019e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:13.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.527+0000 7f3006325700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3000102ba0 0x7f300019eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:13.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.527+0000 7f3006325700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30001081c0 0x7f30001986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:13.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.527+0000 7f3006325700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3000112470 con 0x7f30001081c0 2024-03-28T05:22:13.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.527+0000 7f3006325700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30001122f0 con 0x7f3000102ba0 2024-03-28T05:22:13.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.527+0000 7f3006325700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30001125f0 con 0x7f30001021e0 2024-03-28T05:22:13.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.527+0000 7f30048c2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30001081c0 0x7f30001986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:13.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.527+0000 7f2ffffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30001021e0 0x7f300019e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:13.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.527+0000 7f30048c2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30001081c0 0x7f30001986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50506/0 (socket says 172.21.15.23:50506) 2024-03-28T05:22:13.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.527+0000 7f30048c2700 1 -- 172.21.15.23:0/3653542309 learned_addr learned my addr 172.21.15.23:0/3653542309 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:13.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.527+0000 7f2fff7fe700 1 --2- 172.21.15.23:0/3653542309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3000102ba0 0x7f300019eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:13.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.527+0000 7f2ffffff700 1 -- 172.21.15.23:0/3653542309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3000102ba0 msgr2=0x7f300019eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:13.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.527+0000 7f2ffffff700 1 --2- 172.21.15.23:0/3653542309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3000102ba0 0x7f300019eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:13.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.527+0000 7f2ffffff700 1 -- 172.21.15.23:0/3653542309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30001081c0 msgr2=0x7f30001986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:13.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.527+0000 7f2ffffff700 1 --2- 172.21.15.23:0/3653542309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30001081c0 0x7f30001986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:13.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.527+0000 7f2ffffff700 1 -- 172.21.15.23:0/3653542309 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3000198fb0 con 0x7f30001021e0 2024-03-28T05:22:13.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.527+0000 7f30048c2700 1 --2- 172.21.15.23:0/3653542309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30001081c0 0x7f30001986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:22:13.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.527+0000 7f2ffffff700 1 --2- 172.21.15.23:0/3653542309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30001021e0 0x7f300019e600 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00098e0 tx=0x7f2ff000fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:13.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.528+0000 7f2ffd7fa700 1 -- 172.21.15.23:0/3653542309 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0027070 con 0x7f30001021e0 2024-03-28T05:22:13.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.528+0000 7f2ffd7fa700 1 -- 172.21.15.23:0/3653542309 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ff0003680 con 0x7f30001021e0 2024-03-28T05:22:13.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.528+0000 7f2ffd7fa700 1 -- 172.21.15.23:0/3653542309 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff00173f0 con 0x7f30001021e0 2024-03-28T05:22:13.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.528+0000 7f3006325700 1 -- 172.21.15.23:0/3653542309 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3000199240 con 0x7f30001021e0 2024-03-28T05:22:13.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.528+0000 7f3006325700 1 -- 172.21.15.23:0/3653542309 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30001a5580 con 0x7f30001021e0 2024-03-28T05:22:13.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.528+0000 7f3006325700 1 -- 172.21.15.23:0/3653542309 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f30000628a0 con 0x7f30001021e0 2024-03-28T05:22:13.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.529+0000 7f2ffd7fa700 1 -- 172.21.15.23:0/3653542309 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f2ff002a070 con 0x7f30001021e0 2024-03-28T05:22:13.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.530+0000 7f2ffd7fa700 1 --2- 172.21.15.23:0/3653542309 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2fe80a29a0 0x7f2fe80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:13.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.530+0000 7f2fff7fe700 1 --2- 172.21.15.23:0/3653542309 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2fe80a29a0 0x7f2fe80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:13.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.530+0000 7f2ffd7fa700 1 -- 172.21.15.23:0/3653542309 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f2ff00cfd50 con 0x7f30001021e0 2024-03-28T05:22:13.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.530+0000 7f2ffd7fa700 1 --2- 172.21.15.23:0/3653542309 >> [v2:172.21.15.98:6800/555491234,v1:172.21.15.98:6801/555491234] conn(0x7f2fe80adb00 0x7f2fe80affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:13.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.530+0000 7f2ffd7fa700 1 -- 172.21.15.23:0/3653542309 --> [v2:172.21.15.98:6800/555491234,v1:172.21.15.98:6801/555491234] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2ff0052e50 con 0x7f2fe80adb00 2024-03-28T05:22:13.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.530+0000 7f2ffd7fa700 1 -- 172.21.15.23:0/3653542309 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ff00d0100 con 0x7f30001021e0 2024-03-28T05:22:13.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.531+0000 7f30048c2700 1 --2- 172.21.15.23:0/3653542309 >> [v2:172.21.15.98:6800/555491234,v1:172.21.15.98:6801/555491234] conn(0x7f2fe80adb00 0x7f2fe80affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:13.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.531+0000 7f2fff7fe700 1 --2- 172.21.15.23:0/3653542309 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2fe80a29a0 0x7f2fe80a4e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3000199e90 tx=0x7f2ff4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:13.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.531+0000 7f30048c2700 1 --2- 172.21.15.23:0/3653542309 >> [v2:172.21.15.98:6800/555491234,v1:172.21.15.98:6801/555491234] conn(0x7f2fe80adb00 0x7f2fe80affa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:13.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.532+0000 7f2ffd7fa700 1 -- 172.21.15.23:0/3653542309 <== osd.1 v2:172.21.15.98:6800/555491234 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2ff0052e50 con 0x7f2fe80adb00 2024-03-28T05:22:13.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.549+0000 7f3006325700 1 -- 172.21.15.23:0/3653542309 --> [v2:172.21.15.98:6800/555491234,v1:172.21.15.98:6801/555491234] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3000062cd0 con 0x7f2fe80adb00 2024-03-28T05:22:13.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.549+0000 7f2ffd7fa700 1 -- 172.21.15.23:0/3653542309 <== osd.1 v2:172.21.15.98:6800/555491234 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3000062cd0 con 0x7f2fe80adb00 2024-03-28T05:22:13.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.549+0000 7f3006325700 1 -- 172.21.15.23:0/3653542309 >> [v2:172.21.15.98:6800/555491234,v1:172.21.15.98:6801/555491234] conn(0x7f2fe80adb00 msgr2=0x7f2fe80affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:13.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.549+0000 7f3006325700 1 --2- 172.21.15.23:0/3653542309 >> [v2:172.21.15.98:6800/555491234,v1:172.21.15.98:6801/555491234] conn(0x7f2fe80adb00 0x7f2fe80affa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:13.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.549+0000 7f3006325700 1 -- 172.21.15.23:0/3653542309 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2fe80a29a0 msgr2=0x7f2fe80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:13.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.549+0000 7f3006325700 1 --2- 172.21.15.23:0/3653542309 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2fe80a29a0 0x7f2fe80a4e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3000199e90 tx=0x7f2ff4009260 comp rx=0 tx=0).stop 2024-03-28T05:22:13.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.549+0000 7f3006325700 1 -- 172.21.15.23:0/3653542309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30001021e0 msgr2=0x7f300019e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:13.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.549+0000 7f3006325700 1 --2- 172.21.15.23:0/3653542309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30001021e0 0x7f300019e600 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00098e0 tx=0x7f2ff000fa00 comp rx=0 tx=0).stop 2024-03-28T05:22:13.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.549+0000 7f30048c2700 1 -- 172.21.15.23:0/3653542309 reap_dead start 2024-03-28T05:22:13.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.550+0000 7f3006325700 1 -- 172.21.15.23:0/3653542309 shutdown_connections 2024-03-28T05:22:13.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.550+0000 7f3006325700 1 -- 172.21.15.23:0/3653542309 >> 172.21.15.23:0/3653542309 conn(0x7f30000792c0 msgr2=0x7f3000100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:13.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.550+0000 7f3006325700 1 -- 172.21.15.23:0/3653542309 shutdown_connections 2024-03-28T05:22:13.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.550+0000 7f3006325700 1 -- 172.21.15.23:0/3653542309 wait complete. 2024-03-28T05:22:13.724 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:13 smithi023 ceph-mon[31954]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-28T05:22:13.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.737+0000 7f084425c700 1 -- 172.21.15.23:0/3133244231 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f083c102210 msgr2=0x7f083c102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:13.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.737+0000 7f084425c700 1 --2- 172.21.15.23:0/3133244231 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f083c102210 0x7f083c102690 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f08380099c0 tx=0x7f0838009cd0 comp rx=0 tx=0).stop 2024-03-28T05:22:13.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.737+0000 7f084425c700 1 -- 172.21.15.23:0/3133244231 shutdown_connections 2024-03-28T05:22:13.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.737+0000 7f084425c700 1 --2- 172.21.15.23:0/3133244231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f083c102bd0 0x7f083c10f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:13.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.737+0000 7f084425c700 1 --2- 172.21.15.23:0/3133244231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c108220 0x7f083c108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:13.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.737+0000 7f084425c700 1 --2- 172.21.15.23:0/3133244231 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f083c102210 0x7f083c102690 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:13.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.737+0000 7f084425c700 1 -- 172.21.15.23:0/3133244231 >> 172.21.15.23:0/3133244231 conn(0x7f083c0792f0 msgr2=0x7f083c079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:13.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.737+0000 7f084425c700 1 -- 172.21.15.23:0/3133244231 shutdown_connections 2024-03-28T05:22:13.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.737+0000 7f084425c700 1 -- 172.21.15.23:0/3133244231 wait complete. 2024-03-28T05:22:13.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.738+0000 7f084425c700 1 Processor -- start 2024-03-28T05:22:13.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.738+0000 7f084425c700 1 -- start start 2024-03-28T05:22:13.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.738+0000 7f084425c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f083c102210 0x7f083c074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:13.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.738+0000 7f084425c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c102bd0 0x7f083c0749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:13.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.738+0000 7f084425c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f083c108220 0x7f083c074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:13.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.738+0000 7f084425c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c112520 con 0x7f083c102210 2024-03-28T05:22:13.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.738+0000 7f084425c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c1123a0 con 0x7f083c102bd0 2024-03-28T05:22:13.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.738+0000 7f084425c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c1126a0 con 0x7f083c108220 2024-03-28T05:22:13.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.738+0000 7f0841ff8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f083c102210 0x7f083c074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:13.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.738+0000 7f0841ff8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f083c102210 0x7f083c074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50518/0 (socket says 172.21.15.23:50518) 2024-03-28T05:22:13.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.738+0000 7f0841ff8700 1 -- 172.21.15.23:0/1413179162 learned_addr learned my addr 172.21.15.23:0/1413179162 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:13.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.738+0000 7f08427f9700 1 --2- 172.21.15.23:0/1413179162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f083c108220 0x7f083c074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:13.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.738+0000 7f0841ff8700 1 -- 172.21.15.23:0/1413179162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f083c108220 msgr2=0x7f083c074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:13.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.738+0000 7f08417f7700 1 --2- 172.21.15.23:0/1413179162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c102bd0 0x7f083c0749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:13.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.738+0000 7f0841ff8700 1 --2- 172.21.15.23:0/1413179162 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f083c108220 0x7f083c074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:13.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.738+0000 7f0841ff8700 1 -- 172.21.15.23:0/1413179162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c102bd0 msgr2=0x7f083c0749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:13.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.738+0000 7f0841ff8700 1 --2- 172.21.15.23:0/1413179162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c102bd0 0x7f083c0749d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:13.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.738+0000 7f0841ff8700 1 -- 172.21.15.23:0/1413179162 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083c1a6e00 con 0x7f083c102210 2024-03-28T05:22:13.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.739+0000 7f0841ff8700 1 --2- 172.21.15.23:0/1413179162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f083c102210 0x7f083c074490 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f082c00d950 tx=0x7f082c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:13.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.739+0000 7f0832ffd700 1 -- 172.21.15.23:0/1413179162 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c016070 con 0x7f083c102210 2024-03-28T05:22:13.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.739+0000 7f0832ffd700 1 -- 172.21.15.23:0/1413179162 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f082c009b40 con 0x7f083c102210 2024-03-28T05:22:13.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.739+0000 7f084425c700 1 -- 172.21.15.23:0/1413179162 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f083c1a70f0 con 0x7f083c102210 2024-03-28T05:22:13.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.739+0000 7f0832ffd700 1 -- 172.21.15.23:0/1413179162 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c00f620 con 0x7f083c102210 2024-03-28T05:22:13.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.739+0000 7f084425c700 1 -- 172.21.15.23:0/1413179162 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f083c1a7630 con 0x7f083c102210 2024-03-28T05:22:13.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.740+0000 7f084425c700 1 -- 172.21.15.23:0/1413179162 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f083c1016b0 con 0x7f083c102210 2024-03-28T05:22:13.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.741+0000 7f0832ffd700 1 -- 172.21.15.23:0/1413179162 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f082c019070 con 0x7f083c102210 2024-03-28T05:22:13.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.742+0000 7f0832ffd700 1 --2- 172.21.15.23:0/1413179162 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f08280a2920 0x7f08280a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:13.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.742+0000 7f08417f7700 1 --2- 172.21.15.23:0/1413179162 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f08280a2920 0x7f08280a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:13.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.742+0000 7f0832ffd700 1 -- 172.21.15.23:0/1413179162 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f082c0cef90 con 0x7f083c102210 2024-03-28T05:22:13.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.742+0000 7f0832ffd700 1 --2- 172.21.15.23:0/1413179162 >> [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176] conn(0x7f08280ada80 0x7f08280aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:13.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.742+0000 7f0832ffd700 1 -- 172.21.15.23:0/1413179162 --> [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f082c014e80 con 0x7f08280ada80 2024-03-28T05:22:13.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.742+0000 7f0832ffd700 1 -- 172.21.15.23:0/1413179162 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f082c0cf3e0 con 0x7f083c102210 2024-03-28T05:22:13.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.742+0000 7f08427f9700 1 --2- 172.21.15.23:0/1413179162 >> [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176] conn(0x7f08280ada80 0x7f08280aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:13.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.742+0000 7f08417f7700 1 --2- 172.21.15.23:0/1413179162 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f08280a2920 0x7f08280a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f08380098e0 tx=0x7f0838000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:13.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.743+0000 7f08427f9700 1 --2- 172.21.15.23:0/1413179162 >> [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176] conn(0x7f08280ada80 0x7f08280aff20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:13.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.743+0000 7f0832ffd700 1 -- 172.21.15.23:0/1413179162 <== osd.6 v2:172.21.15.23:6818/4188464176 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f082c014e80 con 0x7f08280ada80 2024-03-28T05:22:13.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.760+0000 7f084425c700 1 -- 172.21.15.23:0/1413179162 --> [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f083c1038d0 con 0x7f08280ada80 2024-03-28T05:22:13.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.760+0000 7f0832ffd700 1 -- 172.21.15.23:0/1413179162 <== osd.6 v2:172.21.15.23:6818/4188464176 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f083c1038d0 con 0x7f08280ada80 2024-03-28T05:22:13.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.760+0000 7f084425c700 1 -- 172.21.15.23:0/1413179162 >> [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176] conn(0x7f08280ada80 msgr2=0x7f08280aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:13.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.760+0000 7f084425c700 1 --2- 172.21.15.23:0/1413179162 >> [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176] conn(0x7f08280ada80 0x7f08280aff20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:13.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.760+0000 7f084425c700 1 -- 172.21.15.23:0/1413179162 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f08280a2920 msgr2=0x7f08280a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:13.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.760+0000 7f084425c700 1 --2- 172.21.15.23:0/1413179162 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f08280a2920 0x7f08280a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f08380098e0 tx=0x7f0838000bc0 comp rx=0 tx=0).stop 2024-03-28T05:22:13.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.760+0000 7f084425c700 1 -- 172.21.15.23:0/1413179162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f083c102210 msgr2=0x7f083c074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:13.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.760+0000 7f084425c700 1 --2- 172.21.15.23:0/1413179162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f083c102210 0x7f083c074490 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f082c00d950 tx=0x7f082c00dd10 comp rx=0 tx=0).stop 2024-03-28T05:22:13.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.760+0000 7f08427f9700 1 -- 172.21.15.23:0/1413179162 reap_dead start 2024-03-28T05:22:13.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.760+0000 7f084425c700 1 -- 172.21.15.23:0/1413179162 shutdown_connections 2024-03-28T05:22:13.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.760+0000 7f084425c700 1 -- 172.21.15.23:0/1413179162 >> 172.21.15.23:0/1413179162 conn(0x7f083c0792f0 msgr2=0x7f083c100c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:13.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.760+0000 7f084425c700 1 -- 172.21.15.23:0/1413179162 shutdown_connections 2024-03-28T05:22:13.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:13.761+0000 7f084425c700 1 -- 172.21.15.23:0/1413179162 wait complete. 2024-03-28T05:22:13.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:13 smithi098 ceph-mon[36087]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-28T05:22:13.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:13 smithi113 ceph-mon[37164]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-28T05:22:14.367 INFO:teuthology.orchestra.run.smithi023.stdout:55834574924 2024-03-28T05:22:14.367 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-28T05:22:14.490 INFO:teuthology.orchestra.run.smithi023.stdout:309237645342 2024-03-28T05:22:14.491 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-28T05:22:14.707 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:14 smithi023 ceph-mon[31954]: pgmap v323: 129 pgs: 1 activating, 128 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s; 14 B/s, 0 objects/s recovering 2024-03-28T05:22:14.865 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:14 smithi098 ceph-mon[36087]: pgmap v323: 129 pgs: 1 activating, 128 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s; 14 B/s, 0 objects/s recovering 2024-03-28T05:22:14.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:14 smithi113 ceph-mon[37164]: pgmap v323: 129 pgs: 1 activating, 128 active+clean; 639 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s; 14 B/s, 0 objects/s recovering 2024-03-28T05:22:15.005 INFO:teuthology.orchestra.run.smithi023.stdout:107374182462 2024-03-28T05:22:15.005 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-28T05:22:15.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.761+0000 7f1653248700 1 -- 172.21.15.23:0/4293893955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f164c102e20 msgr2=0x7f164c106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:15.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.761+0000 7f1653248700 1 --2- 172.21.15.23:0/4293893955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f164c102e20 0x7f164c106df0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f16380099c0 tx=0x7f1638009cd0 comp rx=0 tx=0).stop 2024-03-28T05:22:15.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.761+0000 7f1653248700 1 -- 172.21.15.23:0/4293893955 shutdown_connections 2024-03-28T05:22:15.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.761+0000 7f1653248700 1 --2- 172.21.15.23:0/4293893955 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f164c107520 0x7f164c10f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:15.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.761+0000 7f1653248700 1 --2- 172.21.15.23:0/4293893955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f164c102470 0x7f164c102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:15.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.761+0000 7f1653248700 1 --2- 172.21.15.23:0/4293893955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f164c102e20 0x7f164c106df0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:15.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.761+0000 7f1653248700 1 -- 172.21.15.23:0/4293893955 >> 172.21.15.23:0/4293893955 conn(0x7f164c0792b0 msgr2=0x7f164c0796c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:15.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.761+0000 7f1653248700 1 -- 172.21.15.23:0/4293893955 shutdown_connections 2024-03-28T05:22:15.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.761+0000 7f1653248700 1 -- 172.21.15.23:0/4293893955 wait complete. 2024-03-28T05:22:15.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.762+0000 7f1653248700 1 Processor -- start 2024-03-28T05:22:15.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.762+0000 7f1653248700 1 -- start start 2024-03-28T05:22:15.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.762+0000 7f1653248700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f164c102470 0x7f164c19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:15.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.762+0000 7f1653248700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f164c102e20 0x7f164c19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:15.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.762+0000 7f1653248700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f164c107520 0x7f164c1987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:15.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.762+0000 7f1653248700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164c112490 con 0x7f164c102e20 2024-03-28T05:22:15.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.762+0000 7f1653248700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164c112310 con 0x7f164c107520 2024-03-28T05:22:15.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.762+0000 7f1653248700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164c112610 con 0x7f164c102470 2024-03-28T05:22:15.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.762+0000 7f164bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f164c102e20 0x7f164c19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:15.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.762+0000 7f1650fe4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f164c102470 0x7f164c19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:15.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.762+0000 7f164bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f164c102e20 0x7f164c19ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50484/0 (socket says 172.21.15.23:50484) 2024-03-28T05:22:15.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.762+0000 7f164bfff700 1 -- 172.21.15.23:0/2779354795 learned_addr learned my addr 172.21.15.23:0/2779354795 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:15.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.762+0000 7f16517e5700 1 --2- 172.21.15.23:0/2779354795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f164c107520 0x7f164c1987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:15.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.763+0000 7f1650fe4700 1 -- 172.21.15.23:0/2779354795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f164c107520 msgr2=0x7f164c1987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:15.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.763+0000 7f1650fe4700 1 --2- 172.21.15.23:0/2779354795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f164c107520 0x7f164c1987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:15.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.763+0000 7f1650fe4700 1 -- 172.21.15.23:0/2779354795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f164c102e20 msgr2=0x7f164c19ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:15.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.763+0000 7f1650fe4700 1 --2- 172.21.15.23:0/2779354795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f164c102e20 0x7f164c19ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:15.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.763+0000 7f1650fe4700 1 -- 172.21.15.23:0/2779354795 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164c198f70 con 0x7f164c102470 2024-03-28T05:22:15.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.763+0000 7f164bfff700 1 --2- 172.21.15.23:0/2779354795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f164c102e20 0x7f164c19ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:22:15.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.763+0000 7f1650fe4700 1 --2- 172.21.15.23:0/2779354795 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f164c102470 0x7f164c19e670 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f164000d950 tx=0x7f164000dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:15.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.763+0000 7f1649ffb700 1 -- 172.21.15.23:0/2779354795 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1640016070 con 0x7f164c102470 2024-03-28T05:22:15.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.763+0000 7f1649ffb700 1 -- 172.21.15.23:0/2779354795 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f164000fd70 con 0x7f164c102470 2024-03-28T05:22:15.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.763+0000 7f1653248700 1 -- 172.21.15.23:0/2779354795 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f164c199260 con 0x7f164c102470 2024-03-28T05:22:15.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.763+0000 7f1649ffb700 1 -- 172.21.15.23:0/2779354795 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164000e3b0 con 0x7f164c102470 2024-03-28T05:22:15.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.763+0000 7f1653248700 1 -- 172.21.15.23:0/2779354795 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f164c1a5490 con 0x7f164c102470 2024-03-28T05:22:15.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.764+0000 7f1653248700 1 -- 172.21.15.23:0/2779354795 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f164c0628a0 con 0x7f164c102470 2024-03-28T05:22:15.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.765+0000 7f1649ffb700 1 -- 172.21.15.23:0/2779354795 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f1640019070 con 0x7f164c102470 2024-03-28T05:22:15.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.765+0000 7f1649ffb700 1 --2- 172.21.15.23:0/2779354795 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f163c0a2920 0x7f163c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:15.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.766+0000 7f164bfff700 1 --2- 172.21.15.23:0/2779354795 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f163c0a2920 0x7f163c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:15.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.766+0000 7f1649ffb700 1 -- 172.21.15.23:0/2779354795 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f16400cede0 con 0x7f164c102470 2024-03-28T05:22:15.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.766+0000 7f1649ffb700 1 --2- 172.21.15.23:0/2779354795 >> [v2:172.21.15.98:6808/2615902819,v1:172.21.15.98:6809/2615902819] conn(0x7f163c0ada00 0x7f163c0afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:15.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.766+0000 7f1649ffb700 1 -- 172.21.15.23:0/2779354795 --> [v2:172.21.15.98:6808/2615902819,v1:172.21.15.98:6809/2615902819] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1640022e70 con 0x7f163c0ada00 2024-03-28T05:22:15.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.766+0000 7f1649ffb700 1 -- 172.21.15.23:0/2779354795 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16400cf190 con 0x7f164c102470 2024-03-28T05:22:15.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.766+0000 7f164bfff700 1 --2- 172.21.15.23:0/2779354795 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f163c0a2920 0x7f163c0a4de0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f164c199c40 tx=0x7f1638005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:15.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.766+0000 7f16517e5700 1 --2- 172.21.15.23:0/2779354795 >> [v2:172.21.15.98:6808/2615902819,v1:172.21.15.98:6809/2615902819] conn(0x7f163c0ada00 0x7f163c0afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:15.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.767+0000 7f16517e5700 1 --2- 172.21.15.23:0/2779354795 >> [v2:172.21.15.98:6808/2615902819,v1:172.21.15.98:6809/2615902819] conn(0x7f163c0ada00 0x7f163c0afea0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:15.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.768+0000 7f1649ffb700 1 -- 172.21.15.23:0/2779354795 <== osd.4 v2:172.21.15.98:6808/2615902819 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1640022e70 con 0x7f163c0ada00 2024-03-28T05:22:15.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.785+0000 7f1653248700 1 -- 172.21.15.23:0/2779354795 --> [v2:172.21.15.98:6808/2615902819,v1:172.21.15.98:6809/2615902819] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f164c062ca0 con 0x7f163c0ada00 2024-03-28T05:22:15.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.786+0000 7f1649ffb700 1 -- 172.21.15.23:0/2779354795 <== osd.4 v2:172.21.15.98:6808/2615902819 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f164c062ca0 con 0x7f163c0ada00 2024-03-28T05:22:15.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.786+0000 7f1653248700 1 -- 172.21.15.23:0/2779354795 >> [v2:172.21.15.98:6808/2615902819,v1:172.21.15.98:6809/2615902819] conn(0x7f163c0ada00 msgr2=0x7f163c0afea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:15.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.786+0000 7f1653248700 1 --2- 172.21.15.23:0/2779354795 >> [v2:172.21.15.98:6808/2615902819,v1:172.21.15.98:6809/2615902819] conn(0x7f163c0ada00 0x7f163c0afea0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:15.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.786+0000 7f1653248700 1 -- 172.21.15.23:0/2779354795 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f163c0a2920 msgr2=0x7f163c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:15.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.786+0000 7f1653248700 1 --2- 172.21.15.23:0/2779354795 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f163c0a2920 0x7f163c0a4de0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f164c199c40 tx=0x7f1638005fb0 comp rx=0 tx=0).stop 2024-03-28T05:22:15.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.786+0000 7f1653248700 1 -- 172.21.15.23:0/2779354795 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f164c102470 msgr2=0x7f164c19e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:15.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.786+0000 7f1653248700 1 --2- 172.21.15.23:0/2779354795 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f164c102470 0x7f164c19e670 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f164000d950 tx=0x7f164000dc60 comp rx=0 tx=0).stop 2024-03-28T05:22:15.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.786+0000 7f16517e5700 1 -- 172.21.15.23:0/2779354795 reap_dead start 2024-03-28T05:22:15.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.786+0000 7f1653248700 1 -- 172.21.15.23:0/2779354795 shutdown_connections 2024-03-28T05:22:15.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.786+0000 7f1653248700 1 -- 172.21.15.23:0/2779354795 >> 172.21.15.23:0/2779354795 conn(0x7f164c0792b0 msgr2=0x7f164c108650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:15.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.786+0000 7f1653248700 1 -- 172.21.15.23:0/2779354795 shutdown_connections 2024-03-28T05:22:15.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:12.786+0000 7f1653248700 1 -- 172.21.15.23:0/2779354795 wait complete. 2024-03-28T05:22:15.679 INFO:teuthology.orchestra.run.smithi023.stdout:141733920829 2024-03-28T05:22:15.680 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-28T05:22:15.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:15 smithi113 ceph-mon[37164]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-28T05:22:15.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:15 smithi113 ceph-mon[37164]: Cluster is now healthy 2024-03-28T05:22:16.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:15 smithi023 ceph-mon[31954]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-28T05:22:16.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:15 smithi023 ceph-mon[31954]: Cluster is now healthy 2024-03-28T05:22:16.114 INFO:teuthology.orchestra.run.smithi023.stdout:77309411399 2024-03-28T05:22:16.114 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-28T05:22:16.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:15 smithi098 ceph-mon[36087]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-28T05:22:16.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:15 smithi098 ceph-mon[36087]: Cluster is now healthy 2024-03-28T05:22:16.402 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:22:16.403 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:22:16.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:16 smithi023 ceph-mon[31954]: pgmap v324: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 14 B/s, 0 objects/s recovering 2024-03-28T05:22:16.848 INFO:teuthology.orchestra.run.smithi023.stdout:197568495667 2024-03-28T05:22:16.848 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-28T05:22:16.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:16 smithi113 ceph-mon[37164]: pgmap v324: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 14 B/s, 0 objects/s recovering 2024-03-28T05:22:17.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:16 smithi098 ceph-mon[36087]: pgmap v324: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 14 B/s, 0 objects/s recovering 2024-03-28T05:22:17.516 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:22:17.516 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:22:17.516 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:22:17.519 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:22:17.519 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:22:17.524 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:22:17.546 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:22:18.917 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:18 smithi098 ceph-mon[36087]: pgmap v325: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 14 B/s, 0 objects/s recovering 2024-03-28T05:22:18.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:18 smithi023 ceph-mon[31954]: pgmap v325: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 14 B/s, 0 objects/s recovering 2024-03-28T05:22:18.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:18 smithi113 ceph-mon[37164]: pgmap v325: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 14 B/s, 0 objects/s recovering 2024-03-28T05:22:20.735 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:20 smithi023 ceph-mon[31954]: pgmap v326: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 12 B/s, 0 objects/s recovering 2024-03-28T05:22:20.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:20 smithi113 ceph-mon[37164]: pgmap v326: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 12 B/s, 0 objects/s recovering 2024-03-28T05:22:21.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:20 smithi098 ceph-mon[36087]: pgmap v326: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 12 B/s, 0 objects/s recovering 2024-03-28T05:22:21.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.150+0000 7f3a4119e700 1 -- 172.21.15.23:0/3346691526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a3c10eca0 msgr2=0x7f3a3c111090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:21.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.150+0000 7f3a4119e700 1 --2- 172.21.15.23:0/3346691526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a3c10eca0 0x7f3a3c111090 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3a3400b0a0 tx=0x7f3a3400b3b0 comp rx=0 tx=0).stop 2024-03-28T05:22:21.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.150+0000 7f3a4119e700 1 -- 172.21.15.23:0/3346691526 shutdown_connections 2024-03-28T05:22:21.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.150+0000 7f3a4119e700 1 --2- 172.21.15.23:0/3346691526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a3c10eca0 0x7f3a3c111090 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:21.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.150+0000 7f3a4119e700 1 --2- 172.21.15.23:0/3346691526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a3c06a890 0x7f3a3c06ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:21.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.150+0000 7f3a4119e700 1 --2- 172.21.15.23:0/3346691526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a3c06b1b0 0x7f3a3c10e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:21.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.150+0000 7f3a4119e700 1 -- 172.21.15.23:0/3346691526 >> 172.21.15.23:0/3346691526 conn(0x7f3a3c0770e0 msgr2=0x7f3a3c0774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:21.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.150+0000 7f3a4119e700 1 -- 172.21.15.23:0/3346691526 shutdown_connections 2024-03-28T05:22:21.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.150+0000 7f3a4119e700 1 -- 172.21.15.23:0/3346691526 wait complete. 2024-03-28T05:22:21.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.150+0000 7f3a4119e700 1 Processor -- start 2024-03-28T05:22:21.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.151+0000 7f3a4119e700 1 -- start start 2024-03-28T05:22:21.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.151+0000 7f3a4119e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a3c06a890 0x7f3a3c1a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:21.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.151+0000 7f3a4119e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a3c06b1b0 0x7f3a3c1a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:21.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.151+0000 7f3a4119e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a3c10eca0 0x7f3a3c19a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:21.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.151+0000 7f3a4119e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c1146e0 con 0x7f3a3c10eca0 2024-03-28T05:22:21.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.151+0000 7f3a4119e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c114560 con 0x7f3a3c06b1b0 2024-03-28T05:22:21.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.151+0000 7f3a4119e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c114860 con 0x7f3a3c06a890 2024-03-28T05:22:21.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.151+0000 7f3a3b59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a3c10eca0 0x7f3a3c19a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:21.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.151+0000 7f3a3b59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a3c10eca0 0x7f3a3c19a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50532/0 (socket says 172.21.15.23:50532) 2024-03-28T05:22:21.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.151+0000 7f3a3b59e700 1 -- 172.21.15.23:0/3797172289 learned_addr learned my addr 172.21.15.23:0/3797172289 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:21.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.151+0000 7f3a3ad9d700 1 --2- 172.21.15.23:0/3797172289 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a3c06a890 0x7f3a3c1a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:21.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.151+0000 7f3a3a59c700 1 --2- 172.21.15.23:0/3797172289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a3c06b1b0 0x7f3a3c1a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:21.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.151+0000 7f3a3b59e700 1 -- 172.21.15.23:0/3797172289 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a3c06a890 msgr2=0x7f3a3c1a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:21.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.151+0000 7f3a3b59e700 1 --2- 172.21.15.23:0/3797172289 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a3c06a890 0x7f3a3c1a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:21.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.151+0000 7f3a3b59e700 1 -- 172.21.15.23:0/3797172289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a3c06b1b0 msgr2=0x7f3a3c1a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:21.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.151+0000 7f3a3b59e700 1 --2- 172.21.15.23:0/3797172289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a3c06b1b0 0x7f3a3c1a0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:21.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.151+0000 7f3a3b59e700 1 -- 172.21.15.23:0/3797172289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3c19b0c0 con 0x7f3a3c10eca0 2024-03-28T05:22:21.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.151+0000 7f3a3b59e700 1 --2- 172.21.15.23:0/3797172289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a3c10eca0 0x7f3a3c19a960 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3a34006070 tx=0x7f3a34004cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:21.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.152+0000 7f3a23fff700 1 -- 172.21.15.23:0/3797172289 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a34021070 con 0x7f3a3c10eca0 2024-03-28T05:22:21.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.152+0000 7f3a4119e700 1 -- 172.21.15.23:0/3797172289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a3c19b350 con 0x7f3a3c10eca0 2024-03-28T05:22:21.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.152+0000 7f3a23fff700 1 -- 172.21.15.23:0/3797172289 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a34003c00 con 0x7f3a3c10eca0 2024-03-28T05:22:21.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.152+0000 7f3a23fff700 1 -- 172.21.15.23:0/3797172289 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a34004790 con 0x7f3a3c10eca0 2024-03-28T05:22:21.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.152+0000 7f3a4119e700 1 -- 172.21.15.23:0/3797172289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a3c1a7620 con 0x7f3a3c10eca0 2024-03-28T05:22:21.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.153+0000 7f3a4119e700 1 -- 172.21.15.23:0/3797172289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a3c068e50 con 0x7f3a3c10eca0 2024-03-28T05:22:21.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.154+0000 7f3a23fff700 1 -- 172.21.15.23:0/3797172289 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f3a34003db0 con 0x7f3a3c10eca0 2024-03-28T05:22:21.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.154+0000 7f3a23fff700 1 --2- 172.21.15.23:0/3797172289 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3a240a2920 0x7f3a240a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:21.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.154+0000 7f3a3ad9d700 1 --2- 172.21.15.23:0/3797172289 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3a240a2920 0x7f3a240a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:21.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.155+0000 7f3a23fff700 1 -- 172.21.15.23:0/3797172289 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f3a3400f020 con 0x7f3a3c10eca0 2024-03-28T05:22:21.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.155+0000 7f3a3ad9d700 1 --2- 172.21.15.23:0/3797172289 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3a240a2920 0x7f3a240a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c100790 tx=0x7f3a30009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:21.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.158+0000 7f3a23fff700 1 -- 172.21.15.23:0/3797172289 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3a3402a020 con 0x7f3a3c10eca0 2024-03-28T05:22:21.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.367+0000 7f3a4119e700 1 -- 172.21.15.23:0/3797172289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f3a3c02da30 con 0x7f3a3c10eca0 2024-03-28T05:22:21.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.368+0000 7f3a23fff700 1 -- 172.21.15.23:0/3797172289 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f3a3409d310 con 0x7f3a3c10eca0 2024-03-28T05:22:21.366 INFO:teuthology.orchestra.run.smithi023.stdout:163208757305 2024-03-28T05:22:21.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.370+0000 7f3a4119e700 1 -- 172.21.15.23:0/3797172289 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3a240a2920 msgr2=0x7f3a240a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:21.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.370+0000 7f3a4119e700 1 --2- 172.21.15.23:0/3797172289 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3a240a2920 0x7f3a240a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c100790 tx=0x7f3a30009260 comp rx=0 tx=0).stop 2024-03-28T05:22:21.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.370+0000 7f3a4119e700 1 -- 172.21.15.23:0/3797172289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a3c10eca0 msgr2=0x7f3a3c19a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:21.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.370+0000 7f3a4119e700 1 --2- 172.21.15.23:0/3797172289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a3c10eca0 0x7f3a3c19a960 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3a34006070 tx=0x7f3a34004cf0 comp rx=0 tx=0).stop 2024-03-28T05:22:21.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.370+0000 7f3a4119e700 1 -- 172.21.15.23:0/3797172289 shutdown_connections 2024-03-28T05:22:21.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.370+0000 7f3a4119e700 1 --2- 172.21.15.23:0/3797172289 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3a240a2920 0x7f3a240a4de0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:21.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.370+0000 7f3a4119e700 1 --2- 172.21.15.23:0/3797172289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a3c10eca0 0x7f3a3c19a960 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:21.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.370+0000 7f3a4119e700 1 --2- 172.21.15.23:0/3797172289 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3a3c06a890 0x7f3a3c1a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:21.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.370+0000 7f3a4119e700 1 --2- 172.21.15.23:0/3797172289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a3c06b1b0 0x7f3a3c1a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:21.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.370+0000 7f3a4119e700 1 -- 172.21.15.23:0/3797172289 >> 172.21.15.23:0/3797172289 conn(0x7f3a3c0770e0 msgr2=0x7f3a3c110290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:21.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.370+0000 7f3a4119e700 1 -- 172.21.15.23:0/3797172289 shutdown_connections 2024-03-28T05:22:21.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:21.370+0000 7f3a4119e700 1 -- 172.21.15.23:0/3797172289 wait complete. 2024-03-28T05:22:21.988 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:21 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3797172289' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T05:22:21.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:21 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3797172289' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T05:22:22.016 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757301 got 163208757305 for osd.5 2024-03-28T05:22:22.016 DEBUG:teuthology.parallel:result is None 2024-03-28T05:22:22.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:21 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3797172289' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T05:22:22.903 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:22 smithi023 ceph-mon[31954]: pgmap v327: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1001 B/s rd, 1 op/s; 11 B/s, 0 objects/s recovering 2024-03-28T05:22:22.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:22 smithi113 ceph-mon[37164]: pgmap v327: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1001 B/s rd, 1 op/s; 11 B/s, 0 objects/s recovering 2024-03-28T05:22:23.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:22 smithi098 ceph-mon[36087]: pgmap v327: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1001 B/s rd, 1 op/s; 11 B/s, 0 objects/s recovering 2024-03-28T05:22:24.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:24 smithi113 ceph-mon[37164]: pgmap v328: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s; 9 B/s, 0 objects/s recovering 2024-03-28T05:22:25.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:24 smithi023 ceph-mon[31954]: pgmap v328: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s; 9 B/s, 0 objects/s recovering 2024-03-28T05:22:25.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:24 smithi098 ceph-mon[36087]: pgmap v328: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s; 9 B/s, 0 objects/s recovering 2024-03-28T05:22:26.835 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:26 smithi023 ceph-mon[31954]: pgmap v329: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:22:26.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.939+0000 7fc3889ba700 1 -- 172.21.15.23:0/2744953357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc384101a30 msgr2=0x7fc38410ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:26.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.939+0000 7fc3889ba700 1 --2- 172.21.15.23:0/2744953357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc384101a30 0x7fc38410ded0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc37c00b0a0 tx=0x7fc37c00b3b0 comp rx=0 tx=0).stop 2024-03-28T05:22:26.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.940+0000 7fc3889ba700 1 -- 172.21.15.23:0/2744953357 shutdown_connections 2024-03-28T05:22:26.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.940+0000 7fc3889ba700 1 --2- 172.21.15.23:0/2744953357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc384101a30 0x7fc38410ded0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:26.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.940+0000 7fc3889ba700 1 --2- 172.21.15.23:0/2744953357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc384107120 0x7fc384107500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:26.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.940+0000 7fc3889ba700 1 --2- 172.21.15.23:0/2744953357 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc384101070 0x7fc3841014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:26.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.940+0000 7fc3889ba700 1 -- 172.21.15.23:0/2744953357 >> 172.21.15.23:0/2744953357 conn(0x7fc3840780a0 msgr2=0x7fc3840784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:26.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.940+0000 7fc3889ba700 1 -- 172.21.15.23:0/2744953357 shutdown_connections 2024-03-28T05:22:26.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.940+0000 7fc3889ba700 1 -- 172.21.15.23:0/2744953357 wait complete. 2024-03-28T05:22:26.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.940+0000 7fc3889ba700 1 Processor -- start 2024-03-28T05:22:26.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.940+0000 7fc3889ba700 1 -- start start 2024-03-28T05:22:26.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.941+0000 7fc3889ba700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc384101070 0x7fc384102d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:26.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.941+0000 7fc3889ba700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc384101a30 0x7fc384103260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:26.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.941+0000 7fc3889ba700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc384107120 0x7fc3841037a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:26.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.941+0000 7fc3889ba700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3841113e0 con 0x7fc384101070 2024-03-28T05:22:26.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.941+0000 7fc3889ba700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc384111260 con 0x7fc384101a30 2024-03-28T05:22:26.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.941+0000 7fc3889ba700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc384111560 con 0x7fc384107120 2024-03-28T05:22:26.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.941+0000 7fc38259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc384101070 0x7fc384102d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:26.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.941+0000 7fc38259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc384101070 0x7fc384102d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56454/0 (socket says 172.21.15.23:56454) 2024-03-28T05:22:26.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.941+0000 7fc38259c700 1 -- 172.21.15.23:0/3037968942 learned_addr learned my addr 172.21.15.23:0/3037968942 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:26.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.941+0000 7fc382d9d700 1 --2- 172.21.15.23:0/3037968942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc384107120 0x7fc3841037a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:26.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.941+0000 7fc381d9b700 1 --2- 172.21.15.23:0/3037968942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc384101a30 0x7fc384103260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:26.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.941+0000 7fc38259c700 1 -- 172.21.15.23:0/3037968942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc384107120 msgr2=0x7fc3841037a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:26.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.941+0000 7fc38259c700 1 --2- 172.21.15.23:0/3037968942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc384107120 0x7fc3841037a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:26.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.941+0000 7fc38259c700 1 -- 172.21.15.23:0/3037968942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc384101a30 msgr2=0x7fc384103260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:26.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.941+0000 7fc38259c700 1 --2- 172.21.15.23:0/3037968942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc384101a30 0x7fc384103260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:26.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.941+0000 7fc38259c700 1 -- 172.21.15.23:0/3037968942 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3841a6d90 con 0x7fc384101070 2024-03-28T05:22:26.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.941+0000 7fc38259c700 1 --2- 172.21.15.23:0/3037968942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc384101070 0x7fc384102d20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc37400b530 tx=0x7fc37400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:26.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.941+0000 7fc3737fe700 1 -- 172.21.15.23:0/3037968942 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc374015020 con 0x7fc384101070 2024-03-28T05:22:26.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.942+0000 7fc3889ba700 1 -- 172.21.15.23:0/3037968942 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3841a7080 con 0x7fc384101070 2024-03-28T05:22:26.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.942+0000 7fc3737fe700 1 -- 172.21.15.23:0/3037968942 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc374011d10 con 0x7fc384101070 2024-03-28T05:22:26.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.942+0000 7fc3737fe700 1 -- 172.21.15.23:0/3037968942 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37400f2e0 con 0x7fc384101070 2024-03-28T05:22:26.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.942+0000 7fc3889ba700 1 -- 172.21.15.23:0/3037968942 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3841a75c0 con 0x7fc384101070 2024-03-28T05:22:26.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.943+0000 7fc3889ba700 1 -- 172.21.15.23:0/3037968942 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc384068e50 con 0x7fc384101070 2024-03-28T05:22:26.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.944+0000 7fc3737fe700 1 -- 172.21.15.23:0/3037968942 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fc374018020 con 0x7fc384101070 2024-03-28T05:22:26.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.944+0000 7fc3737fe700 1 --2- 172.21.15.23:0/3037968942 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc36c0a2920 0x7fc36c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:26.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.944+0000 7fc381d9b700 1 --2- 172.21.15.23:0/3037968942 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc36c0a2920 0x7fc36c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:26.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.944+0000 7fc3737fe700 1 -- 172.21.15.23:0/3037968942 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fc3740ceae0 con 0x7fc384101070 2024-03-28T05:22:26.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.945+0000 7fc381d9b700 1 --2- 172.21.15.23:0/3037968942 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc36c0a2920 0x7fc36c0a4de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc384078610 tx=0x7fc378009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:26.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:26.948+0000 7fc3737fe700 1 -- 172.21.15.23:0/3037968942 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc37401f020 con 0x7fc384101070 2024-03-28T05:22:26.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:26 smithi113 ceph-mon[37164]: pgmap v329: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:22:27.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:26 smithi098 ceph-mon[36087]: pgmap v329: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:22:27.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.161+0000 7fc3889ba700 1 -- 172.21.15.23:0/3037968942 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7fc384104250 con 0x7fc384101070 2024-03-28T05:22:27.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.161+0000 7fc3737fe700 1 -- 172.21.15.23:0/3037968942 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fc384104250 con 0x7fc384101070 2024-03-28T05:22:27.159 INFO:teuthology.orchestra.run.smithi023.stdout:343597383707 2024-03-28T05:22:27.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.163+0000 7fc3889ba700 1 -- 172.21.15.23:0/3037968942 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc36c0a2920 msgr2=0x7fc36c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:27.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.163+0000 7fc3889ba700 1 --2- 172.21.15.23:0/3037968942 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc36c0a2920 0x7fc36c0a4de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc384078610 tx=0x7fc378009260 comp rx=0 tx=0).stop 2024-03-28T05:22:27.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.163+0000 7fc3889ba700 1 -- 172.21.15.23:0/3037968942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc384101070 msgr2=0x7fc384102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:27.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.163+0000 7fc3889ba700 1 --2- 172.21.15.23:0/3037968942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc384101070 0x7fc384102d20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc37400b530 tx=0x7fc37400b840 comp rx=0 tx=0).stop 2024-03-28T05:22:27.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.163+0000 7fc3889ba700 1 -- 172.21.15.23:0/3037968942 shutdown_connections 2024-03-28T05:22:27.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.163+0000 7fc3889ba700 1 --2- 172.21.15.23:0/3037968942 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc36c0a2920 0x7fc36c0a4de0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:27.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.163+0000 7fc3889ba700 1 --2- 172.21.15.23:0/3037968942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc384107120 0x7fc3841037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:27.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.163+0000 7fc3889ba700 1 --2- 172.21.15.23:0/3037968942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc384101070 0x7fc384102d20 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:27.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.163+0000 7fc3889ba700 1 --2- 172.21.15.23:0/3037968942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc384101a30 0x7fc384103260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:27.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.163+0000 7fc3889ba700 1 -- 172.21.15.23:0/3037968942 >> 172.21.15.23:0/3037968942 conn(0x7fc3840780a0 msgr2=0x7fc384100360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:27.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.164+0000 7fc3889ba700 1 -- 172.21.15.23:0/3037968942 shutdown_connections 2024-03-28T05:22:27.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.164+0000 7fc3889ba700 1 -- 172.21.15.23:0/3037968942 wait complete. 2024-03-28T05:22:27.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.158+0000 7fc174193700 1 -- 172.21.15.23:0/2827865151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc16c1023d0 msgr2=0x7fc16c1027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:27.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.158+0000 7fc174193700 1 --2- 172.21.15.23:0/2827865151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc16c1023d0 0x7fc16c1027b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc168009920 tx=0x7fc168009c30 comp rx=0 tx=0).stop 2024-03-28T05:22:27.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.158+0000 7fc174193700 1 -- 172.21.15.23:0/2827865151 shutdown_connections 2024-03-28T05:22:27.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.158+0000 7fc174193700 1 --2- 172.21.15.23:0/2827865151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc16c1074b0 0x7fc16c10f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:27.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.158+0000 7fc174193700 1 --2- 172.21.15.23:0/2827865151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc16c1023d0 0x7fc16c1027b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:27.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.158+0000 7fc174193700 1 --2- 172.21.15.23:0/2827865151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc16c102d80 0x7fc16c106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:27.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.158+0000 7fc174193700 1 -- 172.21.15.23:0/2827865151 >> 172.21.15.23:0/2827865151 conn(0x7fc16c0fdc40 msgr2=0x7fc16c100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:27.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.158+0000 7fc174193700 1 -- 172.21.15.23:0/2827865151 shutdown_connections 2024-03-28T05:22:27.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.159+0000 7fc174193700 1 -- 172.21.15.23:0/2827865151 wait complete. 2024-03-28T05:22:27.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.159+0000 7fc174193700 1 Processor -- start 2024-03-28T05:22:27.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.159+0000 7fc174193700 1 -- start start 2024-03-28T05:22:27.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.160+0000 7fc174193700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc16c1023d0 0x7fc16c19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:27.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.160+0000 7fc174193700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc16c102d80 0x7fc16c19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:27.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.160+0000 7fc174193700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc16c1074b0 0x7fc16c198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:27.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.160+0000 7fc171f2f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc16c1023d0 0x7fc16c19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:27.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.160+0000 7fc174193700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c1124b0 con 0x7fc16c1023d0 2024-03-28T05:22:27.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.160+0000 7fc171f2f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc16c1023d0 0x7fc16c19e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56474/0 (socket says 172.21.15.23:56474) 2024-03-28T05:22:27.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.160+0000 7fc174193700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c112330 con 0x7fc16c1074b0 2024-03-28T05:22:27.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.160+0000 7fc171f2f700 1 -- 172.21.15.23:0/1123219852 learned_addr learned my addr 172.21.15.23:0/1123219852 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:27.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.160+0000 7fc174193700 1 -- 172.21.15.23:0/1123219852 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c112630 con 0x7fc16c102d80 2024-03-28T05:22:27.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.160+0000 7fc17172e700 1 --2- 172.21.15.23:0/1123219852 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc16c102d80 0x7fc16c19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:27.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.160+0000 7fc17172e700 1 -- 172.21.15.23:0/1123219852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc16c1074b0 msgr2=0x7fc16c198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:27.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.160+0000 7fc17172e700 1 --2- 172.21.15.23:0/1123219852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc16c1074b0 0x7fc16c198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:27.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.160+0000 7fc17172e700 1 -- 172.21.15.23:0/1123219852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc16c1023d0 msgr2=0x7fc16c19e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:27.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.160+0000 7fc17172e700 1 --2- 172.21.15.23:0/1123219852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc16c1023d0 0x7fc16c19e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:27.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.160+0000 7fc17172e700 1 -- 172.21.15.23:0/1123219852 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16c198ff0 con 0x7fc16c102d80 2024-03-28T05:22:27.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.160+0000 7fc17172e700 1 --2- 172.21.15.23:0/1123219852 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc16c102d80 0x7fc16c19eb80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc15c00d720 tx=0x7fc15c00da30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:27.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.160+0000 7fc162ffd700 1 -- 172.21.15.23:0/1123219852 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c016070 con 0x7fc16c102d80 2024-03-28T05:22:27.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.161+0000 7fc162ffd700 1 -- 172.21.15.23:0/1123219852 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc15c005020 con 0x7fc16c102d80 2024-03-28T05:22:27.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.161+0000 7fc174193700 1 -- 172.21.15.23:0/1123219852 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc16c1992e0 con 0x7fc16c102d80 2024-03-28T05:22:27.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.161+0000 7fc162ffd700 1 -- 172.21.15.23:0/1123219852 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c00fb30 con 0x7fc16c102d80 2024-03-28T05:22:27.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.161+0000 7fc174193700 1 -- 172.21.15.23:0/1123219852 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc16c1a54a0 con 0x7fc16c102d80 2024-03-28T05:22:27.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.161+0000 7fc174193700 1 -- 172.21.15.23:0/1123219852 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc16c1038b0 con 0x7fc16c102d80 2024-03-28T05:22:27.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.163+0000 7fc162ffd700 1 -- 172.21.15.23:0/1123219852 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fc15c0051d0 con 0x7fc16c102d80 2024-03-28T05:22:27.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.163+0000 7fc162ffd700 1 --2- 172.21.15.23:0/1123219852 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc1580a29a0 0x7fc1580a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:27.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.163+0000 7fc162ffd700 1 -- 172.21.15.23:0/1123219852 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fc15c0ced40 con 0x7fc16c102d80 2024-03-28T05:22:27.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.163+0000 7fc171f2f700 1 --2- 172.21.15.23:0/1123219852 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc1580a29a0 0x7fc1580a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:27.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.164+0000 7fc171f2f700 1 --2- 172.21.15.23:0/1123219852 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc1580a29a0 0x7fc1580a4e60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc168009680 tx=0x7fc168019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:27.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.167+0000 7fc162ffd700 1 -- 172.21.15.23:0/1123219852 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc15c09b8c0 con 0x7fc16c102d80 2024-03-28T05:22:27.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.377+0000 7fc174193700 1 -- 172.21.15.23:0/1123219852 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc16c02dbf0 con 0x7fc16c102d80 2024-03-28T05:22:27.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.377+0000 7fc162ffd700 1 -- 172.21.15.23:0/1123219852 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc15c09b1c0 con 0x7fc16c102d80 2024-03-28T05:22:27.376 INFO:teuthology.orchestra.run.smithi023.stdout:55834574927 2024-03-28T05:22:27.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.379+0000 7fc174193700 1 -- 172.21.15.23:0/1123219852 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc1580a29a0 msgr2=0x7fc1580a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:27.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.379+0000 7fc174193700 1 --2- 172.21.15.23:0/1123219852 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc1580a29a0 0x7fc1580a4e60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc168009680 tx=0x7fc168019040 comp rx=0 tx=0).stop 2024-03-28T05:22:27.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.379+0000 7fc174193700 1 -- 172.21.15.23:0/1123219852 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc16c102d80 msgr2=0x7fc16c19eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:27.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.379+0000 7fc174193700 1 --2- 172.21.15.23:0/1123219852 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc16c102d80 0x7fc16c19eb80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc15c00d720 tx=0x7fc15c00da30 comp rx=0 tx=0).stop 2024-03-28T05:22:27.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.379+0000 7fc174193700 1 -- 172.21.15.23:0/1123219852 shutdown_connections 2024-03-28T05:22:27.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.379+0000 7fc174193700 1 --2- 172.21.15.23:0/1123219852 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc1580a29a0 0x7fc1580a4e60 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:27.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.379+0000 7fc174193700 1 --2- 172.21.15.23:0/1123219852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc16c1074b0 0x7fc16c198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:27.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.379+0000 7fc174193700 1 --2- 172.21.15.23:0/1123219852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc16c1023d0 0x7fc16c19e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:27.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.379+0000 7fc174193700 1 --2- 172.21.15.23:0/1123219852 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc16c102d80 0x7fc16c19eb80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:27.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.379+0000 7fc174193700 1 -- 172.21.15.23:0/1123219852 >> 172.21.15.23:0/1123219852 conn(0x7fc16c0fdc40 msgr2=0x7fc16c0fffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:27.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.380+0000 7fc174193700 1 -- 172.21.15.23:0/1123219852 shutdown_connections 2024-03-28T05:22:27.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:27.380+0000 7fc174193700 1 -- 172.21.15.23:0/1123219852 wait complete. 2024-03-28T05:22:27.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:27 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3037968942' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T05:22:27.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:27 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1123219852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T05:22:28.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:27 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3037968942' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T05:22:28.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:27 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1123219852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T05:22:28.094 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574924 got 55834574927 for osd.1 2024-03-28T05:22:28.094 DEBUG:teuthology.parallel:result is None 2024-03-28T05:22:28.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:27 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3037968942' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T05:22:28.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:27 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1123219852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T05:22:28.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:28 smithi113 ceph-mon[37164]: pgmap v330: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:29.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:28 smithi023 ceph-mon[31954]: pgmap v330: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:29.089 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383703 got 343597383707 for osd.11 2024-03-28T05:22:29.089 DEBUG:teuthology.parallel:result is None 2024-03-28T05:22:29.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:28 smithi098 ceph-mon[36087]: pgmap v330: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:30.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.538+0000 7fd4dca51700 1 -- 172.21.15.23:0/2124806248 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4d8102ba0 msgr2=0x7fd4d810efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:30.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.538+0000 7fd4dca51700 1 --2- 172.21.15.23:0/2124806248 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4d8102ba0 0x7fd4d810efe0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd4d000b0d0 tx=0x7fd4d000b3e0 comp rx=0 tx=0).stop 2024-03-28T05:22:30.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.538+0000 7fd4dca51700 1 -- 172.21.15.23:0/2124806248 shutdown_connections 2024-03-28T05:22:30.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.538+0000 7fd4dca51700 1 --2- 172.21.15.23:0/2124806248 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4d8102ba0 0x7fd4d810efe0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:30.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.538+0000 7fd4dca51700 1 --2- 172.21.15.23:0/2124806248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d81081c0 0x7fd4d81085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:30.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.538+0000 7fd4dca51700 1 --2- 172.21.15.23:0/2124806248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4d81021e0 0x7fd4d8102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:30.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.538+0000 7fd4dca51700 1 -- 172.21.15.23:0/2124806248 >> 172.21.15.23:0/2124806248 conn(0x7fd4d80792c0 msgr2=0x7fd4d80796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:30.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.538+0000 7fd4dca51700 1 -- 172.21.15.23:0/2124806248 shutdown_connections 2024-03-28T05:22:30.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.538+0000 7fd4dca51700 1 -- 172.21.15.23:0/2124806248 wait complete. 2024-03-28T05:22:30.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.539+0000 7fd4dca51700 1 Processor -- start 2024-03-28T05:22:30.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.539+0000 7fd4dca51700 1 -- start start 2024-03-28T05:22:30.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.539+0000 7fd4dca51700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4d81021e0 0x7fd4d819e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:30.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.539+0000 7fd4dca51700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4d8102ba0 0x7fd4d819e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:30.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.539+0000 7fd4dca51700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d81081c0 0x7fd4d8198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:30.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.539+0000 7fd4dca51700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d8112440 con 0x7fd4d8102ba0 2024-03-28T05:22:30.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.539+0000 7fd4dca51700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d81122c0 con 0x7fd4d81081c0 2024-03-28T05:22:30.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.539+0000 7fd4dca51700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d81125c0 con 0x7fd4d81021e0 2024-03-28T05:22:30.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.539+0000 7fd4d5d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4d8102ba0 0x7fd4d819e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:30.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.539+0000 7fd4d5d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4d8102ba0 0x7fd4d819e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56492/0 (socket says 172.21.15.23:56492) 2024-03-28T05:22:30.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.539+0000 7fd4d5d9b700 1 -- 172.21.15.23:0/714730707 learned_addr learned my addr 172.21.15.23:0/714730707 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:30.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.539+0000 7fd4d659c700 1 --2- 172.21.15.23:0/714730707 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4d81021e0 0x7fd4d819e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:30.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.539+0000 7fd4d6d9d700 1 --2- 172.21.15.23:0/714730707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d81081c0 0x7fd4d8198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:30.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.539+0000 7fd4d5d9b700 1 -- 172.21.15.23:0/714730707 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4d81021e0 msgr2=0x7fd4d819e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:30.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.539+0000 7fd4d5d9b700 1 --2- 172.21.15.23:0/714730707 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4d81021e0 0x7fd4d819e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:30.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.539+0000 7fd4d5d9b700 1 -- 172.21.15.23:0/714730707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d81081c0 msgr2=0x7fd4d8198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:30.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.539+0000 7fd4d5d9b700 1 --2- 172.21.15.23:0/714730707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d81081c0 0x7fd4d8198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:30.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.539+0000 7fd4d5d9b700 1 -- 172.21.15.23:0/714730707 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d8198d60 con 0x7fd4d8102ba0 2024-03-28T05:22:30.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.540+0000 7fd4d5d9b700 1 --2- 172.21.15.23:0/714730707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4d8102ba0 0x7fd4d819e980 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc00e9a0 tx=0x7fd4cc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:30.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.540+0000 7fd4c77fe700 1 -- 172.21.15.23:0/714730707 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc00ca10 con 0x7fd4d8102ba0 2024-03-28T05:22:30.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.540+0000 7fd4dca51700 1 -- 172.21.15.23:0/714730707 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d8199050 con 0x7fd4d8102ba0 2024-03-28T05:22:30.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.540+0000 7fd4c77fe700 1 -- 172.21.15.23:0/714730707 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd4cc00cbb0 con 0x7fd4d8102ba0 2024-03-28T05:22:30.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.540+0000 7fd4c77fe700 1 -- 172.21.15.23:0/714730707 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc018680 con 0x7fd4d8102ba0 2024-03-28T05:22:30.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.540+0000 7fd4dca51700 1 -- 172.21.15.23:0/714730707 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4d81a5310 con 0x7fd4d8102ba0 2024-03-28T05:22:30.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.541+0000 7fd4dca51700 1 -- 172.21.15.23:0/714730707 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4d8068e50 con 0x7fd4d8102ba0 2024-03-28T05:22:30.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.542+0000 7fd4c77fe700 1 -- 172.21.15.23:0/714730707 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fd4cc010430 con 0x7fd4d8102ba0 2024-03-28T05:22:30.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.542+0000 7fd4c77fe700 1 --2- 172.21.15.23:0/714730707 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd4c00a2920 0x7fd4c00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:30.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.543+0000 7fd4d659c700 1 --2- 172.21.15.23:0/714730707 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd4c00a2920 0x7fd4c00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:30.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.543+0000 7fd4c77fe700 1 -- 172.21.15.23:0/714730707 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fd4cc0cf680 con 0x7fd4d8102ba0 2024-03-28T05:22:30.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.543+0000 7fd4d659c700 1 --2- 172.21.15.23:0/714730707 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd4c00a2920 0x7fd4c00a4de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd4c8009c60 tx=0x7fd4c8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:30.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.546+0000 7fd4c77fe700 1 -- 172.21.15.23:0/714730707 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd4cc09c200 con 0x7fd4d8102ba0 2024-03-28T05:22:30.706 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:30 smithi023 ceph-mon[31954]: pgmap v331: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:30.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.707+0000 7ff61da93700 1 -- 172.21.15.23:0/1060909926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6181074b0 msgr2=0x7ff61810f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:30.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.707+0000 7ff61da93700 1 --2- 172.21.15.23:0/1060909926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6181074b0 0x7ff61810f080 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff61000b0d0 tx=0x7ff61000b3e0 comp rx=0 tx=0).stop 2024-03-28T05:22:30.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.707+0000 7ff61da93700 1 -- 172.21.15.23:0/1060909926 shutdown_connections 2024-03-28T05:22:30.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.707+0000 7ff61da93700 1 --2- 172.21.15.23:0/1060909926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6181074b0 0x7ff61810f080 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:30.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.707+0000 7ff61da93700 1 --2- 172.21.15.23:0/1060909926 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6181023d0 0x7ff6181027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:30.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.707+0000 7ff61da93700 1 --2- 172.21.15.23:0/1060909926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff618102d80 0x7ff618106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:30.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.707+0000 7ff61da93700 1 -- 172.21.15.23:0/1060909926 >> 172.21.15.23:0/1060909926 conn(0x7ff6180fdc40 msgr2=0x7ff618100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:30.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.707+0000 7ff61da93700 1 -- 172.21.15.23:0/1060909926 shutdown_connections 2024-03-28T05:22:30.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.707+0000 7ff61da93700 1 -- 172.21.15.23:0/1060909926 wait complete. 2024-03-28T05:22:30.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.708+0000 7ff61da93700 1 Processor -- start 2024-03-28T05:22:30.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.708+0000 7ff61da93700 1 -- start start 2024-03-28T05:22:30.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.708+0000 7ff61da93700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6181023d0 0x7ff61819e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:30.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.708+0000 7ff61da93700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff618102d80 0x7ff61819eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:30.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.708+0000 7ff61da93700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6181074b0 0x7ff6181986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:30.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.708+0000 7ff61da93700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6181124a0 con 0x7ff618102d80 2024-03-28T05:22:30.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.708+0000 7ff61da93700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff618112320 con 0x7ff6181023d0 2024-03-28T05:22:30.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.708+0000 7ff61da93700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff618112620 con 0x7ff6181074b0 2024-03-28T05:22:30.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.708+0000 7ff616ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff618102d80 0x7ff61819eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:30.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.708+0000 7ff617fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6181074b0 0x7ff6181986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:30.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.708+0000 7ff616ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff618102d80 0x7ff61819eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56500/0 (socket says 172.21.15.23:56500) 2024-03-28T05:22:30.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.708+0000 7ff616ffd700 1 -- 172.21.15.23:0/1357788792 learned_addr learned my addr 172.21.15.23:0/1357788792 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:30.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.708+0000 7ff6177fe700 1 --2- 172.21.15.23:0/1357788792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6181023d0 0x7ff61819e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:30.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.709+0000 7ff617fff700 1 -- 172.21.15.23:0/1357788792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6181023d0 msgr2=0x7ff61819e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:30.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.709+0000 7ff617fff700 1 --2- 172.21.15.23:0/1357788792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6181023d0 0x7ff61819e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:30.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.709+0000 7ff617fff700 1 -- 172.21.15.23:0/1357788792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff618102d80 msgr2=0x7ff61819eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:30.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.709+0000 7ff617fff700 1 --2- 172.21.15.23:0/1357788792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff618102d80 0x7ff61819eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:30.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.709+0000 7ff617fff700 1 -- 172.21.15.23:0/1357788792 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff618198fc0 con 0x7ff6181074b0 2024-03-28T05:22:30.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.709+0000 7ff617fff700 1 --2- 172.21.15.23:0/1357788792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6181074b0 0x7ff6181986e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff61000b770 tx=0x7ff610007a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:30.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.709+0000 7ff614ff9700 1 -- 172.21.15.23:0/1357788792 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff610022070 con 0x7ff6181074b0 2024-03-28T05:22:30.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.709+0000 7ff614ff9700 1 -- 172.21.15.23:0/1357788792 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff61001cda0 con 0x7ff6181074b0 2024-03-28T05:22:30.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.709+0000 7ff61da93700 1 -- 172.21.15.23:0/1357788792 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff618199250 con 0x7ff6181074b0 2024-03-28T05:22:30.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.709+0000 7ff614ff9700 1 -- 172.21.15.23:0/1357788792 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff610026bb0 con 0x7ff6181074b0 2024-03-28T05:22:30.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.709+0000 7ff61da93700 1 -- 172.21.15.23:0/1357788792 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6180743c0 con 0x7ff6181074b0 2024-03-28T05:22:30.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.710+0000 7ff61da93700 1 -- 172.21.15.23:0/1357788792 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6181038b0 con 0x7ff6181074b0 2024-03-28T05:22:30.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.711+0000 7ff614ff9700 1 -- 172.21.15.23:0/1357788792 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7ff61002d020 con 0x7ff6181074b0 2024-03-28T05:22:30.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.712+0000 7ff614ff9700 1 --2- 172.21.15.23:0/1357788792 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff6000a2920 0x7ff6000a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:30.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.712+0000 7ff6177fe700 1 --2- 172.21.15.23:0/1357788792 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff6000a2920 0x7ff6000a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:30.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.712+0000 7ff614ff9700 1 -- 172.21.15.23:0/1357788792 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7ff61000f020 con 0x7ff6181074b0 2024-03-28T05:22:30.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.712+0000 7ff6177fe700 1 --2- 172.21.15.23:0/1357788792 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff6000a2920 0x7ff6000a4de0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff60c009680 tx=0x7ff60c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:30.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.721+0000 7ff614ff9700 1 -- 172.21.15.23:0/1357788792 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff6100a0560 con 0x7ff6181074b0 2024-03-28T05:22:30.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.757+0000 7fd4dca51700 1 -- 172.21.15.23:0/714730707 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7fd4d81a5540 con 0x7fd4d8102ba0 2024-03-28T05:22:30.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.758+0000 7fd4c77fe700 1 -- 172.21.15.23:0/714730707 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fd4cc09bb00 con 0x7fd4d8102ba0 2024-03-28T05:22:30.756 INFO:teuthology.orchestra.run.smithi023.stdout:309237645345 2024-03-28T05:22:30.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.760+0000 7fd4dca51700 1 -- 172.21.15.23:0/714730707 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd4c00a2920 msgr2=0x7fd4c00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:30.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.760+0000 7fd4dca51700 1 --2- 172.21.15.23:0/714730707 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd4c00a2920 0x7fd4c00a4de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd4c8009c60 tx=0x7fd4c8009260 comp rx=0 tx=0).stop 2024-03-28T05:22:30.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.760+0000 7fd4dca51700 1 -- 172.21.15.23:0/714730707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4d8102ba0 msgr2=0x7fd4d819e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:30.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.760+0000 7fd4dca51700 1 --2- 172.21.15.23:0/714730707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4d8102ba0 0x7fd4d819e980 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc00e9a0 tx=0x7fd4cc00ecb0 comp rx=0 tx=0).stop 2024-03-28T05:22:30.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.760+0000 7fd4dca51700 1 -- 172.21.15.23:0/714730707 shutdown_connections 2024-03-28T05:22:30.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.760+0000 7fd4dca51700 1 --2- 172.21.15.23:0/714730707 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd4c00a2920 0x7fd4c00a4de0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:30.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.760+0000 7fd4dca51700 1 --2- 172.21.15.23:0/714730707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd4d81081c0 0x7fd4d8198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:30.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.760+0000 7fd4dca51700 1 --2- 172.21.15.23:0/714730707 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd4d81021e0 0x7fd4d819e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:30.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.760+0000 7fd4dca51700 1 --2- 172.21.15.23:0/714730707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4d8102ba0 0x7fd4d819e980 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:30.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.760+0000 7fd4dca51700 1 -- 172.21.15.23:0/714730707 >> 172.21.15.23:0/714730707 conn(0x7fd4d80792c0 msgr2=0x7fd4d8100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:30.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.760+0000 7fd4dca51700 1 -- 172.21.15.23:0/714730707 shutdown_connections 2024-03-28T05:22:30.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.760+0000 7fd4dca51700 1 -- 172.21.15.23:0/714730707 wait complete. 2024-03-28T05:22:30.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.929+0000 7ff61da93700 1 -- 172.21.15.23:0/1357788792 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7ff618068e50 con 0x7ff6181074b0 2024-03-28T05:22:30.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.929+0000 7ff614ff9700 1 -- 172.21.15.23:0/1357788792 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff61009fe60 con 0x7ff6181074b0 2024-03-28T05:22:30.928 INFO:teuthology.orchestra.run.smithi023.stdout:219043332145 2024-03-28T05:22:30.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.931+0000 7ff61da93700 1 -- 172.21.15.23:0/1357788792 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff6000a2920 msgr2=0x7ff6000a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:30.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.931+0000 7ff61da93700 1 --2- 172.21.15.23:0/1357788792 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff6000a2920 0x7ff6000a4de0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff60c009680 tx=0x7ff60c009260 comp rx=0 tx=0).stop 2024-03-28T05:22:30.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.931+0000 7ff61da93700 1 -- 172.21.15.23:0/1357788792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6181074b0 msgr2=0x7ff6181986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:30.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.931+0000 7ff61da93700 1 --2- 172.21.15.23:0/1357788792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6181074b0 0x7ff6181986e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff61000b770 tx=0x7ff610007a20 comp rx=0 tx=0).stop 2024-03-28T05:22:30.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.931+0000 7ff61da93700 1 -- 172.21.15.23:0/1357788792 shutdown_connections 2024-03-28T05:22:30.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.931+0000 7ff61da93700 1 --2- 172.21.15.23:0/1357788792 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff6000a2920 0x7ff6000a4de0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:30.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.931+0000 7ff61da93700 1 --2- 172.21.15.23:0/1357788792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff6181074b0 0x7ff6181986e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:30.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.931+0000 7ff61da93700 1 --2- 172.21.15.23:0/1357788792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6181023d0 0x7ff61819e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:30.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.931+0000 7ff61da93700 1 --2- 172.21.15.23:0/1357788792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff618102d80 0x7ff61819eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:30.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.931+0000 7ff61da93700 1 -- 172.21.15.23:0/1357788792 >> 172.21.15.23:0/1357788792 conn(0x7ff6180fdc40 msgr2=0x7ff6180fe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:30.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.931+0000 7ff61da93700 1 -- 172.21.15.23:0/1357788792 shutdown_connections 2024-03-28T05:22:30.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:30.931+0000 7ff61da93700 1 -- 172.21.15.23:0/1357788792 wait complete. 2024-03-28T05:22:30.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:30 smithi113 ceph-mon[37164]: pgmap v331: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:31.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:30 smithi098 ceph-mon[36087]: pgmap v331: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:31.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.612+0000 7f0dea176700 1 -- 172.21.15.23:0/95792148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0de4107260 msgr2=0x7f0de4107640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:31.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.612+0000 7f0dea176700 1 --2- 172.21.15.23:0/95792148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0de4107260 0x7f0de4107640 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0dd40099c0 tx=0x7f0dd4009cd0 comp rx=0 tx=0).stop 2024-03-28T05:22:31.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.612+0000 7f0dea176700 1 -- 172.21.15.23:0/95792148 shutdown_connections 2024-03-28T05:22:31.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.612+0000 7f0dea176700 1 --2- 172.21.15.23:0/95792148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0de4101550 0x7f0de410f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:31.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.612+0000 7f0dea176700 1 --2- 172.21.15.23:0/95792148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0de4107260 0x7f0de4107640 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:31.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.612+0000 7f0dea176700 1 --2- 172.21.15.23:0/95792148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0de4100b90 0x7f0de4101010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:31.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.612+0000 7f0dea176700 1 -- 172.21.15.23:0/95792148 >> 172.21.15.23:0/95792148 conn(0x7f0de40763e0 msgr2=0x7f0de40767f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:31.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.612+0000 7f0dea176700 1 -- 172.21.15.23:0/95792148 shutdown_connections 2024-03-28T05:22:31.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.613+0000 7f0dea176700 1 -- 172.21.15.23:0/95792148 wait complete. 2024-03-28T05:22:31.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.613+0000 7f0dea176700 1 Processor -- start 2024-03-28T05:22:31.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.614+0000 7f0dea176700 1 -- start start 2024-03-28T05:22:31.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.614+0000 7f0dea176700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0de4100b90 0x7f0de4101d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:31.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.614+0000 7f0dea176700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0de4101550 0x7f0de41022b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:31.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.614+0000 7f0dea176700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0de4107260 0x7f0de41027f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:31.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.614+0000 7f0dea176700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de4112590 con 0x7f0de4107260 2024-03-28T05:22:31.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.614+0000 7f0dea176700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de4112410 con 0x7f0de4101550 2024-03-28T05:22:31.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.614+0000 7f0dea176700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de4112710 con 0x7f0de4100b90 2024-03-28T05:22:31.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.614+0000 7f0de3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0de4107260 0x7f0de41027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:31.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.614+0000 7f0de37fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0de4100b90 0x7f0de4101d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:31.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.614+0000 7f0de3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0de4107260 0x7f0de41027f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56532/0 (socket says 172.21.15.23:56532) 2024-03-28T05:22:31.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.614+0000 7f0de3fff700 1 -- 172.21.15.23:0/1115004609 learned_addr learned my addr 172.21.15.23:0/1115004609 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:31.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.614+0000 7f0de2ffd700 1 --2- 172.21.15.23:0/1115004609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0de4101550 0x7f0de41022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:31.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.614+0000 7f0de3fff700 1 -- 172.21.15.23:0/1115004609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0de4100b90 msgr2=0x7f0de4101d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:31.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.614+0000 7f0de3fff700 1 --2- 172.21.15.23:0/1115004609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0de4100b90 0x7f0de4101d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:31.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.614+0000 7f0de3fff700 1 -- 172.21.15.23:0/1115004609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0de4101550 msgr2=0x7f0de41022b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:31.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.614+0000 7f0de3fff700 1 --2- 172.21.15.23:0/1115004609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0de4101550 0x7f0de41022b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:31.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.614+0000 7f0de3fff700 1 -- 172.21.15.23:0/1115004609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de41a6e70 con 0x7f0de4107260 2024-03-28T05:22:31.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.614+0000 7f0de3fff700 1 --2- 172.21.15.23:0/1115004609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0de4107260 0x7f0de41027f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc00b970 tx=0x7f0ddc00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:31.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.615+0000 7f0de0ff9700 1 -- 172.21.15.23:0/1115004609 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc00d860 con 0x7f0de4107260 2024-03-28T05:22:31.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.615+0000 7f0dea176700 1 -- 172.21.15.23:0/1115004609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0de41a7160 con 0x7f0de4107260 2024-03-28T05:22:31.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.615+0000 7f0de0ff9700 1 -- 172.21.15.23:0/1115004609 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ddc00e430 con 0x7f0de4107260 2024-03-28T05:22:31.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.615+0000 7f0de0ff9700 1 -- 172.21.15.23:0/1115004609 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc017660 con 0x7f0de4107260 2024-03-28T05:22:31.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.615+0000 7f0dea176700 1 -- 172.21.15.23:0/1115004609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0de41a7650 con 0x7f0de4107260 2024-03-28T05:22:31.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.616+0000 7f0dea176700 1 -- 172.21.15.23:0/1115004609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0de4068e50 con 0x7f0de4107260 2024-03-28T05:22:31.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.617+0000 7f0de0ff9700 1 -- 172.21.15.23:0/1115004609 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f0ddc00e5e0 con 0x7f0de4107260 2024-03-28T05:22:31.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.618+0000 7f0de0ff9700 1 --2- 172.21.15.23:0/1115004609 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f0dcc0a2920 0x7f0dcc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:31.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.618+0000 7f0de37fe700 1 --2- 172.21.15.23:0/1115004609 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f0dcc0a2920 0x7f0dcc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:31.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.618+0000 7f0de0ff9700 1 -- 172.21.15.23:0/1115004609 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f0ddc0ceba0 con 0x7f0de4107260 2024-03-28T05:22:31.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.618+0000 7f0de37fe700 1 --2- 172.21.15.23:0/1115004609 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f0dcc0a2920 0x7f0dcc0a4de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4000c00 tx=0x7f0dd4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:31.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.623+0000 7f0de0ff9700 1 -- 172.21.15.23:0/1115004609 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0ddc013030 con 0x7f0de4107260 2024-03-28T05:22:31.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:31 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/714730707' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T05:22:31.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:31 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1357788792' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T05:22:31.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:31 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/130343495' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T05:22:31.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.841+0000 7f0dea176700 1 -- 172.21.15.23:0/1115004609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f0de41a7910 con 0x7f0de4107260 2024-03-28T05:22:31.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.841+0000 7f0de0ff9700 1 -- 172.21.15.23:0/1115004609 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f0ddc09b720 con 0x7f0de4107260 2024-03-28T05:22:31.840 INFO:teuthology.orchestra.run.smithi023.stdout:141733920833 2024-03-28T05:22:31.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.843+0000 7f0dea176700 1 -- 172.21.15.23:0/1115004609 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f0dcc0a2920 msgr2=0x7f0dcc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:31.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.843+0000 7f0dea176700 1 --2- 172.21.15.23:0/1115004609 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f0dcc0a2920 0x7f0dcc0a4de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4000c00 tx=0x7f0dd4005fb0 comp rx=0 tx=0).stop 2024-03-28T05:22:31.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.843+0000 7f0dea176700 1 -- 172.21.15.23:0/1115004609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0de4107260 msgr2=0x7f0de41027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:31.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.843+0000 7f0dea176700 1 --2- 172.21.15.23:0/1115004609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0de4107260 0x7f0de41027f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc00b970 tx=0x7f0ddc00bc80 comp rx=0 tx=0).stop 2024-03-28T05:22:31.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.843+0000 7f0dea176700 1 -- 172.21.15.23:0/1115004609 shutdown_connections 2024-03-28T05:22:31.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.843+0000 7f0dea176700 1 --2- 172.21.15.23:0/1115004609 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f0dcc0a2920 0x7f0dcc0a4de0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:31.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.843+0000 7f0dea176700 1 --2- 172.21.15.23:0/1115004609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0de4107260 0x7f0de41027f0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:31.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.843+0000 7f0dea176700 1 --2- 172.21.15.23:0/1115004609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0de4100b90 0x7f0de4101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:31.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.843+0000 7f0dea176700 1 --2- 172.21.15.23:0/1115004609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0de4101550 0x7f0de41022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:31.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.843+0000 7f0dea176700 1 -- 172.21.15.23:0/1115004609 >> 172.21.15.23:0/1115004609 conn(0x7f0de40763e0 msgr2=0x7f0de40feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:31.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.843+0000 7f0dea176700 1 -- 172.21.15.23:0/1115004609 shutdown_connections 2024-03-28T05:22:31.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.844+0000 7f0dea176700 1 -- 172.21.15.23:0/1115004609 wait complete. 2024-03-28T05:22:31.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:31 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/714730707' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T05:22:31.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:31 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1357788792' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T05:22:31.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:31 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/130343495' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T05:22:32.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:31 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/714730707' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T05:22:32.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:31 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1357788792' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T05:22:32.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:31 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/130343495' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T05:22:32.470 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920829 got 141733920833 for osd.4 2024-03-28T05:22:32.471 DEBUG:teuthology.parallel:result is None 2024-03-28T05:22:32.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:32 smithi113 ceph-mon[37164]: pgmap v332: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:22:32.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:32 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1115004609' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T05:22:32.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.271+0000 7fbedf1d4700 1 -- 172.21.15.23:0/685020081 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbed81074a0 msgr2=0x7fbed810f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:32.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.271+0000 7fbedf1d4700 1 --2- 172.21.15.23:0/685020081 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbed81074a0 0x7fbed810f070 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fbed000b0a0 tx=0x7fbed000b3b0 comp rx=0 tx=0).stop 2024-03-28T05:22:32.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.271+0000 7fbedf1d4700 1 -- 172.21.15.23:0/685020081 shutdown_connections 2024-03-28T05:22:32.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.271+0000 7fbedf1d4700 1 --2- 172.21.15.23:0/685020081 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbed81074a0 0x7fbed810f070 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:32.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.271+0000 7fbedf1d4700 1 --2- 172.21.15.23:0/685020081 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed8102470 0x7fbed8102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:32.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.271+0000 7fbedf1d4700 1 --2- 172.21.15.23:0/685020081 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbed8102e20 0x7fbed8106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:32.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.271+0000 7fbedf1d4700 1 -- 172.21.15.23:0/685020081 >> 172.21.15.23:0/685020081 conn(0x7fbed8079220 msgr2=0x7fbed8079630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:32.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.271+0000 7fbedf1d4700 1 -- 172.21.15.23:0/685020081 shutdown_connections 2024-03-28T05:22:32.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.271+0000 7fbedf1d4700 1 -- 172.21.15.23:0/685020081 wait complete. 2024-03-28T05:22:32.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.272+0000 7fbedf1d4700 1 Processor -- start 2024-03-28T05:22:32.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.272+0000 7fbedf1d4700 1 -- start start 2024-03-28T05:22:32.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.272+0000 7fbedf1d4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbed8102470 0x7fbed8074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:32.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.272+0000 7fbedf1d4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbed8102e20 0x7fbed80749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:32.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.272+0000 7fbedf1d4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed81074a0 0x7fbed8074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:32.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.272+0000 7fbedf1d4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed8112500 con 0x7fbed8102470 2024-03-28T05:22:32.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.272+0000 7fbedf1d4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed8112380 con 0x7fbed81074a0 2024-03-28T05:22:32.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.272+0000 7fbedf1d4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed8112680 con 0x7fbed8102e20 2024-03-28T05:22:32.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.272+0000 7fbedcf70700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbed8102470 0x7fbed8074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:32.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.272+0000 7fbedcf70700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbed8102470 0x7fbed8074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56510/0 (socket says 172.21.15.23:56510) 2024-03-28T05:22:32.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.272+0000 7fbedcf70700 1 -- 172.21.15.23:0/130343495 learned_addr learned my addr 172.21.15.23:0/130343495 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:32.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.272+0000 7fbed7fff700 1 --2- 172.21.15.23:0/130343495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbed8102e20 0x7fbed80749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:32.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.273+0000 7fbedcf70700 1 -- 172.21.15.23:0/130343495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbed8102e20 msgr2=0x7fbed80749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:32.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.273+0000 7fbedcf70700 1 --2- 172.21.15.23:0/130343495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbed8102e20 0x7fbed80749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:32.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.273+0000 7fbedcf70700 1 -- 172.21.15.23:0/130343495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed81074a0 msgr2=0x7fbed8074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:32.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.273+0000 7fbedcf70700 1 --2- 172.21.15.23:0/130343495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed81074a0 0x7fbed8074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:32.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.273+0000 7fbedcf70700 1 -- 172.21.15.23:0/130343495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed81a6eb0 con 0x7fbed8102470 2024-03-28T05:22:32.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.273+0000 7fbedcf70700 1 --2- 172.21.15.23:0/130343495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbed8102470 0x7fbed8074490 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbecc00b530 tx=0x7fbecc00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:32.999 INFO:teuthology.orchestra.run.smithi023.stdout:253403070508 2024-03-28T05:22:33.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.273+0000 7fbed5ffb700 1 -- 172.21.15.23:0/130343495 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc015020 con 0x7fbed8102470 2024-03-28T05:22:33.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.273+0000 7fbed5ffb700 1 -- 172.21.15.23:0/130343495 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbecc011d10 con 0x7fbed8102470 2024-03-28T05:22:33.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.273+0000 7fbedf1d4700 1 -- 172.21.15.23:0/130343495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbed81a71a0 con 0x7fbed8102470 2024-03-28T05:22:33.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.273+0000 7fbed5ffb700 1 -- 172.21.15.23:0/130343495 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc00f2e0 con 0x7fbed8102470 2024-03-28T05:22:33.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.273+0000 7fbedf1d4700 1 -- 172.21.15.23:0/130343495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbed81a76e0 con 0x7fbed8102470 2024-03-28T05:22:33.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.274+0000 7fbedf1d4700 1 -- 172.21.15.23:0/130343495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbed8068e50 con 0x7fbed8102470 2024-03-28T05:22:33.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.275+0000 7fbed5ffb700 1 -- 172.21.15.23:0/130343495 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fbecc018020 con 0x7fbed8102470 2024-03-28T05:22:33.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.276+0000 7fbed5ffb700 1 --2- 172.21.15.23:0/130343495 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fbec80a2920 0x7fbec80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:33.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.276+0000 7fbed7fff700 1 --2- 172.21.15.23:0/130343495 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fbec80a2920 0x7fbec80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:33.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.276+0000 7fbed5ffb700 1 -- 172.21.15.23:0/130343495 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fbecc0ce680 con 0x7fbed8102470 2024-03-28T05:22:33.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.276+0000 7fbed7fff700 1 --2- 172.21.15.23:0/130343495 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fbec80a2920 0x7fbec80a4de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbed81088b0 tx=0x7fbec4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:33.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.280+0000 7fbed5ffb700 1 -- 172.21.15.23:0/130343495 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbecc01f020 con 0x7fbed8102470 2024-03-28T05:22:33.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.486+0000 7fbedf1d4700 1 -- 172.21.15.23:0/130343495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7fbed802da30 con 0x7fbed8102470 2024-03-28T05:22:33.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.486+0000 7fbed5ffb700 1 -- 172.21.15.23:0/130343495 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fbecc09b370 con 0x7fbed8102470 2024-03-28T05:22:33.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.489+0000 7fbedf1d4700 1 -- 172.21.15.23:0/130343495 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fbec80a2920 msgr2=0x7fbec80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:33.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.489+0000 7fbedf1d4700 1 --2- 172.21.15.23:0/130343495 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fbec80a2920 0x7fbec80a4de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbed81088b0 tx=0x7fbec4009260 comp rx=0 tx=0).stop 2024-03-28T05:22:33.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.489+0000 7fbedf1d4700 1 -- 172.21.15.23:0/130343495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbed8102470 msgr2=0x7fbed8074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:33.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.489+0000 7fbedf1d4700 1 --2- 172.21.15.23:0/130343495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbed8102470 0x7fbed8074490 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbecc00b530 tx=0x7fbecc00b840 comp rx=0 tx=0).stop 2024-03-28T05:22:33.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.489+0000 7fbedf1d4700 1 -- 172.21.15.23:0/130343495 shutdown_connections 2024-03-28T05:22:33.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.489+0000 7fbedf1d4700 1 --2- 172.21.15.23:0/130343495 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fbec80a2920 0x7fbec80a4de0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:33.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.489+0000 7fbedf1d4700 1 --2- 172.21.15.23:0/130343495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbed81074a0 0x7fbed8074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:33.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.489+0000 7fbedf1d4700 1 --2- 172.21.15.23:0/130343495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbed8102470 0x7fbed8074490 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:33.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.489+0000 7fbedf1d4700 1 --2- 172.21.15.23:0/130343495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbed8102e20 0x7fbed80749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:33.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.489+0000 7fbedf1d4700 1 -- 172.21.15.23:0/130343495 >> 172.21.15.23:0/130343495 conn(0x7fbed8079220 msgr2=0x7fbed8100260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:33.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.489+0000 7fbedf1d4700 1 -- 172.21.15.23:0/130343495 shutdown_connections 2024-03-28T05:22:33.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:31.489+0000 7fbedf1d4700 1 -- 172.21.15.23:0/130343495 wait complete. 2024-03-28T05:22:33.002 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:32 smithi023 ceph-mon[31954]: pgmap v332: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:22:33.002 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:32 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1115004609' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T05:22:33.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:32 smithi098 ceph-mon[36087]: pgmap v332: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:22:33.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:32 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1115004609' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T05:22:33.436 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:22:33.679 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070503 got 253403070508 for osd.8 2024-03-28T05:22:33.680 DEBUG:teuthology.parallel:result is None 2024-03-28T05:22:34.107 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332141 got 219043332145 for osd.7 2024-03-28T05:22:34.107 DEBUG:teuthology.parallel:result is None 2024-03-28T05:22:34.853 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645342 got 309237645345 for osd.10 2024-03-28T05:22:34.854 DEBUG:teuthology.parallel:result is None 2024-03-28T05:22:34.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.317+0000 7f0c8cb07700 1 -- 172.21.15.23:0/2877799817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c88069e40 msgr2=0x7f0c8806a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:34.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.317+0000 7f0c8cb07700 1 --2- 172.21.15.23:0/2877799817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c88069e40 0x7f0c8806a2a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0c780099c0 tx=0x7f0c78009cd0 comp rx=0 tx=0).stop 2024-03-28T05:22:34.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.317+0000 7f0c8cb07700 1 -- 172.21.15.23:0/2877799817 shutdown_connections 2024-03-28T05:22:34.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.317+0000 7f0c8cb07700 1 --2- 172.21.15.23:0/2877799817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c8806a7e0 0x7f0c8810f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:34.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.317+0000 7f0c8cb07700 1 --2- 172.21.15.23:0/2877799817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c881081e0 0x7f0c881085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:34.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.317+0000 7f0c8cb07700 1 --2- 172.21.15.23:0/2877799817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c88069e40 0x7f0c8806a2a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:34.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.317+0000 7f0c8cb07700 1 -- 172.21.15.23:0/2877799817 >> 172.21.15.23:0/2877799817 conn(0x7f0c88077fc0 msgr2=0x7f0c880783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:34.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.317+0000 7f0c8cb07700 1 -- 172.21.15.23:0/2877799817 shutdown_connections 2024-03-28T05:22:34.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.317+0000 7f0c8cb07700 1 -- 172.21.15.23:0/2877799817 wait complete. 2024-03-28T05:22:34.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.318+0000 7f0c8cb07700 1 Processor -- start 2024-03-28T05:22:34.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.318+0000 7f0c8cb07700 1 -- start start 2024-03-28T05:22:34.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.318+0000 7f0c8cb07700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c88069e40 0x7f0c881a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:34.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.318+0000 7f0c8cb07700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c8806a7e0 0x7f0c881a0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:34.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.318+0000 7f0c8cb07700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c881081e0 0x7f0c8819a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:34.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.318+0000 7f0c8cb07700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c88112440 con 0x7f0c881081e0 2024-03-28T05:22:34.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.318+0000 7f0c8cb07700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c881122c0 con 0x7f0c88069e40 2024-03-28T05:22:34.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.318+0000 7f0c8cb07700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c881125c0 con 0x7f0c8806a7e0 2024-03-28T05:22:34.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.318+0000 7f0c86d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c881081e0 0x7f0c8819a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:34.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.318+0000 7f0c85d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c8806a7e0 0x7f0c881a0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:34.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.318+0000 7f0c86d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c881081e0 0x7f0c8819a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36054/0 (socket says 172.21.15.23:36054) 2024-03-28T05:22:34.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.318+0000 7f0c85d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c8806a7e0 0x7f0c881a0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:36814/0 (socket says 172.21.15.23:36814) 2024-03-28T05:22:34.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.318+0000 7f0c86d9d700 1 -- 172.21.15.23:0/2571111720 learned_addr learned my addr 172.21.15.23:0/2571111720 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:34.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.318+0000 7f0c8659c700 1 --2- 172.21.15.23:0/2571111720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c88069e40 0x7f0c881a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:34.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.319+0000 7f0c86d9d700 1 -- 172.21.15.23:0/2571111720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c8806a7e0 msgr2=0x7f0c881a0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:34.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.319+0000 7f0c86d9d700 1 --2- 172.21.15.23:0/2571111720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c8806a7e0 0x7f0c881a0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:34.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.319+0000 7f0c86d9d700 1 -- 172.21.15.23:0/2571111720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c88069e40 msgr2=0x7f0c881a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:34.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.319+0000 7f0c86d9d700 1 --2- 172.21.15.23:0/2571111720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c88069e40 0x7f0c881a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:34.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.319+0000 7f0c86d9d700 1 -- 172.21.15.23:0/2571111720 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c8819b090 con 0x7f0c881081e0 2024-03-28T05:22:34.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.319+0000 7f0c85d9b700 1 --2- 172.21.15.23:0/2571111720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c8806a7e0 0x7f0c881a0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:22:34.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.319+0000 7f0c8659c700 1 --2- 172.21.15.23:0/2571111720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c88069e40 0x7f0c881a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:22:34.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.319+0000 7f0c86d9d700 1 --2- 172.21.15.23:0/2571111720 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c881081e0 0x7f0c8819a930 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0c8000ef30 tx=0x7f0c8000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:34.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.319+0000 7f0c777fe700 1 -- 172.21.15.23:0/2571111720 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c800092e0 con 0x7f0c881081e0 2024-03-28T05:22:34.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.319+0000 7f0c8cb07700 1 -- 172.21.15.23:0/2571111720 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c8819b320 con 0x7f0c881081e0 2024-03-28T05:22:34.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.319+0000 7f0c777fe700 1 -- 172.21.15.23:0/2571111720 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c80004bb0 con 0x7f0c881081e0 2024-03-28T05:22:34.907 INFO:teuthology.orchestra.run.smithi023.stdout:34359738453 2024-03-28T05:22:34.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.319+0000 7f0c777fe700 1 -- 172.21.15.23:0/2571111720 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c80004280 con 0x7f0c881081e0 2024-03-28T05:22:34.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.319+0000 7f0c8cb07700 1 -- 172.21.15.23:0/2571111720 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c881a7640 con 0x7f0c881081e0 2024-03-28T05:22:34.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.320+0000 7f0c8cb07700 1 -- 172.21.15.23:0/2571111720 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c88068e50 con 0x7f0c881081e0 2024-03-28T05:22:34.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.321+0000 7f0c777fe700 1 -- 172.21.15.23:0/2571111720 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f0c8001b030 con 0x7f0c881081e0 2024-03-28T05:22:34.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.322+0000 7f0c777fe700 1 --2- 172.21.15.23:0/2571111720 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f0c700a2920 0x7f0c700a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:34.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.322+0000 7f0c8659c700 1 --2- 172.21.15.23:0/2571111720 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f0c700a2920 0x7f0c700a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:34.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.322+0000 7f0c777fe700 1 -- 172.21.15.23:0/2571111720 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f0c800cf320 con 0x7f0c881081e0 2024-03-28T05:22:34.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.322+0000 7f0c8659c700 1 --2- 172.21.15.23:0/2571111720 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f0c700a2920 0x7f0c700a4de0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0c8819ba70 tx=0x7f0c7c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:34.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.326+0000 7f0c777fe700 1 -- 172.21.15.23:0/2571111720 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0c80022020 con 0x7f0c881081e0 2024-03-28T05:22:34.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.533+0000 7f0c8cb07700 1 -- 172.21.15.23:0/2571111720 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f0c881a7810 con 0x7f0c881081e0 2024-03-28T05:22:34.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.533+0000 7f0c777fe700 1 -- 172.21.15.23:0/2571111720 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0c80010040 con 0x7f0c881081e0 2024-03-28T05:22:34.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.535+0000 7f0c8cb07700 1 -- 172.21.15.23:0/2571111720 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f0c700a2920 msgr2=0x7f0c700a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:34.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.535+0000 7f0c8cb07700 1 --2- 172.21.15.23:0/2571111720 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f0c700a2920 0x7f0c700a4de0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0c8819ba70 tx=0x7f0c7c009260 comp rx=0 tx=0).stop 2024-03-28T05:22:34.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.535+0000 7f0c8cb07700 1 -- 172.21.15.23:0/2571111720 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c881081e0 msgr2=0x7f0c8819a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:34.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.535+0000 7f0c8cb07700 1 --2- 172.21.15.23:0/2571111720 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c881081e0 0x7f0c8819a930 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0c8000ef30 tx=0x7f0c8000c3a0 comp rx=0 tx=0).stop 2024-03-28T05:22:34.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.535+0000 7f0c8cb07700 1 -- 172.21.15.23:0/2571111720 shutdown_connections 2024-03-28T05:22:34.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.535+0000 7f0c8cb07700 1 --2- 172.21.15.23:0/2571111720 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f0c700a2920 0x7f0c700a4de0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:34.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.535+0000 7f0c8cb07700 1 --2- 172.21.15.23:0/2571111720 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c881081e0 0x7f0c8819a930 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:34.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.535+0000 7f0c8cb07700 1 --2- 172.21.15.23:0/2571111720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c88069e40 0x7f0c881a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:34.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.535+0000 7f0c8cb07700 1 --2- 172.21.15.23:0/2571111720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0c8806a7e0 0x7f0c881a0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:34.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.535+0000 7f0c8cb07700 1 -- 172.21.15.23:0/2571111720 >> 172.21.15.23:0/2571111720 conn(0x7f0c88077fc0 msgr2=0x7f0c8810d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:34.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.535+0000 7f0c8cb07700 1 -- 172.21.15.23:0/2571111720 shutdown_connections 2024-03-28T05:22:34.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:34.535+0000 7f0c8cb07700 1 -- 172.21.15.23:0/2571111720 wait complete. 2024-03-28T05:22:34.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:34 smithi113 ceph-mon[37164]: pgmap v333: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:34.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:34 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2571111720' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T05:22:35.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:34 smithi023 ceph-mon[31954]: pgmap v333: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:35.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:34 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2571111720' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T05:22:35.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:34 smithi098 ceph-mon[36087]: pgmap v333: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:35.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:34 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2571111720' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T05:22:35.569 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:22:35.570 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:22:35.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.767+0000 7fb914d9c700 1 -- 172.21.15.23:0/1195941064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9101081c0 msgr2=0x7fb9101085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:35.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.767+0000 7fb914d9c700 1 --2- 172.21.15.23:0/1195941064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9101081c0 0x7fb9101085a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb9000099c0 tx=0x7fb900009cd0 comp rx=0 tx=0).stop 2024-03-28T05:22:35.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.767+0000 7fb914d9c700 1 -- 172.21.15.23:0/1195941064 shutdown_connections 2024-03-28T05:22:35.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.767+0000 7fb914d9c700 1 --2- 172.21.15.23:0/1195941064 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb910102ba0 0x7fb91010efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:35.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.767+0000 7fb914d9c700 1 --2- 172.21.15.23:0/1195941064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9101081c0 0x7fb9101085a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:35.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.767+0000 7fb914d9c700 1 --2- 172.21.15.23:0/1195941064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9101021e0 0x7fb910102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:35.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.767+0000 7fb914d9c700 1 -- 172.21.15.23:0/1195941064 >> 172.21.15.23:0/1195941064 conn(0x7fb9100792c0 msgr2=0x7fb9100796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:35.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.767+0000 7fb914d9c700 1 -- 172.21.15.23:0/1195941064 shutdown_connections 2024-03-28T05:22:35.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.767+0000 7fb914d9c700 1 -- 172.21.15.23:0/1195941064 wait complete. 2024-03-28T05:22:35.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.768+0000 7fb914d9c700 1 Processor -- start 2024-03-28T05:22:35.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.768+0000 7fb914d9c700 1 -- start start 2024-03-28T05:22:35.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.768+0000 7fb914d9c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9101021e0 0x7fb91019e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:35.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.768+0000 7fb914d9c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb910102ba0 0x7fb91019eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:35.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.768+0000 7fb914d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9101081c0 0x7fb9101986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:35.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.768+0000 7fb914d9c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb910112470 con 0x7fb9101081c0 2024-03-28T05:22:35.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.768+0000 7fb914d9c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9101122f0 con 0x7fb910102ba0 2024-03-28T05:22:35.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.768+0000 7fb914d9c700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9101125f0 con 0x7fb9101021e0 2024-03-28T05:22:35.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.768+0000 7fb90ed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9101081c0 0x7fb9101986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:35.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.768+0000 7fb90ed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9101081c0 0x7fb9101986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36064/0 (socket says 172.21.15.23:36064) 2024-03-28T05:22:35.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.768+0000 7fb90ed9d700 1 -- 172.21.15.23:0/1242205135 learned_addr learned my addr 172.21.15.23:0/1242205135 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:35.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.768+0000 7fb90e59c700 1 --2- 172.21.15.23:0/1242205135 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9101021e0 0x7fb91019e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:35.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.768+0000 7fb90dd9b700 1 --2- 172.21.15.23:0/1242205135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb910102ba0 0x7fb91019eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:35.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.768+0000 7fb90ed9d700 1 -- 172.21.15.23:0/1242205135 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9101021e0 msgr2=0x7fb91019e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:35.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.768+0000 7fb90ed9d700 1 --2- 172.21.15.23:0/1242205135 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9101021e0 0x7fb91019e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:35.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.768+0000 7fb90ed9d700 1 -- 172.21.15.23:0/1242205135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb910102ba0 msgr2=0x7fb91019eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:35.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.769+0000 7fb90ed9d700 1 --2- 172.21.15.23:0/1242205135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb910102ba0 0x7fb91019eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:35.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.769+0000 7fb90ed9d700 1 -- 172.21.15.23:0/1242205135 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb910198fb0 con 0x7fb9101081c0 2024-03-28T05:22:35.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.769+0000 7fb90e59c700 1 --2- 172.21.15.23:0/1242205135 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9101021e0 0x7fb91019e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:22:35.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.769+0000 7fb90ed9d700 1 --2- 172.21.15.23:0/1242205135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9101081c0 0x7fb9101986d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb90800ef30 tx=0x7fb90800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:35.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.769+0000 7fb8ff7fe700 1 -- 172.21.15.23:0/1242205135 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9080092e0 con 0x7fb9101081c0 2024-03-28T05:22:35.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.769+0000 7fb8ff7fe700 1 -- 172.21.15.23:0/1242205135 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb908004bb0 con 0x7fb9101081c0 2024-03-28T05:22:35.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.769+0000 7fb914d9c700 1 -- 172.21.15.23:0/1242205135 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb910199240 con 0x7fb9101081c0 2024-03-28T05:22:35.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.769+0000 7fb914d9c700 1 -- 172.21.15.23:0/1242205135 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9101a53b0 con 0x7fb9101081c0 2024-03-28T05:22:35.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.769+0000 7fb8ff7fe700 1 -- 172.21.15.23:0/1242205135 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb908004280 con 0x7fb9101081c0 2024-03-28T05:22:35.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.770+0000 7fb914d9c700 1 -- 172.21.15.23:0/1242205135 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb910103870 con 0x7fb9101081c0 2024-03-28T05:22:35.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.771+0000 7fb8ff7fe700 1 -- 172.21.15.23:0/1242205135 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fb90801b030 con 0x7fb9101081c0 2024-03-28T05:22:35.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.772+0000 7fb8ff7fe700 1 --2- 172.21.15.23:0/1242205135 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb8f80a2920 0x7fb8f80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:35.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.772+0000 7fb90e59c700 1 --2- 172.21.15.23:0/1242205135 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb8f80a2920 0x7fb8f80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:35.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.772+0000 7fb8ff7fe700 1 -- 172.21.15.23:0/1242205135 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fb9080ce9a0 con 0x7fb9101081c0 2024-03-28T05:22:35.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.772+0000 7fb90e59c700 1 --2- 172.21.15.23:0/1242205135 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb8f80a2920 0x7fb8f80a4de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb900000c00 tx=0x7fb900005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:35.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.776+0000 7fb8ff7fe700 1 -- 172.21.15.23:0/1242205135 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb908022020 con 0x7fb9101081c0 2024-03-28T05:22:35.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.983+0000 7fb914d9c700 1 -- 172.21.15.23:0/1242205135 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7fb91002da30 con 0x7fb9101081c0 2024-03-28T05:22:35.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.983+0000 7fb8ff7fe700 1 -- 172.21.15.23:0/1242205135 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb908010040 con 0x7fb9101081c0 2024-03-28T05:22:35.982 INFO:teuthology.orchestra.run.smithi023.stdout:287762808873 2024-03-28T05:22:35.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.986+0000 7fb914d9c700 1 -- 172.21.15.23:0/1242205135 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb8f80a2920 msgr2=0x7fb8f80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:35.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.986+0000 7fb914d9c700 1 --2- 172.21.15.23:0/1242205135 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb8f80a2920 0x7fb8f80a4de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb900000c00 tx=0x7fb900005fb0 comp rx=0 tx=0).stop 2024-03-28T05:22:35.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.986+0000 7fb914d9c700 1 -- 172.21.15.23:0/1242205135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9101081c0 msgr2=0x7fb9101986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:35.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.986+0000 7fb914d9c700 1 --2- 172.21.15.23:0/1242205135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9101081c0 0x7fb9101986d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb90800ef30 tx=0x7fb90800c3a0 comp rx=0 tx=0).stop 2024-03-28T05:22:35.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.986+0000 7fb914d9c700 1 -- 172.21.15.23:0/1242205135 shutdown_connections 2024-03-28T05:22:35.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.986+0000 7fb914d9c700 1 --2- 172.21.15.23:0/1242205135 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb8f80a2920 0x7fb8f80a4de0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:35.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.986+0000 7fb914d9c700 1 --2- 172.21.15.23:0/1242205135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9101081c0 0x7fb9101986d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:35.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.986+0000 7fb914d9c700 1 --2- 172.21.15.23:0/1242205135 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9101021e0 0x7fb91019e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:35.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.986+0000 7fb914d9c700 1 --2- 172.21.15.23:0/1242205135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb910102ba0 0x7fb91019eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:35.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.986+0000 7fb914d9c700 1 -- 172.21.15.23:0/1242205135 >> 172.21.15.23:0/1242205135 conn(0x7fb9100792c0 msgr2=0x7fb910100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:35.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.986+0000 7fb914d9c700 1 -- 172.21.15.23:0/1242205135 shutdown_connections 2024-03-28T05:22:35.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:35.986+0000 7fb914d9c700 1 -- 172.21.15.23:0/1242205135 wait complete. 2024-03-28T05:22:36.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:36 smithi113 ceph-mon[37164]: pgmap v334: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:22:36.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:36 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1242205135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T05:22:36.997 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:36 smithi023 ceph-mon[31954]: pgmap v334: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:22:36.997 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:36 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1242205135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T05:22:37.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:36 smithi098 ceph-mon[36087]: pgmap v334: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:22:37.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:36 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1242205135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T05:22:37.690 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808867 got 287762808873 for osd.9 2024-03-28T05:22:37.691 DEBUG:teuthology.parallel:result is None 2024-03-28T05:22:38.133 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738447 got 34359738453 for osd.0 2024-03-28T05:22:38.133 DEBUG:teuthology.parallel:result is None 2024-03-28T05:22:38.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:38 smithi023 ceph-mon[31954]: pgmap v335: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:38.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:38 smithi113 ceph-mon[37164]: pgmap v335: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:39.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:38 smithi098 ceph-mon[36087]: pgmap v335: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:39.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.887+0000 7f43b0a55700 1 -- 172.21.15.23:0/406830996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43ac1071c0 msgr2=0x7f43ac1075a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:39.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.887+0000 7f43b0a55700 1 --2- 172.21.15.23:0/406830996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43ac1071c0 0x7f43ac1075a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f439c0099c0 tx=0x7f439c009cd0 comp rx=0 tx=0).stop 2024-03-28T05:22:39.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.887+0000 7f43b0a55700 1 -- 172.21.15.23:0/406830996 shutdown_connections 2024-03-28T05:22:39.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.887+0000 7f43b0a55700 1 --2- 172.21.15.23:0/406830996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43ac101d30 0x7f43ac10df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:39.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.887+0000 7f43b0a55700 1 --2- 172.21.15.23:0/406830996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43ac1071c0 0x7f43ac1075a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:39.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.887+0000 7f43b0a55700 1 --2- 172.21.15.23:0/406830996 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43ac101370 0x7f43ac1017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:39.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.887+0000 7f43b0a55700 1 -- 172.21.15.23:0/406830996 >> 172.21.15.23:0/406830996 conn(0x7f43ac077f50 msgr2=0x7f43ac078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:39.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.887+0000 7f43b0a55700 1 -- 172.21.15.23:0/406830996 shutdown_connections 2024-03-28T05:22:39.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.887+0000 7f43b0a55700 1 -- 172.21.15.23:0/406830996 wait complete. 2024-03-28T05:22:39.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.888+0000 7f43b0a55700 1 Processor -- start 2024-03-28T05:22:39.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.888+0000 7f43b0a55700 1 -- start start 2024-03-28T05:22:39.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.888+0000 7f43b0a55700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43ac101370 0x7f43ac197550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:39.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.888+0000 7f43b0a55700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43ac101d30 0x7f43ac197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:39.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.888+0000 7f43b0a55700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43ac1071c0 0x7f43ac197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:39.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.888+0000 7f43b0a55700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac1113c0 con 0x7f43ac101370 2024-03-28T05:22:39.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.888+0000 7f43b0a55700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac111240 con 0x7f43ac1071c0 2024-03-28T05:22:39.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.888+0000 7f43b0a55700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac111540 con 0x7f43ac101d30 2024-03-28T05:22:39.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.888+0000 7f43aa59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43ac101370 0x7f43ac197550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:39.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.888+0000 7f43a9d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43ac101d30 0x7f43ac197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:39.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.888+0000 7f43a9d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43ac101d30 0x7f43ac197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:36846/0 (socket says 172.21.15.23:36846) 2024-03-28T05:22:39.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.888+0000 7f43aa59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43ac101370 0x7f43ac197550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36096/0 (socket says 172.21.15.23:36096) 2024-03-28T05:22:39.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.888+0000 7f43a9d9b700 1 -- 172.21.15.23:0/2123081989 learned_addr learned my addr 172.21.15.23:0/2123081989 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:39.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.888+0000 7f43aad9d700 1 --2- 172.21.15.23:0/2123081989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43ac1071c0 0x7f43ac197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:39.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.888+0000 7f43a9d9b700 1 -- 172.21.15.23:0/2123081989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43ac1071c0 msgr2=0x7f43ac197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:39.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.888+0000 7f43a9d9b700 1 --2- 172.21.15.23:0/2123081989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43ac1071c0 0x7f43ac197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:39.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.888+0000 7f43a9d9b700 1 -- 172.21.15.23:0/2123081989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43ac101370 msgr2=0x7f43ac197550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:39.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.888+0000 7f43a9d9b700 1 --2- 172.21.15.23:0/2123081989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43ac101370 0x7f43ac197550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:39.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.888+0000 7f43a9d9b700 1 -- 172.21.15.23:0/2123081989 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ac1a6dc0 con 0x7f43ac101d30 2024-03-28T05:22:39.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.889+0000 7f43a9d9b700 1 --2- 172.21.15.23:0/2123081989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43ac101d30 0x7f43ac197a90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f43a000e9a0 tx=0x7f43a000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:39.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.889+0000 7f439b7fe700 1 -- 172.21.15.23:0/2123081989 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a000cbd0 con 0x7f43ac101d30 2024-03-28T05:22:39.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.889+0000 7f43b0a55700 1 -- 172.21.15.23:0/2123081989 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43ac1a70b0 con 0x7f43ac101d30 2024-03-28T05:22:39.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.889+0000 7f439b7fe700 1 -- 172.21.15.23:0/2123081989 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43a000cd70 con 0x7f43ac101d30 2024-03-28T05:22:39.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.889+0000 7f439b7fe700 1 -- 172.21.15.23:0/2123081989 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a0018750 con 0x7f43ac101d30 2024-03-28T05:22:39.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.889+0000 7f43b0a55700 1 -- 172.21.15.23:0/2123081989 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43ac1a7640 con 0x7f43ac101d30 2024-03-28T05:22:39.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.890+0000 7f43b0a55700 1 -- 172.21.15.23:0/2123081989 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43ac068e50 con 0x7f43ac101d30 2024-03-28T05:22:39.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.891+0000 7f439b7fe700 1 -- 172.21.15.23:0/2123081989 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f43a0010430 con 0x7f43ac101d30 2024-03-28T05:22:39.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.891+0000 7f439b7fe700 1 --2- 172.21.15.23:0/2123081989 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f43940a2910 0x7f43940a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:39.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.892+0000 7f43aa59c700 1 --2- 172.21.15.23:0/2123081989 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f43940a2910 0x7f43940a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:39.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.892+0000 7f439b7fe700 1 -- 172.21.15.23:0/2123081989 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f43a00cf050 con 0x7f43ac101d30 2024-03-28T05:22:39.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.892+0000 7f43aa59c700 1 --2- 172.21.15.23:0/2123081989 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f43940a2910 0x7f43940a4dd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f439c000c00 tx=0x7f439c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:39.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:39.900+0000 7f439b7fe700 1 -- 172.21.15.23:0/2123081989 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f43a009bd90 con 0x7f43ac101d30 2024-03-28T05:22:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.094+0000 7fd1620f6700 1 -- 172.21.15.23:0/556666397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd15c101090 msgr2=0x7fd15c1014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.094+0000 7fd1620f6700 1 --2- 172.21.15.23:0/556666397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd15c101090 0x7fd15c1014f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd150009990 tx=0x7fd150009ca0 comp rx=0 tx=0).stop 2024-03-28T05:22:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.094+0000 7fd1620f6700 1 -- 172.21.15.23:0/556666397 shutdown_connections 2024-03-28T05:22:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.094+0000 7fd1620f6700 1 --2- 172.21.15.23:0/556666397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd15c101a30 0x7fd15c10df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.094+0000 7fd1620f6700 1 --2- 172.21.15.23:0/556666397 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd15c1070d0 0x7fd15c1074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.094+0000 7fd1620f6700 1 --2- 172.21.15.23:0/556666397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd15c101090 0x7fd15c1014f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.094+0000 7fd1620f6700 1 -- 172.21.15.23:0/556666397 >> 172.21.15.23:0/556666397 conn(0x7fd15c077fc0 msgr2=0x7fd15c0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.094+0000 7fd1620f6700 1 -- 172.21.15.23:0/556666397 shutdown_connections 2024-03-28T05:22:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.094+0000 7fd1620f6700 1 -- 172.21.15.23:0/556666397 wait complete. 2024-03-28T05:22:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.094+0000 7fd1620f6700 1 Processor -- start 2024-03-28T05:22:40.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.095+0000 7fd1620f6700 1 -- start start 2024-03-28T05:22:40.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.095+0000 7fd1620f6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd15c101090 0x7fd15c19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:40.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.095+0000 7fd1620f6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd15c101a30 0x7fd15c19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:40.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.095+0000 7fd1620f6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd15c1070d0 0x7fd15c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:40.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.095+0000 7fd1620f6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c111330 con 0x7fd15c1070d0 2024-03-28T05:22:40.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.095+0000 7fd1620f6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c1111b0 con 0x7fd15c101a30 2024-03-28T05:22:40.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.095+0000 7fd1620f6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c1114b0 con 0x7fd15c101090 2024-03-28T05:22:40.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.095+0000 7fd15bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd15c1070d0 0x7fd15c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:40.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.095+0000 7fd15bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd15c1070d0 0x7fd15c198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36116/0 (socket says 172.21.15.23:36116) 2024-03-28T05:22:40.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.095+0000 7fd15bfff700 1 -- 172.21.15.23:0/1512429905 learned_addr learned my addr 172.21.15.23:0/1512429905 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:40.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.095+0000 7fd15b7fe700 1 --2- 172.21.15.23:0/1512429905 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd15c101090 0x7fd15c19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:40.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.095+0000 7fd15affd700 1 --2- 172.21.15.23:0/1512429905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd15c101a30 0x7fd15c19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:40.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.095+0000 7fd15bfff700 1 -- 172.21.15.23:0/1512429905 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd15c101090 msgr2=0x7fd15c19e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:40.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.095+0000 7fd15bfff700 1 --2- 172.21.15.23:0/1512429905 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd15c101090 0x7fd15c19e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:40.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.095+0000 7fd15bfff700 1 -- 172.21.15.23:0/1512429905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd15c101a30 msgr2=0x7fd15c19ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:40.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.095+0000 7fd15bfff700 1 --2- 172.21.15.23:0/1512429905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd15c101a30 0x7fd15c19ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:40.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.095+0000 7fd15bfff700 1 -- 172.21.15.23:0/1512429905 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c199020 con 0x7fd15c1070d0 2024-03-28T05:22:40.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.095+0000 7fd15affd700 1 --2- 172.21.15.23:0/1512429905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd15c101a30 0x7fd15c19ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:22:40.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.096+0000 7fd15bfff700 1 --2- 172.21.15.23:0/1512429905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd15c1070d0 0x7fd15c198740 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd15400ada0 tx=0x7fd15400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:40.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.096+0000 7fd158ff9700 1 -- 172.21.15.23:0/1512429905 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd154017020 con 0x7fd15c1070d0 2024-03-28T05:22:40.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.096+0000 7fd1620f6700 1 -- 172.21.15.23:0/1512429905 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd15c199310 con 0x7fd15c1070d0 2024-03-28T05:22:40.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.096+0000 7fd158ff9700 1 -- 172.21.15.23:0/1512429905 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd15400a0d0 con 0x7fd15c1070d0 2024-03-28T05:22:40.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.096+0000 7fd158ff9700 1 -- 172.21.15.23:0/1512429905 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd154005480 con 0x7fd15c1070d0 2024-03-28T05:22:40.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.096+0000 7fd1620f6700 1 -- 172.21.15.23:0/1512429905 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd15c1a55f0 con 0x7fd15c1070d0 2024-03-28T05:22:40.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.097+0000 7fd1620f6700 1 -- 172.21.15.23:0/1512429905 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd15c102780 con 0x7fd15c1070d0 2024-03-28T05:22:40.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.099+0000 7fd158ff9700 1 -- 172.21.15.23:0/1512429905 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd154005620 con 0x7fd15c1070d0 2024-03-28T05:22:40.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.099+0000 7fd158ff9700 1 --2- 172.21.15.23:0/1512429905 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd1440a2890 0x7fd1440a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:40.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.099+0000 7fd158ff9700 1 -- 172.21.15.23:0/1512429905 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fd1540ceb40 con 0x7fd15c1070d0 2024-03-28T05:22:40.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.099+0000 7fd15b7fe700 1 --2- 172.21.15.23:0/1512429905 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd1440a2890 0x7fd1440a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:40.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.100+0000 7fd15b7fe700 1 --2- 172.21.15.23:0/1512429905 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd1440a2890 0x7fd1440a4d50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009be0 tx=0x7fd14c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:40.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.107+0000 7fd158ff9700 1 -- 172.21.15.23:0/1512429905 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd154021020 con 0x7fd15c1070d0 2024-03-28T05:22:40.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.117+0000 7f43b0a55700 1 -- 172.21.15.23:0/2123081989 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f43ac02da30 con 0x7f43ac101d30 2024-03-28T05:22:40.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.118+0000 7f439b7fe700 1 -- 172.21.15.23:0/2123081989 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f43a009b690 con 0x7f43ac101d30 2024-03-28T05:22:40.117 INFO:teuthology.orchestra.run.smithi023.stdout:107374182471 2024-03-28T05:22:40.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.120+0000 7f43b0a55700 1 -- 172.21.15.23:0/2123081989 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f43940a2910 msgr2=0x7f43940a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:40.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.120+0000 7f43b0a55700 1 --2- 172.21.15.23:0/2123081989 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f43940a2910 0x7f43940a4dd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f439c000c00 tx=0x7f439c005fb0 comp rx=0 tx=0).stop 2024-03-28T05:22:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.120+0000 7f43b0a55700 1 -- 172.21.15.23:0/2123081989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43ac101d30 msgr2=0x7f43ac197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.120+0000 7f43b0a55700 1 --2- 172.21.15.23:0/2123081989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43ac101d30 0x7f43ac197a90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f43a000e9a0 tx=0x7f43a000ecb0 comp rx=0 tx=0).stop 2024-03-28T05:22:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.120+0000 7f43b0a55700 1 -- 172.21.15.23:0/2123081989 shutdown_connections 2024-03-28T05:22:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.120+0000 7f43b0a55700 1 --2- 172.21.15.23:0/2123081989 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f43940a2910 0x7f43940a4dd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.120+0000 7f43b0a55700 1 --2- 172.21.15.23:0/2123081989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43ac1071c0 0x7f43ac197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.120+0000 7f43b0a55700 1 --2- 172.21.15.23:0/2123081989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43ac101370 0x7f43ac197550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.120+0000 7f43b0a55700 1 --2- 172.21.15.23:0/2123081989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f43ac101d30 0x7f43ac197a90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.120+0000 7f43b0a55700 1 -- 172.21.15.23:0/2123081989 >> 172.21.15.23:0/2123081989 conn(0x7f43ac077f50 msgr2=0x7f43ac100030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.121+0000 7f43b0a55700 1 -- 172.21.15.23:0/2123081989 shutdown_connections 2024-03-28T05:22:40.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.121+0000 7f43b0a55700 1 -- 172.21.15.23:0/2123081989 wait complete. 2024-03-28T05:22:40.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.319+0000 7fd1620f6700 1 -- 172.21.15.23:0/1512429905 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fd15c1a57c0 con 0x7fd15c1070d0 2024-03-28T05:22:40.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.319+0000 7fd158ff9700 1 -- 172.21.15.23:0/1512429905 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd15409b910 con 0x7fd15c1070d0 2024-03-28T05:22:40.318 INFO:teuthology.orchestra.run.smithi023.stdout:197568495672 2024-03-28T05:22:40.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.322+0000 7fd1620f6700 1 -- 172.21.15.23:0/1512429905 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd1440a2890 msgr2=0x7fd1440a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:40.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.322+0000 7fd1620f6700 1 --2- 172.21.15.23:0/1512429905 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd1440a2890 0x7fd1440a4d50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009be0 tx=0x7fd14c009260 comp rx=0 tx=0).stop 2024-03-28T05:22:40.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.322+0000 7fd1620f6700 1 -- 172.21.15.23:0/1512429905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd15c1070d0 msgr2=0x7fd15c198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:40.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.322+0000 7fd1620f6700 1 --2- 172.21.15.23:0/1512429905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd15c1070d0 0x7fd15c198740 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd15400ada0 tx=0x7fd15400c330 comp rx=0 tx=0).stop 2024-03-28T05:22:40.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.322+0000 7fd1620f6700 1 -- 172.21.15.23:0/1512429905 shutdown_connections 2024-03-28T05:22:40.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.322+0000 7fd1620f6700 1 --2- 172.21.15.23:0/1512429905 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fd1440a2890 0x7fd1440a4d50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:40.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.322+0000 7fd1620f6700 1 --2- 172.21.15.23:0/1512429905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd15c1070d0 0x7fd15c198740 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:40.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.322+0000 7fd1620f6700 1 --2- 172.21.15.23:0/1512429905 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd15c101090 0x7fd15c19e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:40.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.322+0000 7fd1620f6700 1 --2- 172.21.15.23:0/1512429905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd15c101a30 0x7fd15c19ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:40.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.322+0000 7fd1620f6700 1 -- 172.21.15.23:0/1512429905 >> 172.21.15.23:0/1512429905 conn(0x7fd15c077fc0 msgr2=0x7fd15c1003d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:40.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.322+0000 7fd1620f6700 1 -- 172.21.15.23:0/1512429905 shutdown_connections 2024-03-28T05:22:40.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:40.322+0000 7fd1620f6700 1 -- 172.21.15.23:0/1512429905 wait complete. 2024-03-28T05:22:40.954 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:40 smithi023 ceph-mon[31954]: pgmap v336: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:40.955 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:40 smithi023 ceph-mon[31954]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-28T05:22:40.955 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:40 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2123081989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T05:22:40.955 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:40 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1512429905' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T05:22:40.980 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495667 got 197568495672 for osd.6 2024-03-28T05:22:40.980 DEBUG:teuthology.parallel:result is None 2024-03-28T05:22:40.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:40 smithi113 ceph-mon[37164]: pgmap v336: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:40.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:40 smithi113 ceph-mon[37164]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-28T05:22:40.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:40 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2123081989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T05:22:40.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:40 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1512429905' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T05:22:41.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:40 smithi098 ceph-mon[36087]: pgmap v336: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:41.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:40 smithi098 ceph-mon[36087]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-28T05:22:41.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:40 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2123081989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T05:22:41.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:40 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1512429905' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T05:22:41.829 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182462 got 107374182471 for osd.3 2024-03-28T05:22:41.829 DEBUG:teuthology.parallel:result is None 2024-03-28T05:22:42.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:41 smithi023 ceph-mon[31954]: pgmap v337: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:22:42.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:41 smithi098 ceph-mon[36087]: pgmap v337: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:22:42.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:41 smithi113 ceph-mon[37164]: pgmap v337: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:22:42.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.356+0000 7f39e6cd6700 1 -- 172.21.15.23:0/1283466281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39e006b1f0 msgr2=0x7f39e010f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:42.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.356+0000 7f39e6cd6700 1 --2- 172.21.15.23:0/1283466281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39e006b1f0 0x7f39e010f030 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f39d800b0a0 tx=0x7f39d800b3b0 comp rx=0 tx=0).stop 2024-03-28T05:22:42.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.356+0000 7f39e6cd6700 1 -- 172.21.15.23:0/1283466281 shutdown_connections 2024-03-28T05:22:42.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.356+0000 7f39e6cd6700 1 --2- 172.21.15.23:0/1283466281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39e006b1f0 0x7f39e010f030 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:42.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.356+0000 7f39e6cd6700 1 --2- 172.21.15.23:0/1283466281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e0108280 0x7f39e0108660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:42.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.356+0000 7f39e6cd6700 1 --2- 172.21.15.23:0/1283466281 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39e006a830 0x7f39e006acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:42.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.356+0000 7f39e6cd6700 1 -- 172.21.15.23:0/1283466281 >> 172.21.15.23:0/1283466281 conn(0x7f39e0077f50 msgr2=0x7f39e0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:42.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.356+0000 7f39e6cd6700 1 -- 172.21.15.23:0/1283466281 shutdown_connections 2024-03-28T05:22:42.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.356+0000 7f39e6cd6700 1 -- 172.21.15.23:0/1283466281 wait complete. 2024-03-28T05:22:42.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.357+0000 7f39e6cd6700 1 Processor -- start 2024-03-28T05:22:42.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.357+0000 7f39e6cd6700 1 -- start start 2024-03-28T05:22:42.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.357+0000 7f39e6cd6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39e006a830 0x7f39e01a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:42.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.357+0000 7f39e6cd6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39e006b1f0 0x7f39e01a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:42.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.357+0000 7f39e6cd6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e0108280 0x7f39e019a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:42.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.357+0000 7f39e6cd6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e01124d0 con 0x7f39e006b1f0 2024-03-28T05:22:42.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.357+0000 7f39e6cd6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e0112350 con 0x7f39e0108280 2024-03-28T05:22:42.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.357+0000 7f39e6cd6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e0112650 con 0x7f39e006a830 2024-03-28T05:22:42.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.357+0000 7f39dffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39e006b1f0 0x7f39e01a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:42.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.357+0000 7f39dffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39e006b1f0 0x7f39e01a0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36132/0 (socket says 172.21.15.23:36132) 2024-03-28T05:22:42.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.357+0000 7f39dffff700 1 -- 172.21.15.23:0/3263723268 learned_addr learned my addr 172.21.15.23:0/3263723268 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:42.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.357+0000 7f39e4a72700 1 --2- 172.21.15.23:0/3263723268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39e006a830 0x7f39e01a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:42.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.357+0000 7f39e5273700 1 --2- 172.21.15.23:0/3263723268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e0108280 0x7f39e019a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:42.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.357+0000 7f39dffff700 1 -- 172.21.15.23:0/3263723268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39e006a830 msgr2=0x7f39e01a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:42.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.357+0000 7f39dffff700 1 --2- 172.21.15.23:0/3263723268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39e006a830 0x7f39e01a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:42.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.357+0000 7f39dffff700 1 -- 172.21.15.23:0/3263723268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e0108280 msgr2=0x7f39e019a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:42.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.357+0000 7f39dffff700 1 --2- 172.21.15.23:0/3263723268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e0108280 0x7f39e019a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:42.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.357+0000 7f39dffff700 1 -- 172.21.15.23:0/3263723268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e019b070 con 0x7f39e006b1f0 2024-03-28T05:22:42.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.358+0000 7f39dffff700 1 --2- 172.21.15.23:0/3263723268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39e006b1f0 0x7f39e01a0c90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f39cc00e9c0 tx=0x7f39cc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:42.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.358+0000 7f39ddffb700 1 -- 172.21.15.23:0/3263723268 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39cc00cbd0 con 0x7f39e006b1f0 2024-03-28T05:22:42.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.358+0000 7f39e6cd6700 1 -- 172.21.15.23:0/3263723268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e019b360 con 0x7f39e006b1f0 2024-03-28T05:22:42.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.358+0000 7f39ddffb700 1 -- 172.21.15.23:0/3263723268 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39cc00cd70 con 0x7f39e006b1f0 2024-03-28T05:22:42.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.358+0000 7f39ddffb700 1 -- 172.21.15.23:0/3263723268 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39cc010430 con 0x7f39e006b1f0 2024-03-28T05:22:42.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.358+0000 7f39e6cd6700 1 -- 172.21.15.23:0/3263723268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e01a7620 con 0x7f39e006b1f0 2024-03-28T05:22:42.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.359+0000 7f39e6cd6700 1 -- 172.21.15.23:0/3263723268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e0068e50 con 0x7f39e006b1f0 2024-03-28T05:22:42.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.360+0000 7f39ddffb700 1 -- 172.21.15.23:0/3263723268 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f39cc004750 con 0x7f39e006b1f0 2024-03-28T05:22:42.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.361+0000 7f39ddffb700 1 --2- 172.21.15.23:0/3263723268 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f39d00a2890 0x7f39d00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:42.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.361+0000 7f39ddffb700 1 -- 172.21.15.23:0/3263723268 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f39cc0cf2a0 con 0x7f39e006b1f0 2024-03-28T05:22:42.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.361+0000 7f39e4a72700 1 --2- 172.21.15.23:0/3263723268 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f39d00a2890 0x7f39d00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:42.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.361+0000 7f39e4a72700 1 --2- 172.21.15.23:0/3263723268 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f39d00a2890 0x7f39d00a4d50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f39e019ba50 tx=0x7f39d40073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:42.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.364+0000 7f39ddffb700 1 -- 172.21.15.23:0/3263723268 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f39cc053740 con 0x7f39e006b1f0 2024-03-28T05:22:42.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.571+0000 7f39e6cd6700 1 -- 172.21.15.23:0/3263723268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f39e01a78b0 con 0x7f39e006b1f0 2024-03-28T05:22:42.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.572+0000 7f39ddffb700 1 -- 172.21.15.23:0/3263723268 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f39cc09bfe0 con 0x7f39e006b1f0 2024-03-28T05:22:42.571 INFO:teuthology.orchestra.run.smithi023.stdout:77309411405 2024-03-28T05:22:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.574+0000 7f39e6cd6700 1 -- 172.21.15.23:0/3263723268 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f39d00a2890 msgr2=0x7f39d00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.574+0000 7f39e6cd6700 1 --2- 172.21.15.23:0/3263723268 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f39d00a2890 0x7f39d00a4d50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f39e019ba50 tx=0x7f39d40073d0 comp rx=0 tx=0).stop 2024-03-28T05:22:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.574+0000 7f39e6cd6700 1 -- 172.21.15.23:0/3263723268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39e006b1f0 msgr2=0x7f39e01a0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.574+0000 7f39e6cd6700 1 --2- 172.21.15.23:0/3263723268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39e006b1f0 0x7f39e01a0c90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f39cc00e9c0 tx=0x7f39cc00ecd0 comp rx=0 tx=0).stop 2024-03-28T05:22:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.574+0000 7f39e6cd6700 1 -- 172.21.15.23:0/3263723268 shutdown_connections 2024-03-28T05:22:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.574+0000 7f39e6cd6700 1 --2- 172.21.15.23:0/3263723268 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f39d00a2890 0x7f39d00a4d50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:42.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.574+0000 7f39e6cd6700 1 --2- 172.21.15.23:0/3263723268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e0108280 0x7f39e019a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:42.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.574+0000 7f39e6cd6700 1 --2- 172.21.15.23:0/3263723268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f39e006a830 0x7f39e01a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:42.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.574+0000 7f39e6cd6700 1 --2- 172.21.15.23:0/3263723268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39e006b1f0 0x7f39e01a0c90 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:42.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.574+0000 7f39e6cd6700 1 -- 172.21.15.23:0/3263723268 >> 172.21.15.23:0/3263723268 conn(0x7f39e0077f50 msgr2=0x7f39e00ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:42.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.574+0000 7f39e6cd6700 1 -- 172.21.15.23:0/3263723268 shutdown_connections 2024-03-28T05:22:42.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:42.574+0000 7f39e6cd6700 1 -- 172.21.15.23:0/3263723268 wait complete. 2024-03-28T05:22:43.236 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:43 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:22:43.236 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:43 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:22:43.236 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:43 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3263723268' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T05:22:43.236 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:43 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:22:43.236 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:43 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:22:43.236 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:43 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:22:43.260 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411399 got 77309411405 for osd.2 2024-03-28T05:22:43.260 DEBUG:teuthology.parallel:result is None 2024-03-28T05:22:43.260 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-28T05:22:43.260 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T05:22:43.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:43 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:22:43.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:43 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:22:43.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:43 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3263723268' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T05:22:43.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:43 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:22:43.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:43 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:22:43.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:43 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:22:43.502 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:22:43.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:43 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:22:43.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:43 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:22:43.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:43 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3263723268' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T05:22:43.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:43 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:22:43.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:43 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:22:43.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:43 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' 2024-03-28T05:22:44.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:44 smithi023 ceph-mon[31954]: pgmap v338: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 1 op/s 2024-03-28T05:22:44.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:44 smithi113 ceph-mon[37164]: pgmap v338: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 1 op/s 2024-03-28T05:22:44.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:44 smithi098 ceph-mon[36087]: pgmap v338: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 1 op/s 2024-03-28T05:22:45.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.161+0000 7f6b32705700 1 -- 172.21.15.23:0/3176537562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b2c102e20 msgr2=0x7f6b2c106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:45.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.161+0000 7f6b32705700 1 --2- 172.21.15.23:0/3176537562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b2c102e20 0x7f6b2c106dc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6b20009990 tx=0x7f6b20009ca0 comp rx=0 tx=0).stop 2024-03-28T05:22:45.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.161+0000 7f6b32705700 1 -- 172.21.15.23:0/3176537562 shutdown_connections 2024-03-28T05:22:45.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.161+0000 7f6b32705700 1 --2- 172.21.15.23:0/3176537562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6b2c1074a0 0x7f6b2c10f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:45.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.161+0000 7f6b32705700 1 --2- 172.21.15.23:0/3176537562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b2c102470 0x7f6b2c102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:45.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.161+0000 7f6b32705700 1 --2- 172.21.15.23:0/3176537562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b2c102e20 0x7f6b2c106dc0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:45.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.161+0000 7f6b32705700 1 -- 172.21.15.23:0/3176537562 >> 172.21.15.23:0/3176537562 conn(0x7f6b2c079220 msgr2=0x7f6b2c079630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:45.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.161+0000 7f6b32705700 1 -- 172.21.15.23:0/3176537562 shutdown_connections 2024-03-28T05:22:45.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.161+0000 7f6b32705700 1 -- 172.21.15.23:0/3176537562 wait complete. 2024-03-28T05:22:45.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.161+0000 7f6b32705700 1 Processor -- start 2024-03-28T05:22:45.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.161+0000 7f6b32705700 1 -- start start 2024-03-28T05:22:45.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.162+0000 7f6b32705700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b2c102470 0x7f6b2c074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:45.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.162+0000 7f6b32705700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b2c102e20 0x7f6b2c0749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:45.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.162+0000 7f6b32705700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6b2c1074a0 0x7f6b2c074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:45.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.162+0000 7f6b32705700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c112440 con 0x7f6b2c102470 2024-03-28T05:22:45.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.162+0000 7f6b32705700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c1122c0 con 0x7f6b2c102e20 2024-03-28T05:22:45.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.162+0000 7f6b32705700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c1125c0 con 0x7f6b2c1074a0 2024-03-28T05:22:45.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.162+0000 7f6b2bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b2c102470 0x7f6b2c074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:45.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.162+0000 7f6b2bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b2c102470 0x7f6b2c074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43792/0 (socket says 172.21.15.23:43792) 2024-03-28T05:22:45.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.162+0000 7f6b2bfff700 1 -- 172.21.15.23:0/1573717622 learned_addr learned my addr 172.21.15.23:0/1573717622 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:45.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.162+0000 7f6b30ca2700 1 --2- 172.21.15.23:0/1573717622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6b2c1074a0 0x7f6b2c074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:45.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.162+0000 7f6b2b7fe700 1 --2- 172.21.15.23:0/1573717622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b2c102e20 0x7f6b2c0749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:45.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.162+0000 7f6b2bfff700 1 -- 172.21.15.23:0/1573717622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6b2c1074a0 msgr2=0x7f6b2c074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:45.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.162+0000 7f6b2bfff700 1 --2- 172.21.15.23:0/1573717622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6b2c1074a0 0x7f6b2c074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:45.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.162+0000 7f6b2bfff700 1 -- 172.21.15.23:0/1573717622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b2c102e20 msgr2=0x7f6b2c0749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:45.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.162+0000 7f6b2bfff700 1 --2- 172.21.15.23:0/1573717622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b2c102e20 0x7f6b2c0749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:45.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.162+0000 7f6b2bfff700 1 -- 172.21.15.23:0/1573717622 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2c1a6e00 con 0x7f6b2c102470 2024-03-28T05:22:45.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.162+0000 7f6b2bfff700 1 --2- 172.21.15.23:0/1573717622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b2c102470 0x7f6b2c074490 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c00c9b0 tx=0x7f6b1c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:45.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.163+0000 7f6b297fa700 1 -- 172.21.15.23:0/1573717622 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c007b00 con 0x7f6b2c102470 2024-03-28T05:22:45.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.163+0000 7f6b297fa700 1 -- 172.21.15.23:0/1573717622 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b1c00ce50 con 0x7f6b2c102470 2024-03-28T05:22:45.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.163+0000 7f6b32705700 1 -- 172.21.15.23:0/1573717622 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b2c1a70f0 con 0x7f6b2c102470 2024-03-28T05:22:45.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.163+0000 7f6b297fa700 1 -- 172.21.15.23:0/1573717622 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c018660 con 0x7f6b2c102470 2024-03-28T05:22:45.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.163+0000 7f6b32705700 1 -- 172.21.15.23:0/1573717622 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b2c1a7600 con 0x7f6b2c102470 2024-03-28T05:22:45.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.164+0000 7f6b32705700 1 -- 172.21.15.23:0/1573717622 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b2c068e50 con 0x7f6b2c102470 2024-03-28T05:22:45.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.165+0000 7f6b297fa700 1 -- 172.21.15.23:0/1573717622 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6b1c00f450 con 0x7f6b2c102470 2024-03-28T05:22:45.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.165+0000 7f6b297fa700 1 --2- 172.21.15.23:0/1573717622 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f6b140a2890 0x7f6b140a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:45.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.166+0000 7f6b2b7fe700 1 --2- 172.21.15.23:0/1573717622 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f6b140a2890 0x7f6b140a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:45.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.166+0000 7f6b297fa700 1 -- 172.21.15.23:0/1573717622 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f6b1c0cecc0 con 0x7f6b2c102470 2024-03-28T05:22:45.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.166+0000 7f6b2b7fe700 1 --2- 172.21.15.23:0/1573717622 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f6b140a2890 0x7f6b140a4d50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6b20006010 tx=0x7f6b20005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:45.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.170+0000 7f6b297fa700 1 -- 172.21.15.23:0/1573717622 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6b1c014030 con 0x7f6b2c102470 2024-03-28T05:22:45.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.372+0000 7f6b32705700 1 -- 172.21.15.23:0/1573717622 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6b2c0625d0 con 0x7f6b140a2890 2024-03-28T05:22:45.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.382+0000 7f6b297fa700 1 -- 172.21.15.23:0/1573717622 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340506 (secure 0 0 0) 0x7f6b2c0625d0 con 0x7f6b140a2890 2024-03-28T05:22:45.384 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:22:45.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.385+0000 7f6b32705700 1 -- 172.21.15.23:0/1573717622 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f6b140a2890 msgr2=0x7f6b140a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:45.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.385+0000 7f6b32705700 1 --2- 172.21.15.23:0/1573717622 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f6b140a2890 0x7f6b140a4d50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6b20006010 tx=0x7f6b20005960 comp rx=0 tx=0).stop 2024-03-28T05:22:45.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.385+0000 7f6b32705700 1 -- 172.21.15.23:0/1573717622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b2c102470 msgr2=0x7f6b2c074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:45.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.385+0000 7f6b32705700 1 --2- 172.21.15.23:0/1573717622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b2c102470 0x7f6b2c074490 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c00c9b0 tx=0x7f6b1c00ccc0 comp rx=0 tx=0).stop 2024-03-28T05:22:45.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.385+0000 7f6b32705700 1 -- 172.21.15.23:0/1573717622 shutdown_connections 2024-03-28T05:22:45.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.385+0000 7f6b32705700 1 --2- 172.21.15.23:0/1573717622 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f6b140a2890 0x7f6b140a4d50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:45.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.385+0000 7f6b32705700 1 --2- 172.21.15.23:0/1573717622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6b2c1074a0 0x7f6b2c074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:45.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.385+0000 7f6b32705700 1 --2- 172.21.15.23:0/1573717622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b2c102470 0x7f6b2c074490 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:45.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.385+0000 7f6b32705700 1 --2- 172.21.15.23:0/1573717622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b2c102e20 0x7f6b2c0749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:45.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.385+0000 7f6b32705700 1 -- 172.21.15.23:0/1573717622 >> 172.21.15.23:0/1573717622 conn(0x7f6b2c079220 msgr2=0x7f6b2c1002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:45.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.385+0000 7f6b32705700 1 -- 172.21.15.23:0/1573717622 shutdown_connections 2024-03-28T05:22:45.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:45.385+0000 7f6b32705700 1 -- 172.21.15.23:0/1573717622 wait complete. 2024-03-28T05:22:45.388 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-28T05:22:46.047 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-03-28T05:22:44.745120+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654491,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":297,"num_read_kb":265,"num_write":268,"num_write_kb":3034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1180853,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":712,"ondisk_log_size":712,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367264,"kb_used_data":13952,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574560,"statfs":{"total":1151940427776,"available":1151564349440,"internally_reserved":0,"allocated":14286848,"data_stored":7949183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":249,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":16,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003503"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.766601+0000","last_change":"2024-03-28T05:21:08.043720+0000","last_active":"2024-03-28T05:22:38.766601+0000","last_peered":"2024-03-28T05:22:38.766601+0000","last_clean":"2024-03-28T05:22:38.766601+0000","last_became_active":"2024-03-28T05:21:08.043250+0000","last_became_peered":"2024-03-28T05:21:08.043250+0000","last_unstale":"2024-03-28T05:22:38.766601+0000","last_undegraded":"2024-03-28T05:22:38.766601+0000","last_fullsized":"2024-03-28T05:22:38.766601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:29:10.144086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.772608+0000","last_change":"2024-03-28T05:21:08.053247+0000","last_active":"2024-03-28T05:22:38.772608+0000","last_peered":"2024-03-28T05:22:38.772608+0000","last_clean":"2024-03-28T05:22:38.772608+0000","last_became_active":"2024-03-28T05:21:08.052941+0000","last_became_peered":"2024-03-28T05:21:08.052941+0000","last_unstale":"2024-03-28T05:22:38.772608+0000","last_undegraded":"2024-03-28T05:22:38.772608+0000","last_fullsized":"2024-03-28T05:22:38.772608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:41:22.223858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.759932+0000","last_change":"2024-03-28T05:21:08.051938+0000","last_active":"2024-03-28T05:22:38.759932+0000","last_peered":"2024-03-28T05:22:38.759932+0000","last_clean":"2024-03-28T05:22:38.759932+0000","last_became_active":"2024-03-28T05:21:08.050366+0000","last_became_peered":"2024-03-28T05:21:08.050366+0000","last_unstale":"2024-03-28T05:22:38.759932+0000","last_undegraded":"2024-03-28T05:22:38.759932+0000","last_fullsized":"2024-03-28T05:22:38.759932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:27:29.325009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.747999+0000","last_change":"2024-03-28T05:21:08.043371+0000","last_active":"2024-03-28T05:22:38.747999+0000","last_peered":"2024-03-28T05:22:38.747999+0000","last_clean":"2024-03-28T05:22:38.747999+0000","last_became_active":"2024-03-28T05:21:08.043038+0000","last_became_peered":"2024-03-28T05:21:08.043038+0000","last_unstale":"2024-03-28T05:22:38.747999+0000","last_undegraded":"2024-03-28T05:22:38.747999+0000","last_fullsized":"2024-03-28T05:22:38.747999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:16:14.137248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.740392+0000","last_change":"2024-03-28T05:21:08.048678+0000","last_active":"2024-03-28T05:22:38.740392+0000","last_peered":"2024-03-28T05:22:38.740392+0000","last_clean":"2024-03-28T05:22:38.740392+0000","last_became_active":"2024-03-28T05:21:08.048272+0000","last_became_peered":"2024-03-28T05:21:08.048272+0000","last_unstale":"2024-03-28T05:22:38.740392+0000","last_undegraded":"2024-03-28T05:22:38.740392+0000","last_fullsized":"2024-03-28T05:22:38.740392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:34:29.029249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.772420+0000","last_change":"2024-03-28T05:21:08.041764+0000","last_active":"2024-03-28T05:22:38.772420+0000","last_peered":"2024-03-28T05:22:38.772420+0000","last_clean":"2024-03-28T05:22:38.772420+0000","last_became_active":"2024-03-28T05:21:08.041337+0000","last_became_peered":"2024-03-28T05:21:08.041337+0000","last_unstale":"2024-03-28T05:22:38.772420+0000","last_undegraded":"2024-03-28T05:22:38.772420+0000","last_fullsized":"2024-03-28T05:22:38.772420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:42:29.136829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.764626+0000","last_change":"2024-03-28T05:22:10.518391+0000","last_active":"2024-03-28T05:22:38.764626+0000","last_peered":"2024-03-28T05:22:38.764626+0000","last_clean":"2024-03-28T05:22:38.764626+0000","last_became_active":"2024-03-28T05:22:10.518052+0000","last_became_peered":"2024-03-28T05:22:10.518052+0000","last_unstale":"2024-03-28T05:22:38.764626+0000","last_undegraded":"2024-03-28T05:22:38.764626+0000","last_fullsized":"2024-03-28T05:22:38.764626+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:39:06.337034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.772436+0000","last_change":"2024-03-28T05:21:08.048605+0000","last_active":"2024-03-28T05:22:38.772436+0000","last_peered":"2024-03-28T05:22:38.772436+0000","last_clean":"2024-03-28T05:22:38.772436+0000","last_became_active":"2024-03-28T05:21:08.048279+0000","last_became_peered":"2024-03-28T05:21:08.048279+0000","last_unstale":"2024-03-28T05:22:38.772436+0000","last_undegraded":"2024-03-28T05:22:38.772436+0000","last_fullsized":"2024-03-28T05:22:38.772436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:34:00.935133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.769056+0000","last_change":"2024-03-28T05:21:08.051357+0000","last_active":"2024-03-28T05:22:38.769056+0000","last_peered":"2024-03-28T05:22:38.769056+0000","last_clean":"2024-03-28T05:22:38.769056+0000","last_became_active":"2024-03-28T05:21:08.051012+0000","last_became_peered":"2024-03-28T05:21:08.051012+0000","last_unstale":"2024-03-28T05:22:38.769056+0000","last_undegraded":"2024-03-28T05:22:38.769056+0000","last_fullsized":"2024-03-28T05:22:38.769056+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:43:42.090619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.769083+0000","last_change":"2024-03-28T05:21:08.038727+0000","last_active":"2024-03-28T05:22:38.769083+0000","last_peered":"2024-03-28T05:22:38.769083+0000","last_clean":"2024-03-28T05:22:38.769083+0000","last_became_active":"2024-03-28T05:21:08.038425+0000","last_became_peered":"2024-03-28T05:21:08.038425+0000","last_unstale":"2024-03-28T05:22:38.769083+0000","last_undegraded":"2024-03-28T05:22:38.769083+0000","last_fullsized":"2024-03-28T05:22:38.769083+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:06:53.786136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.764629+0000","last_change":"2024-03-28T05:21:08.054698+0000","last_active":"2024-03-28T05:22:38.764629+0000","last_peered":"2024-03-28T05:22:38.764629+0000","last_clean":"2024-03-28T05:22:38.764629+0000","last_became_active":"2024-03-28T05:21:08.054310+0000","last_became_peered":"2024-03-28T05:21:08.054310+0000","last_unstale":"2024-03-28T05:22:38.764629+0000","last_undegraded":"2024-03-28T05:22:38.764629+0000","last_fullsized":"2024-03-28T05:22:38.764629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:09:47.996381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.769179+0000","last_change":"2024-03-28T05:21:08.042826+0000","last_active":"2024-03-28T05:22:38.769179+0000","last_peered":"2024-03-28T05:22:38.769179+0000","last_clean":"2024-03-28T05:22:38.769179+0000","last_became_active":"2024-03-28T05:21:08.042487+0000","last_became_peered":"2024-03-28T05:21:08.042487+0000","last_unstale":"2024-03-28T05:22:38.769179+0000","last_undegraded":"2024-03-28T05:22:38.769179+0000","last_fullsized":"2024-03-28T05:22:38.769179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:47:06.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.768692+0000","last_change":"2024-03-28T05:21:08.030791+0000","last_active":"2024-03-28T05:22:38.768692+0000","last_peered":"2024-03-28T05:22:38.768692+0000","last_clean":"2024-03-28T05:22:38.768692+0000","last_became_active":"2024-03-28T05:21:08.030447+0000","last_became_peered":"2024-03-28T05:21:08.030447+0000","last_unstale":"2024-03-28T05:22:38.768692+0000","last_undegraded":"2024-03-28T05:22:38.768692+0000","last_fullsized":"2024-03-28T05:22:38.768692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:54:53.454037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.772664+0000","last_change":"2024-03-28T05:21:08.042477+0000","last_active":"2024-03-28T05:22:38.772664+0000","last_peered":"2024-03-28T05:22:38.772664+0000","last_clean":"2024-03-28T05:22:38.772664+0000","last_became_active":"2024-03-28T05:21:08.042164+0000","last_became_peered":"2024-03-28T05:21:08.042164+0000","last_unstale":"2024-03-28T05:22:38.772664+0000","last_undegraded":"2024-03-28T05:22:38.772664+0000","last_fullsized":"2024-03-28T05:22:38.772664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:44:25.566195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.765642+0000","last_change":"2024-03-28T05:21:08.050635+0000","last_active":"2024-03-28T05:22:38.765642+0000","last_peered":"2024-03-28T05:22:38.765642+0000","last_clean":"2024-03-28T05:22:38.765642+0000","last_became_active":"2024-03-28T05:21:08.050217+0000","last_became_peered":"2024-03-28T05:21:08.050217+0000","last_unstale":"2024-03-28T05:22:38.765642+0000","last_undegraded":"2024-03-28T05:22:38.765642+0000","last_fullsized":"2024-03-28T05:22:38.765642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:13:49.252755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.770510+0000","last_change":"2024-03-28T05:21:08.048021+0000","last_active":"2024-03-28T05:22:38.770510+0000","last_peered":"2024-03-28T05:22:38.770510+0000","last_clean":"2024-03-28T05:22:38.770510+0000","last_became_active":"2024-03-28T05:21:08.047381+0000","last_became_peered":"2024-03-28T05:21:08.047381+0000","last_unstale":"2024-03-28T05:22:38.770510+0000","last_undegraded":"2024-03-28T05:22:38.770510+0000","last_fullsized":"2024-03-28T05:22:38.770510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:42:39.718804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.766474+0000","last_change":"2024-03-28T05:21:08.044774+0000","last_active":"2024-03-28T05:22:38.766474+0000","last_peered":"2024-03-28T05:22:38.766474+0000","last_clean":"2024-03-28T05:22:38.766474+0000","last_became_active":"2024-03-28T05:21:08.044470+0000","last_became_peered":"2024-03-28T05:21:08.044470+0000","last_unstale":"2024-03-28T05:22:38.766474+0000","last_undegraded":"2024-03-28T05:22:38.766474+0000","last_fullsized":"2024-03-28T05:22:38.766474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:26:11.354288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.771590+0000","last_change":"2024-03-28T05:21:08.042871+0000","last_active":"2024-03-28T05:22:38.771590+0000","last_peered":"2024-03-28T05:22:38.771590+0000","last_clean":"2024-03-28T05:22:38.771590+0000","last_became_active":"2024-03-28T05:21:08.042506+0000","last_became_peered":"2024-03-28T05:21:08.042506+0000","last_unstale":"2024-03-28T05:22:38.771590+0000","last_undegraded":"2024-03-28T05:22:38.771590+0000","last_fullsized":"2024-03-28T05:22:38.771590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:34:50.424380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.751454+0000","last_change":"2024-03-28T05:21:08.045056+0000","last_active":"2024-03-28T05:22:38.751454+0000","last_peered":"2024-03-28T05:22:38.751454+0000","last_clean":"2024-03-28T05:22:38.751454+0000","last_became_active":"2024-03-28T05:21:08.044719+0000","last_became_peered":"2024-03-28T05:21:08.044719+0000","last_unstale":"2024-03-28T05:22:38.751454+0000","last_undegraded":"2024-03-28T05:22:38.751454+0000","last_fullsized":"2024-03-28T05:22:38.751454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:23:12.479829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.508781+0000","last_change":"2024-03-28T05:21:12.295239+0000","last_active":"2024-03-28T05:22:10.508781+0000","last_peered":"2024-03-28T05:22:10.508781+0000","last_clean":"2024-03-28T05:22:10.508781+0000","last_became_active":"2024-03-28T05:21:12.294843+0000","last_became_peered":"2024-03-28T05:21:12.294843+0000","last_unstale":"2024-03-28T05:22:10.508781+0000","last_undegraded":"2024-03-28T05:22:10.508781+0000","last_fullsized":"2024-03-28T05:22:10.508781+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:36:35.291983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.513954+0000","last_change":"2024-03-28T05:21:07.270833+0000","last_active":"2024-03-28T05:22:10.513954+0000","last_peered":"2024-03-28T05:22:10.513954+0000","last_clean":"2024-03-28T05:22:10.513954+0000","last_became_active":"2024-03-28T05:21:07.270288+0000","last_became_peered":"2024-03-28T05:21:07.270288+0000","last_unstale":"2024-03-28T05:22:10.513954+0000","last_undegraded":"2024-03-28T05:22:10.513954+0000","last_fullsized":"2024-03-28T05:22:10.513954+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:41:05.494851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.741663+0000","last_change":"2024-03-28T05:22:10.515698+0000","last_active":"2024-03-28T05:22:38.741663+0000","last_peered":"2024-03-28T05:22:38.741663+0000","last_clean":"2024-03-28T05:22:38.741663+0000","last_became_active":"2024-03-28T05:22:10.515275+0000","last_became_peered":"2024-03-28T05:22:10.515275+0000","last_unstale":"2024-03-28T05:22:38.741663+0000","last_undegraded":"2024-03-28T05:22:38.741663+0000","last_fullsized":"2024-03-28T05:22:38.741663+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:16:03.215899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.511056+0000","last_change":"2024-03-28T05:21:12.295599+0000","last_active":"2024-03-28T05:22:10.511056+0000","last_peered":"2024-03-28T05:22:10.511056+0000","last_clean":"2024-03-28T05:22:10.511056+0000","last_became_active":"2024-03-28T05:21:12.295161+0000","last_became_peered":"2024-03-28T05:21:12.295161+0000","last_unstale":"2024-03-28T05:22:10.511056+0000","last_undegraded":"2024-03-28T05:22:10.511056+0000","last_fullsized":"2024-03-28T05:22:10.511056+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:28:39.097018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.276535+0000","last_change":"2024-03-28T05:21:07.264532+0000","last_active":"2024-03-28T05:22:11.276535+0000","last_peered":"2024-03-28T05:22:11.276535+0000","last_clean":"2024-03-28T05:22:11.276535+0000","last_became_active":"2024-03-28T05:21:07.264121+0000","last_became_peered":"2024-03-28T05:21:07.264121+0000","last_unstale":"2024-03-28T05:22:11.276535+0000","last_undegraded":"2024-03-28T05:22:11.276535+0000","last_fullsized":"2024-03-28T05:22:11.276535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:40:01.927165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.771177+0000","last_change":"2024-03-28T05:21:08.045547+0000","last_active":"2024-03-28T05:22:38.771177+0000","last_peered":"2024-03-28T05:22:38.771177+0000","last_clean":"2024-03-28T05:22:38.771177+0000","last_became_active":"2024-03-28T05:21:08.045243+0000","last_became_peered":"2024-03-28T05:21:08.045243+0000","last_unstale":"2024-03-28T05:22:38.771177+0000","last_undegraded":"2024-03-28T05:22:38.771177+0000","last_fullsized":"2024-03-28T05:22:38.771177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:24:51.178938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.511119+0000","last_change":"2024-03-28T05:21:12.301461+0000","last_active":"2024-03-28T05:22:10.511119+0000","last_peered":"2024-03-28T05:22:10.511119+0000","last_clean":"2024-03-28T05:22:10.511119+0000","last_became_active":"2024-03-28T05:21:12.300765+0000","last_became_peered":"2024-03-28T05:21:12.300765+0000","last_unstale":"2024-03-28T05:22:10.511119+0000","last_undegraded":"2024-03-28T05:22:10.511119+0000","last_fullsized":"2024-03-28T05:22:10.511119+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:10:38.609978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.365258+0000","last_change":"2024-03-28T05:21:07.269209+0000","last_active":"2024-03-28T05:22:11.365258+0000","last_peered":"2024-03-28T05:22:11.365258+0000","last_clean":"2024-03-28T05:22:11.365258+0000","last_became_active":"2024-03-28T05:21:07.268068+0000","last_became_peered":"2024-03-28T05:21:07.268068+0000","last_unstale":"2024-03-28T05:22:11.365258+0000","last_undegraded":"2024-03-28T05:22:11.365258+0000","last_fullsized":"2024-03-28T05:22:11.365258+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:44:53.135439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.769265+0000","last_change":"2024-03-28T05:21:08.045233+0000","last_active":"2024-03-28T05:22:38.769265+0000","last_peered":"2024-03-28T05:22:38.769265+0000","last_clean":"2024-03-28T05:22:38.769265+0000","last_became_active":"2024-03-28T05:21:08.044878+0000","last_became_peered":"2024-03-28T05:21:08.044878+0000","last_unstale":"2024-03-28T05:22:38.769265+0000","last_undegraded":"2024-03-28T05:22:38.769265+0000","last_fullsized":"2024-03-28T05:22:38.769265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:15:50.830519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.511121+0000","last_change":"2024-03-28T05:21:12.296326+0000","last_active":"2024-03-28T05:22:10.511121+0000","last_peered":"2024-03-28T05:22:10.511121+0000","last_clean":"2024-03-28T05:22:10.511121+0000","last_became_active":"2024-03-28T05:21:12.295863+0000","last_became_peered":"2024-03-28T05:21:12.295863+0000","last_unstale":"2024-03-28T05:22:10.511121+0000","last_undegraded":"2024-03-28T05:22:10.511121+0000","last_fullsized":"2024-03-28T05:22:10.511121+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:26:49.340900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.365112+0000","last_change":"2024-03-28T05:21:07.263689+0000","last_active":"2024-03-28T05:22:11.365112+0000","last_peered":"2024-03-28T05:22:11.365112+0000","last_clean":"2024-03-28T05:22:11.365112+0000","last_became_active":"2024-03-28T05:21:07.263410+0000","last_became_peered":"2024-03-28T05:21:07.263410+0000","last_unstale":"2024-03-28T05:22:11.365112+0000","last_undegraded":"2024-03-28T05:22:11.365112+0000","last_fullsized":"2024-03-28T05:22:11.365112+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:51:03.017620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.764612+0000","last_change":"2024-03-28T05:22:11.367486+0000","last_active":"2024-03-28T05:22:38.764612+0000","last_peered":"2024-03-28T05:22:38.764612+0000","last_clean":"2024-03-28T05:22:38.764612+0000","last_became_active":"2024-03-28T05:22:11.367149+0000","last_became_peered":"2024-03-28T05:22:11.367149+0000","last_unstale":"2024-03-28T05:22:38.764612+0000","last_undegraded":"2024-03-28T05:22:38.764612+0000","last_fullsized":"2024-03-28T05:22:38.764612+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:24:37.312732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.508870+0000","last_change":"2024-03-28T05:21:12.284248+0000","last_active":"2024-03-28T05:22:10.508870+0000","last_peered":"2024-03-28T05:22:10.508870+0000","last_clean":"2024-03-28T05:22:10.508870+0000","last_became_active":"2024-03-28T05:21:12.283887+0000","last_became_peered":"2024-03-28T05:21:12.283887+0000","last_unstale":"2024-03-28T05:22:10.508870+0000","last_undegraded":"2024-03-28T05:22:10.508870+0000","last_fullsized":"2024-03-28T05:22:10.508870+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:50:50.871119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.512917+0000","last_change":"2024-03-28T05:21:07.266436+0000","last_active":"2024-03-28T05:22:10.512917+0000","last_peered":"2024-03-28T05:22:10.512917+0000","last_clean":"2024-03-28T05:22:10.512917+0000","last_became_active":"2024-03-28T05:21:07.266115+0000","last_became_peered":"2024-03-28T05:21:07.266115+0000","last_unstale":"2024-03-28T05:22:10.512917+0000","last_undegraded":"2024-03-28T05:22:10.512917+0000","last_fullsized":"2024-03-28T05:22:10.512917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:42:06.865456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.764385+0000","last_change":"2024-03-28T05:21:08.047500+0000","last_active":"2024-03-28T05:22:38.764385+0000","last_peered":"2024-03-28T05:22:38.764385+0000","last_clean":"2024-03-28T05:22:38.764385+0000","last_became_active":"2024-03-28T05:21:08.047210+0000","last_became_peered":"2024-03-28T05:21:08.047210+0000","last_unstale":"2024-03-28T05:22:38.764385+0000","last_undegraded":"2024-03-28T05:22:38.764385+0000","last_fullsized":"2024-03-28T05:22:38.764385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:31:02.393561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.367586+0000","last_change":"2024-03-28T05:22:11.367586+0000","last_active":"2024-03-28T05:22:11.367586+0000","last_peered":"2024-03-28T05:22:11.367586+0000","last_clean":"2024-03-28T05:22:11.367586+0000","last_became_active":"2024-03-28T05:22:11.367122+0000","last_became_peered":"2024-03-28T05:22:11.367122+0000","last_unstale":"2024-03-28T05:22:11.367586+0000","last_undegraded":"2024-03-28T05:22:11.367586+0000","last_fullsized":"2024-03-28T05:22:11.367586+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:59:56.177807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.514783+0000","last_change":"2024-03-28T05:21:07.265869+0000","last_active":"2024-03-28T05:22:10.514783+0000","last_peered":"2024-03-28T05:22:10.514783+0000","last_clean":"2024-03-28T05:22:10.514783+0000","last_became_active":"2024-03-28T05:21:07.265586+0000","last_became_peered":"2024-03-28T05:21:07.265586+0000","last_unstale":"2024-03-28T05:22:10.514783+0000","last_undegraded":"2024-03-28T05:22:10.514783+0000","last_fullsized":"2024-03-28T05:22:10.514783+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:17:39.328864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.766512+0000","last_change":"2024-03-28T05:21:08.046780+0000","last_active":"2024-03-28T05:22:38.766512+0000","last_peered":"2024-03-28T05:22:38.766512+0000","last_clean":"2024-03-28T05:22:38.766512+0000","last_became_active":"2024-03-28T05:21:08.046484+0000","last_became_peered":"2024-03-28T05:21:08.046484+0000","last_unstale":"2024-03-28T05:22:38.766512+0000","last_undegraded":"2024-03-28T05:22:38.766512+0000","last_fullsized":"2024-03-28T05:22:38.766512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:02:08.388581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.516843+0000","last_change":"2024-03-28T05:21:12.288723+0000","last_active":"2024-03-28T05:22:10.516843+0000","last_peered":"2024-03-28T05:22:10.516843+0000","last_clean":"2024-03-28T05:22:10.516843+0000","last_became_active":"2024-03-28T05:21:12.288310+0000","last_became_peered":"2024-03-28T05:21:12.288310+0000","last_unstale":"2024-03-28T05:22:10.516843+0000","last_undegraded":"2024-03-28T05:22:10.516843+0000","last_fullsized":"2024-03-28T05:22:10.516843+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:15:39.953732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.513832+0000","last_change":"2024-03-28T05:21:07.257433+0000","last_active":"2024-03-28T05:22:10.513832+0000","last_peered":"2024-03-28T05:22:10.513832+0000","last_clean":"2024-03-28T05:22:10.513832+0000","last_became_active":"2024-03-28T05:21:07.257099+0000","last_became_peered":"2024-03-28T05:21:07.257099+0000","last_unstale":"2024-03-28T05:22:10.513832+0000","last_undegraded":"2024-03-28T05:22:10.513832+0000","last_fullsized":"2024-03-28T05:22:10.513832+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:59:24.580079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.761171+0000","last_change":"2024-03-28T05:21:08.047963+0000","last_active":"2024-03-28T05:22:38.761171+0000","last_peered":"2024-03-28T05:22:38.761171+0000","last_clean":"2024-03-28T05:22:38.761171+0000","last_became_active":"2024-03-28T05:21:08.046403+0000","last_became_peered":"2024-03-28T05:21:08.046403+0000","last_unstale":"2024-03-28T05:22:38.761171+0000","last_undegraded":"2024-03-28T05:22:38.761171+0000","last_fullsized":"2024-03-28T05:22:38.761171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:58:45.280549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.777471+0000","last_change":"2024-03-28T05:21:12.284618+0000","last_active":"2024-03-28T05:22:10.777471+0000","last_peered":"2024-03-28T05:22:10.777471+0000","last_clean":"2024-03-28T05:22:10.777471+0000","last_became_active":"2024-03-28T05:21:12.284185+0000","last_became_peered":"2024-03-28T05:21:12.284185+0000","last_unstale":"2024-03-28T05:22:10.777471+0000","last_undegraded":"2024-03-28T05:22:10.777471+0000","last_fullsized":"2024-03-28T05:22:10.777471+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:15:48.715607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.512753+0000","last_change":"2024-03-28T05:21:07.265493+0000","last_active":"2024-03-28T05:22:10.512753+0000","last_peered":"2024-03-28T05:22:10.512753+0000","last_clean":"2024-03-28T05:22:10.512753+0000","last_became_active":"2024-03-28T05:21:07.265182+0000","last_became_peered":"2024-03-28T05:21:07.265182+0000","last_unstale":"2024-03-28T05:22:10.512753+0000","last_undegraded":"2024-03-28T05:22:10.512753+0000","last_fullsized":"2024-03-28T05:22:10.512753+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:55:00.151367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.771646+0000","last_change":"2024-03-28T05:21:08.041161+0000","last_active":"2024-03-28T05:22:38.771646+0000","last_peered":"2024-03-28T05:22:38.771646+0000","last_clean":"2024-03-28T05:22:38.771646+0000","last_became_active":"2024-03-28T05:21:08.040797+0000","last_became_peered":"2024-03-28T05:21:08.040797+0000","last_unstale":"2024-03-28T05:22:38.771646+0000","last_undegraded":"2024-03-28T05:22:38.771646+0000","last_fullsized":"2024-03-28T05:22:38.771646+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:05:03.387916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.277351+0000","last_change":"2024-03-28T05:21:12.293759+0000","last_active":"2024-03-28T05:22:11.277351+0000","last_peered":"2024-03-28T05:22:11.277351+0000","last_clean":"2024-03-28T05:22:11.277351+0000","last_became_active":"2024-03-28T05:21:12.293355+0000","last_became_peered":"2024-03-28T05:21:12.293355+0000","last_unstale":"2024-03-28T05:22:11.277351+0000","last_undegraded":"2024-03-28T05:22:11.277351+0000","last_fullsized":"2024-03-28T05:22:11.277351+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:29:52.424113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.514423+0000","last_change":"2024-03-28T05:21:07.267179+0000","last_active":"2024-03-28T05:22:10.514423+0000","last_peered":"2024-03-28T05:22:10.514423+0000","last_clean":"2024-03-28T05:22:10.514423+0000","last_became_active":"2024-03-28T05:21:07.266837+0000","last_became_peered":"2024-03-28T05:21:07.266837+0000","last_unstale":"2024-03-28T05:22:10.514423+0000","last_undegraded":"2024-03-28T05:22:10.514423+0000","last_fullsized":"2024-03-28T05:22:10.514423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:58:17.424542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.769698+0000","last_change":"2024-03-28T05:21:08.045789+0000","last_active":"2024-03-28T05:22:38.769698+0000","last_peered":"2024-03-28T05:22:38.769698+0000","last_clean":"2024-03-28T05:22:38.769698+0000","last_became_active":"2024-03-28T05:21:08.045441+0000","last_became_peered":"2024-03-28T05:21:08.045441+0000","last_unstale":"2024-03-28T05:22:38.769698+0000","last_undegraded":"2024-03-28T05:22:38.769698+0000","last_fullsized":"2024-03-28T05:22:38.769698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:15:49.968174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.511944+0000","last_change":"2024-03-28T05:21:12.290300+0000","last_active":"2024-03-28T05:22:10.511944+0000","last_peered":"2024-03-28T05:22:10.511944+0000","last_clean":"2024-03-28T05:22:10.511944+0000","last_became_active":"2024-03-28T05:21:12.289817+0000","last_became_peered":"2024-03-28T05:21:12.289817+0000","last_unstale":"2024-03-28T05:22:10.511944+0000","last_undegraded":"2024-03-28T05:22:10.511944+0000","last_fullsized":"2024-03-28T05:22:10.511944+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:53:50.705017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.516093+0000","last_change":"2024-03-28T05:21:07.261302+0000","last_active":"2024-03-28T05:22:10.516093+0000","last_peered":"2024-03-28T05:22:10.516093+0000","last_clean":"2024-03-28T05:22:10.516093+0000","last_became_active":"2024-03-28T05:21:07.260991+0000","last_became_peered":"2024-03-28T05:21:07.260991+0000","last_unstale":"2024-03-28T05:22:10.516093+0000","last_undegraded":"2024-03-28T05:22:10.516093+0000","last_fullsized":"2024-03-28T05:22:10.516093+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:57:58.411737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.768439+0000","last_change":"2024-03-28T05:21:08.039043+0000","last_active":"2024-03-28T05:22:38.768439+0000","last_peered":"2024-03-28T05:22:38.768439+0000","last_clean":"2024-03-28T05:22:38.768439+0000","last_became_active":"2024-03-28T05:21:08.038734+0000","last_became_peered":"2024-03-28T05:21:08.038734+0000","last_unstale":"2024-03-28T05:22:38.768439+0000","last_undegraded":"2024-03-28T05:22:38.768439+0000","last_fullsized":"2024-03-28T05:22:38.768439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:26:40.155570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.511263+0000","last_change":"2024-03-28T05:21:12.290037+0000","last_active":"2024-03-28T05:22:10.511263+0000","last_peered":"2024-03-28T05:22:10.511263+0000","last_clean":"2024-03-28T05:22:10.511263+0000","last_became_active":"2024-03-28T05:21:12.289526+0000","last_became_peered":"2024-03-28T05:21:12.289526+0000","last_unstale":"2024-03-28T05:22:10.511263+0000","last_undegraded":"2024-03-28T05:22:10.511263+0000","last_fullsized":"2024-03-28T05:22:10.511263+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:06:03.705716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.512912+0000","last_change":"2024-03-28T05:21:07.263945+0000","last_active":"2024-03-28T05:22:10.512912+0000","last_peered":"2024-03-28T05:22:10.512912+0000","last_clean":"2024-03-28T05:22:10.512912+0000","last_became_active":"2024-03-28T05:21:07.263648+0000","last_became_peered":"2024-03-28T05:21:07.263648+0000","last_unstale":"2024-03-28T05:22:10.512912+0000","last_undegraded":"2024-03-28T05:22:10.512912+0000","last_fullsized":"2024-03-28T05:22:10.512912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:37:28.797262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.751357+0000","last_change":"2024-03-28T05:21:08.041653+0000","last_active":"2024-03-28T05:22:38.751357+0000","last_peered":"2024-03-28T05:22:38.751357+0000","last_clean":"2024-03-28T05:22:38.751357+0000","last_became_active":"2024-03-28T05:21:08.041349+0000","last_became_peered":"2024-03-28T05:21:08.041349+0000","last_unstale":"2024-03-28T05:22:38.751357+0000","last_undegraded":"2024-03-28T05:22:38.751357+0000","last_fullsized":"2024-03-28T05:22:38.751357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:51:23.329132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.366467+0000","last_change":"2024-03-28T05:22:11.366467+0000","last_active":"2024-03-28T05:22:11.366467+0000","last_peered":"2024-03-28T05:22:11.366467+0000","last_clean":"2024-03-28T05:22:11.366467+0000","last_became_active":"2024-03-28T05:22:11.366031+0000","last_became_peered":"2024-03-28T05:22:11.366031+0000","last_unstale":"2024-03-28T05:22:11.366467+0000","last_undegraded":"2024-03-28T05:22:11.366467+0000","last_fullsized":"2024-03-28T05:22:11.366467+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:52:17.493615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"91'34","reported_seq":215,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:44.480998+0000","last_change":"2024-03-28T05:21:07.263460+0000","last_active":"2024-03-28T05:22:44.480998+0000","last_peered":"2024-03-28T05:22:44.480998+0000","last_clean":"2024-03-28T05:22:44.480998+0000","last_became_active":"2024-03-28T05:21:07.263118+0000","last_became_peered":"2024-03-28T05:21:07.263118+0000","last_unstale":"2024-03-28T05:22:44.480998+0000","last_undegraded":"2024-03-28T05:22:44.480998+0000","last_fullsized":"2024-03-28T05:22:44.480998+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:56:18.271666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63091,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":155,"num_read_kb":140,"num_write":34,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.749586+0000","last_change":"2024-03-28T05:21:08.041307+0000","last_active":"2024-03-28T05:22:38.749586+0000","last_peered":"2024-03-28T05:22:38.749586+0000","last_clean":"2024-03-28T05:22:38.749586+0000","last_became_active":"2024-03-28T05:21:08.040961+0000","last_became_peered":"2024-03-28T05:21:08.040961+0000","last_unstale":"2024-03-28T05:22:38.749586+0000","last_undegraded":"2024-03-28T05:22:38.749586+0000","last_fullsized":"2024-03-28T05:22:38.749586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:12:40.318091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.513550+0000","last_change":"2024-03-28T05:21:12.287222+0000","last_active":"2024-03-28T05:22:10.513550+0000","last_peered":"2024-03-28T05:22:10.513550+0000","last_clean":"2024-03-28T05:22:10.513550+0000","last_became_active":"2024-03-28T05:21:12.286744+0000","last_became_peered":"2024-03-28T05:21:12.286744+0000","last_unstale":"2024-03-28T05:22:10.513550+0000","last_undegraded":"2024-03-28T05:22:10.513550+0000","last_fullsized":"2024-03-28T05:22:10.513550+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:20:45.698054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.508819+0000","last_change":"2024-03-28T05:21:07.262932+0000","last_active":"2024-03-28T05:22:10.508819+0000","last_peered":"2024-03-28T05:22:10.508819+0000","last_clean":"2024-03-28T05:22:10.508819+0000","last_became_active":"2024-03-28T05:21:07.262497+0000","last_became_peered":"2024-03-28T05:21:07.262497+0000","last_unstale":"2024-03-28T05:22:10.508819+0000","last_undegraded":"2024-03-28T05:22:10.508819+0000","last_fullsized":"2024-03-28T05:22:10.508819+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:56:58.628442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.762747+0000","last_change":"2024-03-28T05:22:10.519929+0000","last_active":"2024-03-28T05:22:38.762747+0000","last_peered":"2024-03-28T05:22:38.762747+0000","last_clean":"2024-03-28T05:22:38.762747+0000","last_became_active":"2024-03-28T05:22:10.519458+0000","last_became_peered":"2024-03-28T05:22:10.519458+0000","last_unstale":"2024-03-28T05:22:38.762747+0000","last_undegraded":"2024-03-28T05:22:38.762747+0000","last_fullsized":"2024-03-28T05:22:38.762747+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:51:16.364386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.516570+0000","last_change":"2024-03-28T05:21:12.294339+0000","last_active":"2024-03-28T05:22:10.516570+0000","last_peered":"2024-03-28T05:22:10.516570+0000","last_clean":"2024-03-28T05:22:10.516570+0000","last_became_active":"2024-03-28T05:21:12.293939+0000","last_became_peered":"2024-03-28T05:21:12.293939+0000","last_unstale":"2024-03-28T05:22:10.516570+0000","last_undegraded":"2024-03-28T05:22:10.516570+0000","last_fullsized":"2024-03-28T05:22:10.516570+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:38:59.851650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.513125+0000","last_change":"2024-03-28T05:21:07.262414+0000","last_active":"2024-03-28T05:22:10.513125+0000","last_peered":"2024-03-28T05:22:10.513125+0000","last_clean":"2024-03-28T05:22:10.513125+0000","last_became_active":"2024-03-28T05:21:07.262121+0000","last_became_peered":"2024-03-28T05:21:07.262121+0000","last_unstale":"2024-03-28T05:22:10.513125+0000","last_undegraded":"2024-03-28T05:22:10.513125+0000","last_fullsized":"2024-03-28T05:22:10.513125+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:15:52.832449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.769149+0000","last_change":"2024-03-28T05:21:08.049263+0000","last_active":"2024-03-28T05:22:38.769149+0000","last_peered":"2024-03-28T05:22:38.769149+0000","last_clean":"2024-03-28T05:22:38.769149+0000","last_became_active":"2024-03-28T05:21:08.048912+0000","last_became_peered":"2024-03-28T05:21:08.048912+0000","last_unstale":"2024-03-28T05:22:38.769149+0000","last_undegraded":"2024-03-28T05:22:38.769149+0000","last_fullsized":"2024-03-28T05:22:38.769149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:36:06.562731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.513666+0000","last_change":"2024-03-28T05:21:12.290345+0000","last_active":"2024-03-28T05:22:10.513666+0000","last_peered":"2024-03-28T05:22:10.513666+0000","last_clean":"2024-03-28T05:22:10.513666+0000","last_became_active":"2024-03-28T05:21:12.289945+0000","last_became_peered":"2024-03-28T05:21:12.289945+0000","last_unstale":"2024-03-28T05:22:10.513666+0000","last_undegraded":"2024-03-28T05:22:10.513666+0000","last_fullsized":"2024-03-28T05:22:10.513666+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:41:43.024707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.511151+0000","last_change":"2024-03-28T05:21:07.262588+0000","last_active":"2024-03-28T05:22:10.511151+0000","last_peered":"2024-03-28T05:22:10.511151+0000","last_clean":"2024-03-28T05:22:10.511151+0000","last_became_active":"2024-03-28T05:21:07.262237+0000","last_became_peered":"2024-03-28T05:21:07.262237+0000","last_unstale":"2024-03-28T05:22:10.511151+0000","last_undegraded":"2024-03-28T05:22:10.511151+0000","last_fullsized":"2024-03-28T05:22:10.511151+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:49:08.207031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.767552+0000","last_change":"2024-03-28T05:21:08.041473+0000","last_active":"2024-03-28T05:22:38.767552+0000","last_peered":"2024-03-28T05:22:38.767552+0000","last_clean":"2024-03-28T05:22:38.767552+0000","last_became_active":"2024-03-28T05:21:08.041171+0000","last_became_peered":"2024-03-28T05:21:08.041171+0000","last_unstale":"2024-03-28T05:22:38.767552+0000","last_undegraded":"2024-03-28T05:22:38.767552+0000","last_fullsized":"2024-03-28T05:22:38.767552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:56:26.652129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.514194+0000","last_change":"2024-03-28T05:21:12.296288+0000","last_active":"2024-03-28T05:22:10.514194+0000","last_peered":"2024-03-28T05:22:10.514194+0000","last_clean":"2024-03-28T05:22:10.514194+0000","last_became_active":"2024-03-28T05:21:12.295896+0000","last_became_peered":"2024-03-28T05:21:12.295896+0000","last_unstale":"2024-03-28T05:22:10.514194+0000","last_undegraded":"2024-03-28T05:22:10.514194+0000","last_fullsized":"2024-03-28T05:22:10.514194+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:05:05.779622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.364912+0000","last_change":"2024-03-28T05:21:07.262448+0000","last_active":"2024-03-28T05:22:11.364912+0000","last_peered":"2024-03-28T05:22:11.364912+0000","last_clean":"2024-03-28T05:22:11.364912+0000","last_became_active":"2024-03-28T05:21:07.261966+0000","last_became_peered":"2024-03-28T05:21:07.261966+0000","last_unstale":"2024-03-28T05:22:11.364912+0000","last_undegraded":"2024-03-28T05:22:11.364912+0000","last_fullsized":"2024-03-28T05:22:11.364912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:45:37.904160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.769963+0000","last_change":"2024-03-28T05:21:08.045929+0000","last_active":"2024-03-28T05:22:38.769963+0000","last_peered":"2024-03-28T05:22:38.769963+0000","last_clean":"2024-03-28T05:22:38.769963+0000","last_became_active":"2024-03-28T05:21:08.045640+0000","last_became_peered":"2024-03-28T05:21:08.045640+0000","last_unstale":"2024-03-28T05:22:38.769963+0000","last_undegraded":"2024-03-28T05:22:38.769963+0000","last_fullsized":"2024-03-28T05:22:38.769963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:38:01.546332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.513279+0000","last_change":"2024-03-28T05:21:12.292516+0000","last_active":"2024-03-28T05:22:10.513279+0000","last_peered":"2024-03-28T05:22:10.513279+0000","last_clean":"2024-03-28T05:22:10.513279+0000","last_became_active":"2024-03-28T05:21:12.292129+0000","last_became_peered":"2024-03-28T05:21:12.292129+0000","last_unstale":"2024-03-28T05:22:10.513279+0000","last_undegraded":"2024-03-28T05:22:10.513279+0000","last_fullsized":"2024-03-28T05:22:10.513279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:32:25.086143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.364999+0000","last_change":"2024-03-28T05:21:07.267442+0000","last_active":"2024-03-28T05:22:11.364999+0000","last_peered":"2024-03-28T05:22:11.364999+0000","last_clean":"2024-03-28T05:22:11.364999+0000","last_became_active":"2024-03-28T05:21:07.267140+0000","last_became_peered":"2024-03-28T05:21:07.267140+0000","last_unstale":"2024-03-28T05:22:11.364999+0000","last_undegraded":"2024-03-28T05:22:11.364999+0000","last_fullsized":"2024-03-28T05:22:11.364999+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:43:05.543211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.763002+0000","last_change":"2024-03-28T05:21:08.040304+0000","last_active":"2024-03-28T05:22:38.763002+0000","last_peered":"2024-03-28T05:22:38.763002+0000","last_clean":"2024-03-28T05:22:38.763002+0000","last_became_active":"2024-03-28T05:21:08.039976+0000","last_became_peered":"2024-03-28T05:21:08.039976+0000","last_unstale":"2024-03-28T05:22:38.763002+0000","last_undegraded":"2024-03-28T05:22:38.763002+0000","last_fullsized":"2024-03-28T05:22:38.763002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:57:33.062202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.365355+0000","last_change":"2024-03-28T05:21:12.289177+0000","last_active":"2024-03-28T05:22:11.365355+0000","last_peered":"2024-03-28T05:22:11.365355+0000","last_clean":"2024-03-28T05:22:11.365355+0000","last_became_active":"2024-03-28T05:21:12.288615+0000","last_became_peered":"2024-03-28T05:21:12.288615+0000","last_unstale":"2024-03-28T05:22:11.365355+0000","last_undegraded":"2024-03-28T05:22:11.365355+0000","last_fullsized":"2024-03-28T05:22:11.365355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:27:51.819911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.513577+0000","last_change":"2024-03-28T05:21:07.272325+0000","last_active":"2024-03-28T05:22:10.513577+0000","last_peered":"2024-03-28T05:22:10.513577+0000","last_clean":"2024-03-28T05:22:10.513577+0000","last_became_active":"2024-03-28T05:21:07.271965+0000","last_became_peered":"2024-03-28T05:21:07.271965+0000","last_unstale":"2024-03-28T05:22:10.513577+0000","last_undegraded":"2024-03-28T05:22:10.513577+0000","last_fullsized":"2024-03-28T05:22:10.513577+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:20:24.723627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.772490+0000","last_change":"2024-03-28T05:21:08.047122+0000","last_active":"2024-03-28T05:22:38.772490+0000","last_peered":"2024-03-28T05:22:38.772490+0000","last_clean":"2024-03-28T05:22:38.772490+0000","last_became_active":"2024-03-28T05:21:08.046828+0000","last_became_peered":"2024-03-28T05:21:08.046828+0000","last_unstale":"2024-03-28T05:22:38.772490+0000","last_undegraded":"2024-03-28T05:22:38.772490+0000","last_fullsized":"2024-03-28T05:22:38.772490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:07:16.940600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.514274+0000","last_change":"2024-03-28T05:21:12.293325+0000","last_active":"2024-03-28T05:22:10.514274+0000","last_peered":"2024-03-28T05:22:10.514274+0000","last_clean":"2024-03-28T05:22:10.514274+0000","last_became_active":"2024-03-28T05:21:12.292876+0000","last_became_peered":"2024-03-28T05:21:12.292876+0000","last_unstale":"2024-03-28T05:22:10.514274+0000","last_undegraded":"2024-03-28T05:22:10.514274+0000","last_fullsized":"2024-03-28T05:22:10.514274+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:35:25.173848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.515530+0000","last_change":"2024-03-28T05:21:07.262435+0000","last_active":"2024-03-28T05:22:10.515530+0000","last_peered":"2024-03-28T05:22:10.515530+0000","last_clean":"2024-03-28T05:22:10.515530+0000","last_became_active":"2024-03-28T05:21:07.262019+0000","last_became_peered":"2024-03-28T05:21:07.262019+0000","last_unstale":"2024-03-28T05:22:10.515530+0000","last_undegraded":"2024-03-28T05:22:10.515530+0000","last_fullsized":"2024-03-28T05:22:10.515530+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:41:31.348300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.770285+0000","last_change":"2024-03-28T05:21:08.043614+0000","last_active":"2024-03-28T05:22:38.770285+0000","last_peered":"2024-03-28T05:22:38.770285+0000","last_clean":"2024-03-28T05:22:38.770285+0000","last_became_active":"2024-03-28T05:21:08.043187+0000","last_became_peered":"2024-03-28T05:21:08.043187+0000","last_unstale":"2024-03-28T05:22:38.770285+0000","last_undegraded":"2024-03-28T05:22:38.770285+0000","last_fullsized":"2024-03-28T05:22:38.770285+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:55:08.972707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.514419+0000","last_change":"2024-03-28T05:21:12.289381+0000","last_active":"2024-03-28T05:22:10.514419+0000","last_peered":"2024-03-28T05:22:10.514419+0000","last_clean":"2024-03-28T05:22:10.514419+0000","last_became_active":"2024-03-28T05:21:12.288987+0000","last_became_peered":"2024-03-28T05:21:12.288987+0000","last_unstale":"2024-03-28T05:22:10.514419+0000","last_undegraded":"2024-03-28T05:22:10.514419+0000","last_fullsized":"2024-03-28T05:22:10.514419+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:31:46.296664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.769526+0000","last_change":"2024-03-28T05:21:08.047077+0000","last_active":"2024-03-28T05:22:38.769526+0000","last_peered":"2024-03-28T05:22:38.769526+0000","last_clean":"2024-03-28T05:22:38.769526+0000","last_became_active":"2024-03-28T05:21:08.046783+0000","last_became_peered":"2024-03-28T05:21:08.046783+0000","last_unstale":"2024-03-28T05:22:38.769526+0000","last_undegraded":"2024-03-28T05:22:38.769526+0000","last_fullsized":"2024-03-28T05:22:38.769526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:52:07.823247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.513733+0000","last_change":"2024-03-28T05:21:07.271028+0000","last_active":"2024-03-28T05:22:10.513733+0000","last_peered":"2024-03-28T05:22:10.513733+0000","last_clean":"2024-03-28T05:22:10.513733+0000","last_became_active":"2024-03-28T05:21:07.270404+0000","last_became_peered":"2024-03-28T05:21:07.270404+0000","last_unstale":"2024-03-28T05:22:10.513733+0000","last_undegraded":"2024-03-28T05:22:10.513733+0000","last_fullsized":"2024-03-28T05:22:10.513733+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:01:23.875231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.372135+0000","last_change":"2024-03-28T05:22:11.372135+0000","last_active":"2024-03-28T05:22:11.372135+0000","last_peered":"2024-03-28T05:22:11.372135+0000","last_clean":"2024-03-28T05:22:11.372135+0000","last_became_active":"2024-03-28T05:22:11.368172+0000","last_became_peered":"2024-03-28T05:22:11.368172+0000","last_unstale":"2024-03-28T05:22:11.372135+0000","last_undegraded":"2024-03-28T05:22:11.372135+0000","last_fullsized":"2024-03-28T05:22:11.372135+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:01:05.027315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.762687+0000","last_change":"2024-03-28T05:21:08.044169+0000","last_active":"2024-03-28T05:22:38.762687+0000","last_peered":"2024-03-28T05:22:38.762687+0000","last_clean":"2024-03-28T05:22:38.762687+0000","last_became_active":"2024-03-28T05:21:08.043836+0000","last_became_peered":"2024-03-28T05:21:08.043836+0000","last_unstale":"2024-03-28T05:22:38.762687+0000","last_undegraded":"2024-03-28T05:22:38.762687+0000","last_fullsized":"2024-03-28T05:22:38.762687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:07:41.128111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.511230+0000","last_change":"2024-03-28T05:21:07.267368+0000","last_active":"2024-03-28T05:22:10.511230+0000","last_peered":"2024-03-28T05:22:10.511230+0000","last_clean":"2024-03-28T05:22:10.511230+0000","last_became_active":"2024-03-28T05:21:07.266995+0000","last_became_peered":"2024-03-28T05:21:07.266995+0000","last_unstale":"2024-03-28T05:22:10.511230+0000","last_undegraded":"2024-03-28T05:22:10.511230+0000","last_fullsized":"2024-03-28T05:22:10.511230+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:24:40.146950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":361,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.515336+0000","last_change":"2024-03-28T05:19:57.662704+0000","last_active":"2024-03-28T05:22:10.515336+0000","last_peered":"2024-03-28T05:22:10.515336+0000","last_clean":"2024-03-28T05:22:10.515336+0000","last_became_active":"2024-03-28T05:19:57.662332+0000","last_became_peered":"2024-03-28T05:19:57.662332+0000","last_unstale":"2024-03-28T05:22:10.515336+0000","last_undegraded":"2024-03-28T05:22:10.515336+0000","last_fullsized":"2024-03-28T05:22:10.515336+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:16:33.255097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:16:33.255097+0000","last_clean_scrub_stamp":"2024-03-28T05:16:33.255097+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:40:47.655643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.519139+0000","last_change":"2024-03-28T05:22:10.519139+0000","last_active":"2024-03-28T05:22:10.519139+0000","last_peered":"2024-03-28T05:22:10.519139+0000","last_clean":"2024-03-28T05:22:10.519139+0000","last_became_active":"2024-03-28T05:22:10.518661+0000","last_became_peered":"2024-03-28T05:22:10.518661+0000","last_unstale":"2024-03-28T05:22:10.519139+0000","last_undegraded":"2024-03-28T05:22:10.519139+0000","last_fullsized":"2024-03-28T05:22:10.519139+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:22:53.757709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.764075+0000","last_change":"2024-03-28T05:21:08.043036+0000","last_active":"2024-03-28T05:22:38.764075+0000","last_peered":"2024-03-28T05:22:38.764075+0000","last_clean":"2024-03-28T05:22:38.764075+0000","last_became_active":"2024-03-28T05:21:08.042467+0000","last_became_peered":"2024-03-28T05:21:08.042467+0000","last_unstale":"2024-03-28T05:22:38.764075+0000","last_undegraded":"2024-03-28T05:22:38.764075+0000","last_fullsized":"2024-03-28T05:22:38.764075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:16:41.718497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.515328+0000","last_change":"2024-03-28T05:21:07.260270+0000","last_active":"2024-03-28T05:22:10.515328+0000","last_peered":"2024-03-28T05:22:10.515328+0000","last_clean":"2024-03-28T05:22:10.515328+0000","last_became_active":"2024-03-28T05:21:07.259727+0000","last_became_peered":"2024-03-28T05:21:07.259727+0000","last_unstale":"2024-03-28T05:22:10.515328+0000","last_undegraded":"2024-03-28T05:22:10.515328+0000","last_fullsized":"2024-03-28T05:22:10.515328+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:19:15.883172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.516594+0000","last_change":"2024-03-28T05:21:12.294439+0000","last_active":"2024-03-28T05:22:10.516594+0000","last_peered":"2024-03-28T05:22:10.516594+0000","last_clean":"2024-03-28T05:22:10.516594+0000","last_became_active":"2024-03-28T05:21:12.294044+0000","last_became_peered":"2024-03-28T05:21:12.294044+0000","last_unstale":"2024-03-28T05:22:10.516594+0000","last_undegraded":"2024-03-28T05:22:10.516594+0000","last_fullsized":"2024-03-28T05:22:10.516594+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:22:51.218266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.365140+0000","last_change":"2024-03-28T05:21:07.272735+0000","last_active":"2024-03-28T05:22:11.365140+0000","last_peered":"2024-03-28T05:22:11.365140+0000","last_clean":"2024-03-28T05:22:11.365140+0000","last_became_active":"2024-03-28T05:21:07.272407+0000","last_became_peered":"2024-03-28T05:21:07.272407+0000","last_unstale":"2024-03-28T05:22:11.365140+0000","last_undegraded":"2024-03-28T05:22:11.365140+0000","last_fullsized":"2024-03-28T05:22:11.365140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:55:02.165183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.760296+0000","last_change":"2024-03-28T05:21:08.042577+0000","last_active":"2024-03-28T05:22:38.760296+0000","last_peered":"2024-03-28T05:22:38.760296+0000","last_clean":"2024-03-28T05:22:38.760296+0000","last_became_active":"2024-03-28T05:21:08.042248+0000","last_became_peered":"2024-03-28T05:21:08.042248+0000","last_unstale":"2024-03-28T05:22:38.760296+0000","last_undegraded":"2024-03-28T05:22:38.760296+0000","last_fullsized":"2024-03-28T05:22:38.760296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:31:15.856177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.277103+0000","last_change":"2024-03-28T05:21:12.294141+0000","last_active":"2024-03-28T05:22:11.277103+0000","last_peered":"2024-03-28T05:22:11.277103+0000","last_clean":"2024-03-28T05:22:11.277103+0000","last_became_active":"2024-03-28T05:21:12.293746+0000","last_became_peered":"2024-03-28T05:21:12.293746+0000","last_unstale":"2024-03-28T05:22:11.277103+0000","last_undegraded":"2024-03-28T05:22:11.277103+0000","last_fullsized":"2024-03-28T05:22:11.277103+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:06:18.871631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.514555+0000","last_change":"2024-03-28T05:21:07.275515+0000","last_active":"2024-03-28T05:22:10.514555+0000","last_peered":"2024-03-28T05:22:10.514555+0000","last_clean":"2024-03-28T05:22:10.514555+0000","last_became_active":"2024-03-28T05:21:07.275188+0000","last_became_peered":"2024-03-28T05:21:07.275188+0000","last_unstale":"2024-03-28T05:22:10.514555+0000","last_undegraded":"2024-03-28T05:22:10.514555+0000","last_fullsized":"2024-03-28T05:22:10.514555+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:34:29.351759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.752739+0000","last_change":"2024-03-28T05:21:08.051019+0000","last_active":"2024-03-28T05:22:38.752739+0000","last_peered":"2024-03-28T05:22:38.752739+0000","last_clean":"2024-03-28T05:22:38.752739+0000","last_became_active":"2024-03-28T05:21:08.050696+0000","last_became_peered":"2024-03-28T05:21:08.050696+0000","last_unstale":"2024-03-28T05:22:38.752739+0000","last_undegraded":"2024-03-28T05:22:38.752739+0000","last_fullsized":"2024-03-28T05:22:38.752739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:06:23.318153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.508602+0000","last_change":"2024-03-28T05:21:12.285980+0000","last_active":"2024-03-28T05:22:10.508602+0000","last_peered":"2024-03-28T05:22:10.508602+0000","last_clean":"2024-03-28T05:22:10.508602+0000","last_became_active":"2024-03-28T05:21:12.285578+0000","last_became_peered":"2024-03-28T05:21:12.285578+0000","last_unstale":"2024-03-28T05:22:10.508602+0000","last_undegraded":"2024-03-28T05:22:10.508602+0000","last_fullsized":"2024-03-28T05:22:10.508602+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:33:58.377753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.511800+0000","last_change":"2024-03-28T05:21:07.256475+0000","last_active":"2024-03-28T05:22:10.511800+0000","last_peered":"2024-03-28T05:22:10.511800+0000","last_clean":"2024-03-28T05:22:10.511800+0000","last_became_active":"2024-03-28T05:21:07.256125+0000","last_became_peered":"2024-03-28T05:21:07.256125+0000","last_unstale":"2024-03-28T05:22:10.511800+0000","last_undegraded":"2024-03-28T05:22:10.511800+0000","last_fullsized":"2024-03-28T05:22:10.511800+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:42.263704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.766697+0000","last_change":"2024-03-28T05:21:08.044417+0000","last_active":"2024-03-28T05:22:38.766697+0000","last_peered":"2024-03-28T05:22:38.766697+0000","last_clean":"2024-03-28T05:22:38.766697+0000","last_became_active":"2024-03-28T05:21:08.043931+0000","last_became_peered":"2024-03-28T05:21:08.043931+0000","last_unstale":"2024-03-28T05:22:38.766697+0000","last_undegraded":"2024-03-28T05:22:38.766697+0000","last_fullsized":"2024-03-28T05:22:38.766697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:43:55.013625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.511492+0000","last_change":"2024-03-28T05:21:12.300641+0000","last_active":"2024-03-28T05:22:10.511492+0000","last_peered":"2024-03-28T05:22:10.511492+0000","last_clean":"2024-03-28T05:22:10.511492+0000","last_became_active":"2024-03-28T05:21:12.300013+0000","last_became_peered":"2024-03-28T05:21:12.300013+0000","last_unstale":"2024-03-28T05:22:10.511492+0000","last_undegraded":"2024-03-28T05:22:10.511492+0000","last_fullsized":"2024-03-28T05:22:10.511492+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:38:42.474798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.798497+0000","last_change":"2024-03-28T05:21:07.261548+0000","last_active":"2024-03-28T05:22:10.798497+0000","last_peered":"2024-03-28T05:22:10.798497+0000","last_clean":"2024-03-28T05:22:10.798497+0000","last_became_active":"2024-03-28T05:21:07.261201+0000","last_became_peered":"2024-03-28T05:21:07.261201+0000","last_unstale":"2024-03-28T05:22:10.798497+0000","last_undegraded":"2024-03-28T05:22:10.798497+0000","last_fullsized":"2024-03-28T05:22:10.798497+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:35:48.899064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.723556+0000","last_change":"2024-03-28T05:21:08.045168+0000","last_active":"2024-03-28T05:22:38.723556+0000","last_peered":"2024-03-28T05:22:38.723556+0000","last_clean":"2024-03-28T05:22:38.723556+0000","last_became_active":"2024-03-28T05:21:08.044654+0000","last_became_peered":"2024-03-28T05:21:08.044654+0000","last_unstale":"2024-03-28T05:22:38.723556+0000","last_undegraded":"2024-03-28T05:22:38.723556+0000","last_fullsized":"2024-03-28T05:22:38.723556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:48:14.592798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.513786+0000","last_change":"2024-03-28T05:21:12.285593+0000","last_active":"2024-03-28T05:22:10.513786+0000","last_peered":"2024-03-28T05:22:10.513786+0000","last_clean":"2024-03-28T05:22:10.513786+0000","last_became_active":"2024-03-28T05:21:12.285202+0000","last_became_peered":"2024-03-28T05:21:12.285202+0000","last_unstale":"2024-03-28T05:22:10.513786+0000","last_undegraded":"2024-03-28T05:22:10.513786+0000","last_fullsized":"2024-03-28T05:22:10.513786+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:00:40.198255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.276436+0000","last_change":"2024-03-28T05:21:07.271352+0000","last_active":"2024-03-28T05:22:11.276436+0000","last_peered":"2024-03-28T05:22:11.276436+0000","last_clean":"2024-03-28T05:22:11.276436+0000","last_became_active":"2024-03-28T05:21:07.271038+0000","last_became_peered":"2024-03-28T05:21:07.271038+0000","last_unstale":"2024-03-28T05:22:11.276436+0000","last_undegraded":"2024-03-28T05:22:11.276436+0000","last_fullsized":"2024-03-28T05:22:11.276436+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:37:41.742550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.760926+0000","last_change":"2024-03-28T05:21:08.043503+0000","last_active":"2024-03-28T05:22:38.760926+0000","last_peered":"2024-03-28T05:22:38.760926+0000","last_clean":"2024-03-28T05:22:38.760926+0000","last_became_active":"2024-03-28T05:21:08.043173+0000","last_became_peered":"2024-03-28T05:21:08.043173+0000","last_unstale":"2024-03-28T05:22:38.760926+0000","last_undegraded":"2024-03-28T05:22:38.760926+0000","last_fullsized":"2024-03-28T05:22:38.760926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:23:30.576539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.511427+0000","last_change":"2024-03-28T05:21:12.300600+0000","last_active":"2024-03-28T05:22:10.511427+0000","last_peered":"2024-03-28T05:22:10.511427+0000","last_clean":"2024-03-28T05:22:10.511427+0000","last_became_active":"2024-03-28T05:21:12.299931+0000","last_became_peered":"2024-03-28T05:21:12.299931+0000","last_unstale":"2024-03-28T05:22:10.511427+0000","last_undegraded":"2024-03-28T05:22:10.511427+0000","last_fullsized":"2024-03-28T05:22:10.511427+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:29:22.405410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.508724+0000","last_change":"2024-03-28T05:21:07.265708+0000","last_active":"2024-03-28T05:22:10.508724+0000","last_peered":"2024-03-28T05:22:10.508724+0000","last_clean":"2024-03-28T05:22:10.508724+0000","last_became_active":"2024-03-28T05:21:07.265416+0000","last_became_peered":"2024-03-28T05:21:07.265416+0000","last_unstale":"2024-03-28T05:22:10.508724+0000","last_undegraded":"2024-03-28T05:22:10.508724+0000","last_fullsized":"2024-03-28T05:22:10.508724+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:14:05.003658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.772076+0000","last_change":"2024-03-28T05:21:08.051458+0000","last_active":"2024-03-28T05:22:38.772076+0000","last_peered":"2024-03-28T05:22:38.772076+0000","last_clean":"2024-03-28T05:22:38.772076+0000","last_became_active":"2024-03-28T05:21:08.050997+0000","last_became_peered":"2024-03-28T05:21:08.050997+0000","last_unstale":"2024-03-28T05:22:38.772076+0000","last_undegraded":"2024-03-28T05:22:38.772076+0000","last_fullsized":"2024-03-28T05:22:38.772076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:33:41.046841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.511465+0000","last_change":"2024-03-28T05:21:12.285977+0000","last_active":"2024-03-28T05:22:10.511465+0000","last_peered":"2024-03-28T05:22:10.511465+0000","last_clean":"2024-03-28T05:22:10.511465+0000","last_became_active":"2024-03-28T05:21:12.285439+0000","last_became_peered":"2024-03-28T05:21:12.285439+0000","last_unstale":"2024-03-28T05:22:10.511465+0000","last_undegraded":"2024-03-28T05:22:10.511465+0000","last_fullsized":"2024-03-28T05:22:10.511465+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:07:34.409001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.364930+0000","last_change":"2024-03-28T05:21:07.268187+0000","last_active":"2024-03-28T05:22:11.364930+0000","last_peered":"2024-03-28T05:22:11.364930+0000","last_clean":"2024-03-28T05:22:11.364930+0000","last_became_active":"2024-03-28T05:21:07.266791+0000","last_became_peered":"2024-03-28T05:21:07.266791+0000","last_unstale":"2024-03-28T05:22:11.364930+0000","last_undegraded":"2024-03-28T05:22:11.364930+0000","last_fullsized":"2024-03-28T05:22:11.364930+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:22:03.683043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.769329+0000","last_change":"2024-03-28T05:21:08.054067+0000","last_active":"2024-03-28T05:22:38.769329+0000","last_peered":"2024-03-28T05:22:38.769329+0000","last_clean":"2024-03-28T05:22:38.769329+0000","last_became_active":"2024-03-28T05:21:08.053759+0000","last_became_peered":"2024-03-28T05:21:08.053759+0000","last_unstale":"2024-03-28T05:22:38.769329+0000","last_undegraded":"2024-03-28T05:22:38.769329+0000","last_fullsized":"2024-03-28T05:22:38.769329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:47:39.329564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.798532+0000","last_change":"2024-03-28T05:21:12.294547+0000","last_active":"2024-03-28T05:22:10.798532+0000","last_peered":"2024-03-28T05:22:10.798532+0000","last_clean":"2024-03-28T05:22:10.798532+0000","last_became_active":"2024-03-28T05:21:12.294146+0000","last_became_peered":"2024-03-28T05:21:12.294146+0000","last_unstale":"2024-03-28T05:22:10.798532+0000","last_undegraded":"2024-03-28T05:22:10.798532+0000","last_fullsized":"2024-03-28T05:22:10.798532+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:24:28.939378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.509253+0000","last_change":"2024-03-28T05:21:07.260749+0000","last_active":"2024-03-28T05:22:10.509253+0000","last_peered":"2024-03-28T05:22:10.509253+0000","last_clean":"2024-03-28T05:22:10.509253+0000","last_became_active":"2024-03-28T05:21:07.260439+0000","last_became_peered":"2024-03-28T05:21:07.260439+0000","last_unstale":"2024-03-28T05:22:10.509253+0000","last_undegraded":"2024-03-28T05:22:10.509253+0000","last_fullsized":"2024-03-28T05:22:10.509253+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:24:04.257080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.771298+0000","last_change":"2024-03-28T05:21:08.051079+0000","last_active":"2024-03-28T05:22:38.771298+0000","last_peered":"2024-03-28T05:22:38.771298+0000","last_clean":"2024-03-28T05:22:38.771298+0000","last_became_active":"2024-03-28T05:21:08.050790+0000","last_became_peered":"2024-03-28T05:21:08.050790+0000","last_unstale":"2024-03-28T05:22:38.771298+0000","last_undegraded":"2024-03-28T05:22:38.771298+0000","last_fullsized":"2024-03-28T05:22:38.771298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:04:39.205068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.777624+0000","last_change":"2024-03-28T05:21:12.291964+0000","last_active":"2024-03-28T05:22:10.777624+0000","last_peered":"2024-03-28T05:22:10.777624+0000","last_clean":"2024-03-28T05:22:10.777624+0000","last_became_active":"2024-03-28T05:21:12.291553+0000","last_became_peered":"2024-03-28T05:21:12.291553+0000","last_unstale":"2024-03-28T05:22:10.777624+0000","last_undegraded":"2024-03-28T05:22:10.777624+0000","last_fullsized":"2024-03-28T05:22:10.777624+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:10:34.180033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.513418+0000","last_change":"2024-03-28T05:21:07.263204+0000","last_active":"2024-03-28T05:22:10.513418+0000","last_peered":"2024-03-28T05:22:10.513418+0000","last_clean":"2024-03-28T05:22:10.513418+0000","last_became_active":"2024-03-28T05:21:07.262919+0000","last_became_peered":"2024-03-28T05:21:07.262919+0000","last_unstale":"2024-03-28T05:22:10.513418+0000","last_undegraded":"2024-03-28T05:22:10.513418+0000","last_fullsized":"2024-03-28T05:22:10.513418+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:53:19.505448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.767552+0000","last_change":"2024-03-28T05:21:08.044499+0000","last_active":"2024-03-28T05:22:38.767552+0000","last_peered":"2024-03-28T05:22:38.767552+0000","last_clean":"2024-03-28T05:22:38.767552+0000","last_became_active":"2024-03-28T05:21:08.043371+0000","last_became_peered":"2024-03-28T05:21:08.043371+0000","last_unstale":"2024-03-28T05:22:38.767552+0000","last_undegraded":"2024-03-28T05:22:38.767552+0000","last_fullsized":"2024-03-28T05:22:38.767552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:44:35.901625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.512587+0000","last_change":"2024-03-28T05:21:12.282443+0000","last_active":"2024-03-28T05:22:10.512587+0000","last_peered":"2024-03-28T05:22:10.512587+0000","last_clean":"2024-03-28T05:22:10.512587+0000","last_became_active":"2024-03-28T05:21:12.282065+0000","last_became_peered":"2024-03-28T05:21:12.282065+0000","last_unstale":"2024-03-28T05:22:10.512587+0000","last_undegraded":"2024-03-28T05:22:10.512587+0000","last_fullsized":"2024-03-28T05:22:10.512587+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:26:55.366554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.514296+0000","last_change":"2024-03-28T05:21:07.266038+0000","last_active":"2024-03-28T05:22:10.514296+0000","last_peered":"2024-03-28T05:22:10.514296+0000","last_clean":"2024-03-28T05:22:10.514296+0000","last_became_active":"2024-03-28T05:21:07.265728+0000","last_became_peered":"2024-03-28T05:21:07.265728+0000","last_unstale":"2024-03-28T05:22:10.514296+0000","last_undegraded":"2024-03-28T05:22:10.514296+0000","last_fullsized":"2024-03-28T05:22:10.514296+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:06:05.539565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.769497+0000","last_change":"2024-03-28T05:21:08.047626+0000","last_active":"2024-03-28T05:22:38.769497+0000","last_peered":"2024-03-28T05:22:38.769497+0000","last_clean":"2024-03-28T05:22:38.769497+0000","last_became_active":"2024-03-28T05:21:08.047251+0000","last_became_peered":"2024-03-28T05:21:08.047251+0000","last_unstale":"2024-03-28T05:22:38.769497+0000","last_undegraded":"2024-03-28T05:22:38.769497+0000","last_fullsized":"2024-03-28T05:22:38.769497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:25:36.043024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.763577+0000","last_change":"2024-03-28T05:21:08.042216+0000","last_active":"2024-03-28T05:22:38.763577+0000","last_peered":"2024-03-28T05:22:38.763577+0000","last_clean":"2024-03-28T05:22:38.763577+0000","last_became_active":"2024-03-28T05:21:08.041830+0000","last_became_peered":"2024-03-28T05:21:08.041830+0000","last_unstale":"2024-03-28T05:22:38.763577+0000","last_undegraded":"2024-03-28T05:22:38.763577+0000","last_fullsized":"2024-03-28T05:22:38.763577+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:56:19.785472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.770350+0000","last_change":"2024-03-28T05:22:10.516542+0000","last_active":"2024-03-28T05:22:38.770350+0000","last_peered":"2024-03-28T05:22:38.770350+0000","last_clean":"2024-03-28T05:22:38.770350+0000","last_became_active":"2024-03-28T05:22:10.516129+0000","last_became_peered":"2024-03-28T05:22:10.516129+0000","last_unstale":"2024-03-28T05:22:38.770350+0000","last_undegraded":"2024-03-28T05:22:38.770350+0000","last_fullsized":"2024-03-28T05:22:38.770350+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:14:49.040098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.761258+0000","last_change":"2024-03-28T05:21:08.043471+0000","last_active":"2024-03-28T05:22:38.761258+0000","last_peered":"2024-03-28T05:22:38.761258+0000","last_clean":"2024-03-28T05:22:38.761258+0000","last_became_active":"2024-03-28T05:21:08.043144+0000","last_became_peered":"2024-03-28T05:21:08.043144+0000","last_unstale":"2024-03-28T05:22:38.761258+0000","last_undegraded":"2024-03-28T05:22:38.761258+0000","last_fullsized":"2024-03-28T05:22:38.761258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:18:40.580589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.765708+0000","last_change":"2024-03-28T05:21:08.050105+0000","last_active":"2024-03-28T05:22:38.765708+0000","last_peered":"2024-03-28T05:22:38.765708+0000","last_clean":"2024-03-28T05:22:38.765708+0000","last_became_active":"2024-03-28T05:21:08.047056+0000","last_became_peered":"2024-03-28T05:21:08.047056+0000","last_unstale":"2024-03-28T05:22:38.765708+0000","last_undegraded":"2024-03-28T05:22:38.765708+0000","last_fullsized":"2024-03-28T05:22:38.765708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:59:54.341517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.763511+0000","last_change":"2024-03-28T05:22:10.778600+0000","last_active":"2024-03-28T05:22:38.763511+0000","last_peered":"2024-03-28T05:22:38.763511+0000","last_clean":"2024-03-28T05:22:38.763511+0000","last_became_active":"2024-03-28T05:22:10.778212+0000","last_became_peered":"2024-03-28T05:22:10.778212+0000","last_unstale":"2024-03-28T05:22:38.763511+0000","last_undegraded":"2024-03-28T05:22:38.763511+0000","last_fullsized":"2024-03-28T05:22:38.763511+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:18:00.460013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.763287+0000","last_change":"2024-03-28T05:21:08.045349+0000","last_active":"2024-03-28T05:22:38.763287+0000","last_peered":"2024-03-28T05:22:38.763287+0000","last_clean":"2024-03-28T05:22:38.763287+0000","last_became_active":"2024-03-28T05:21:08.044477+0000","last_became_peered":"2024-03-28T05:21:08.044477+0000","last_unstale":"2024-03-28T05:22:38.763287+0000","last_undegraded":"2024-03-28T05:22:38.763287+0000","last_fullsized":"2024-03-28T05:22:38.763287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:38:34.732765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.763133+0000","last_change":"2024-03-28T05:21:08.044801+0000","last_active":"2024-03-28T05:22:38.763133+0000","last_peered":"2024-03-28T05:22:38.763133+0000","last_clean":"2024-03-28T05:22:38.763133+0000","last_became_active":"2024-03-28T05:21:08.042208+0000","last_became_peered":"2024-03-28T05:21:08.042208+0000","last_unstale":"2024-03-28T05:22:38.763133+0000","last_undegraded":"2024-03-28T05:22:38.763133+0000","last_fullsized":"2024-03-28T05:22:38.763133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:49:15.663953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.745244+0000","last_change":"2024-03-28T05:21:08.047843+0000","last_active":"2024-03-28T05:22:38.745244+0000","last_peered":"2024-03-28T05:22:38.745244+0000","last_clean":"2024-03-28T05:22:38.745244+0000","last_became_active":"2024-03-28T05:21:08.047466+0000","last_became_peered":"2024-03-28T05:21:08.047466+0000","last_unstale":"2024-03-28T05:22:38.745244+0000","last_undegraded":"2024-03-28T05:22:38.745244+0000","last_fullsized":"2024-03-28T05:22:38.745244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:25:03.424233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.761650+0000","last_change":"2024-03-28T05:21:08.040786+0000","last_active":"2024-03-28T05:22:38.761650+0000","last_peered":"2024-03-28T05:22:38.761650+0000","last_clean":"2024-03-28T05:22:38.761650+0000","last_became_active":"2024-03-28T05:21:08.040495+0000","last_became_peered":"2024-03-28T05:21:08.040495+0000","last_unstale":"2024-03-28T05:22:38.761650+0000","last_undegraded":"2024-03-28T05:22:38.761650+0000","last_fullsized":"2024-03-28T05:22:38.761650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:06:31.626221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.751685+0000","last_change":"2024-03-28T05:21:08.050245+0000","last_active":"2024-03-28T05:22:38.751685+0000","last_peered":"2024-03-28T05:22:38.751685+0000","last_clean":"2024-03-28T05:22:38.751685+0000","last_became_active":"2024-03-28T05:21:08.049932+0000","last_became_peered":"2024-03-28T05:21:08.049932+0000","last_unstale":"2024-03-28T05:22:38.751685+0000","last_undegraded":"2024-03-28T05:22:38.751685+0000","last_fullsized":"2024-03-28T05:22:38.751685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:59:03.545226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.770895+0000","last_change":"2024-03-28T05:21:08.049262+0000","last_active":"2024-03-28T05:22:38.770895+0000","last_peered":"2024-03-28T05:22:38.770895+0000","last_clean":"2024-03-28T05:22:38.770895+0000","last_became_active":"2024-03-28T05:21:08.048578+0000","last_became_peered":"2024-03-28T05:21:08.048578+0000","last_unstale":"2024-03-28T05:22:38.770895+0000","last_undegraded":"2024-03-28T05:22:38.770895+0000","last_fullsized":"2024-03-28T05:22:38.770895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:26:38.614430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.765317+0000","last_change":"2024-03-28T05:21:08.046612+0000","last_active":"2024-03-28T05:22:38.765317+0000","last_peered":"2024-03-28T05:22:38.765317+0000","last_clean":"2024-03-28T05:22:38.765317+0000","last_became_active":"2024-03-28T05:21:08.046312+0000","last_became_peered":"2024-03-28T05:21:08.046312+0000","last_unstale":"2024-03-28T05:22:38.765317+0000","last_undegraded":"2024-03-28T05:22:38.765317+0000","last_fullsized":"2024-03-28T05:22:38.765317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:20:08.379620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.748578+0000","last_change":"2024-03-28T05:21:08.045205+0000","last_active":"2024-03-28T05:22:38.748578+0000","last_peered":"2024-03-28T05:22:38.748578+0000","last_clean":"2024-03-28T05:22:38.748578+0000","last_became_active":"2024-03-28T05:21:08.044901+0000","last_became_peered":"2024-03-28T05:21:08.044901+0000","last_unstale":"2024-03-28T05:22:38.748578+0000","last_undegraded":"2024-03-28T05:22:38.748578+0000","last_fullsized":"2024-03-28T05:22:38.748578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:31:11.045766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63889,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":161,"num_read_kb":146,"num_write":85,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":191667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383711,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":500233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645349,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28672,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716480,"statfs":{"total":95995035648,"available":95965675520,"internally_reserved":0,"allocated":1695744,"data_stored":1149790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808875,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":504911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070511,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1032192,"data_stored":504933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332148,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1622016,"data_stored":1095279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495673,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":496137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1089536,"data_stored":559228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":496227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":559262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182472,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":496241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920836,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":500255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757310,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1622016,"data_stored":1086687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":63125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":63285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T05:22:46.055 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T05:22:46.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:46 smithi023 ceph-mon[31954]: pgmap v339: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:22:46.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:46 smithi023 ceph-mon[31954]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T05:22:46.299 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:22:46.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:46 smithi113 ceph-mon[37164]: pgmap v339: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:22:46.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:46 smithi113 ceph-mon[37164]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T05:22:46.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:46 smithi098 ceph-mon[36087]: pgmap v339: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:22:46.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:46 smithi098 ceph-mon[36087]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T05:22:47.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.886+0000 7f3393540700 1 -- 172.21.15.23:0/1784518689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f338c101490 msgr2=0x7f338c111280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:47.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.886+0000 7f3393540700 1 --2- 172.21.15.23:0/1784518689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f338c101490 0x7f338c111280 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f338400b0d0 tx=0x7f338400b3e0 comp rx=0 tx=0).stop 2024-03-28T05:22:47.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.886+0000 7f3393540700 1 -- 172.21.15.23:0/1784518689 shutdown_connections 2024-03-28T05:22:47.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.886+0000 7f3393540700 1 --2- 172.21.15.23:0/1784518689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f338c101490 0x7f338c111280 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:47.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.886+0000 7f3393540700 1 --2- 172.21.15.23:0/1784518689 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f338c103010 0x7f338c1033f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:47.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.886+0000 7f3393540700 1 --2- 172.21.15.23:0/1784518689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f338c100ad0 0x7f338c100f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:47.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.886+0000 7f3393540700 1 -- 172.21.15.23:0/1784518689 >> 172.21.15.23:0/1784518689 conn(0x7f338c0763f0 msgr2=0x7f338c076800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:47.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.886+0000 7f3393540700 1 -- 172.21.15.23:0/1784518689 shutdown_connections 2024-03-28T05:22:47.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.886+0000 7f3393540700 1 -- 172.21.15.23:0/1784518689 wait complete. 2024-03-28T05:22:47.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.886+0000 7f3393540700 1 Processor -- start 2024-03-28T05:22:47.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.886+0000 7f3393540700 1 -- start start 2024-03-28T05:22:47.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.887+0000 7f3393540700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f338c100ad0 0x7f338c1a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:47.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.887+0000 7f3393540700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f338c101490 0x7f338c1a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:47.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.887+0000 7f3393540700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f338c103010 0x7f338c19a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:47.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.887+0000 7f3393540700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338c1147c0 con 0x7f338c103010 2024-03-28T05:22:47.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.887+0000 7f3393540700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338c114640 con 0x7f338c100ad0 2024-03-28T05:22:47.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.887+0000 7f3393540700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338c114940 con 0x7f338c101490 2024-03-28T05:22:47.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.887+0000 7f3391add700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f338c103010 0x7f338c19a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:47.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.887+0000 7f3391add700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f338c103010 0x7f338c19a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43818/0 (socket says 172.21.15.23:43818) 2024-03-28T05:22:47.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.887+0000 7f3391add700 1 -- 172.21.15.23:0/926148211 learned_addr learned my addr 172.21.15.23:0/926148211 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:47.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.887+0000 7f3390adb700 1 --2- 172.21.15.23:0/926148211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f338c101490 0x7f338c1a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:47.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.887+0000 7f33912dc700 1 --2- 172.21.15.23:0/926148211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f338c100ad0 0x7f338c1a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:47.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.887+0000 7f3391add700 1 -- 172.21.15.23:0/926148211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f338c101490 msgr2=0x7f338c1a0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:47.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.887+0000 7f3391add700 1 --2- 172.21.15.23:0/926148211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f338c101490 0x7f338c1a0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:47.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.887+0000 7f3391add700 1 -- 172.21.15.23:0/926148211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f338c100ad0 msgr2=0x7f338c1a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:47.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.887+0000 7f3391add700 1 --2- 172.21.15.23:0/926148211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f338c100ad0 0x7f338c1a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:47.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.887+0000 7f3391add700 1 -- 172.21.15.23:0/926148211 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338c19b140 con 0x7f338c103010 2024-03-28T05:22:47.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.887+0000 7f3391add700 1 --2- 172.21.15.23:0/926148211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f338c103010 0x7f338c19a9e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3384004da0 tx=0x7f3384004dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:47.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.887+0000 7f337e7fc700 1 -- 172.21.15.23:0/926148211 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3384021020 con 0x7f338c103010 2024-03-28T05:22:47.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.888+0000 7f337e7fc700 1 -- 172.21.15.23:0/926148211 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3384007c40 con 0x7f338c103010 2024-03-28T05:22:47.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.888+0000 7f3393540700 1 -- 172.21.15.23:0/926148211 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f338c19b3d0 con 0x7f338c103010 2024-03-28T05:22:47.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.888+0000 7f337e7fc700 1 -- 172.21.15.23:0/926148211 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33840052a0 con 0x7f338c103010 2024-03-28T05:22:47.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.888+0000 7f3393540700 1 -- 172.21.15.23:0/926148211 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f338c1a76f0 con 0x7f338c103010 2024-03-28T05:22:47.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.889+0000 7f3393540700 1 -- 172.21.15.23:0/926148211 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f338c068e50 con 0x7f338c103010 2024-03-28T05:22:47.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.890+0000 7f337e7fc700 1 -- 172.21.15.23:0/926148211 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3384024020 con 0x7f338c103010 2024-03-28T05:22:47.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.890+0000 7f337e7fc700 1 --2- 172.21.15.23:0/926148211 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f33780a2910 0x7f33780a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:47.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.891+0000 7f33912dc700 1 --2- 172.21.15.23:0/926148211 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f33780a2910 0x7f33780a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:47.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.891+0000 7f337e7fc700 1 -- 172.21.15.23:0/926148211 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f338400f020 con 0x7f338c103010 2024-03-28T05:22:47.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.891+0000 7f33912dc700 1 --2- 172.21.15.23:0/926148211 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f33780a2910 0x7f33780a4dd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f338c19bb20 tx=0x7f338000b5b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:47.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:47.895+0000 7f337e7fc700 1 -- 172.21.15.23:0/926148211 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f338409cca0 con 0x7f338c103010 2024-03-28T05:22:48.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:48.097+0000 7f3393540700 1 -- 172.21.15.23:0/926148211 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f338c19bf10 con 0x7f33780a2910 2024-03-28T05:22:48.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:48.107+0000 7f337e7fc700 1 -- 172.21.15.23:0/926148211 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340482 (secure 0 0 0) 0x7f338c19bf10 con 0x7f33780a2910 2024-03-28T05:22:48.109 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:22:48.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:48.110+0000 7f3393540700 1 -- 172.21.15.23:0/926148211 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f33780a2910 msgr2=0x7f33780a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:48.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:48.110+0000 7f3393540700 1 --2- 172.21.15.23:0/926148211 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f33780a2910 0x7f33780a4dd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f338c19bb20 tx=0x7f338000b5b0 comp rx=0 tx=0).stop 2024-03-28T05:22:48.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:48.110+0000 7f3393540700 1 -- 172.21.15.23:0/926148211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f338c103010 msgr2=0x7f338c19a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:48.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:48.110+0000 7f3393540700 1 --2- 172.21.15.23:0/926148211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f338c103010 0x7f338c19a9e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3384004da0 tx=0x7f3384004dd0 comp rx=0 tx=0).stop 2024-03-28T05:22:48.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:48.110+0000 7f3393540700 1 -- 172.21.15.23:0/926148211 shutdown_connections 2024-03-28T05:22:48.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:48.110+0000 7f3393540700 1 --2- 172.21.15.23:0/926148211 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f33780a2910 0x7f33780a4dd0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:48.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:48.110+0000 7f3393540700 1 --2- 172.21.15.23:0/926148211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f338c103010 0x7f338c19a9e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:48.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:48.110+0000 7f3393540700 1 --2- 172.21.15.23:0/926148211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f338c100ad0 0x7f338c1a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:48.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:48.110+0000 7f3393540700 1 --2- 172.21.15.23:0/926148211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f338c101490 0x7f338c1a0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:48.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:48.110+0000 7f3393540700 1 -- 172.21.15.23:0/926148211 >> 172.21.15.23:0/926148211 conn(0x7f338c0763f0 msgr2=0x7f338c0feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:48.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:48.110+0000 7f3393540700 1 -- 172.21.15.23:0/926148211 shutdown_connections 2024-03-28T05:22:48.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:48.110+0000 7f3393540700 1 -- 172.21.15.23:0/926148211 wait complete. 2024-03-28T05:22:48.112 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-28T05:22:48.469 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:48 smithi023 ceph-mon[31954]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:22:48.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:48 smithi113 ceph-mon[37164]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:22:48.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:48 smithi098 ceph-mon[36087]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:22:48.993 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-03-28T05:22:46.745756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654491,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":297,"num_read_kb":265,"num_write":268,"num_write_kb":3034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1180853,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":712,"ondisk_log_size":712,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367264,"kb_used_data":13952,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574560,"statfs":{"total":1151940427776,"available":1151564349440,"internally_reserved":0,"allocated":14286848,"data_stored":7949183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":249,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":11,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003607"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.766601+0000","last_change":"2024-03-28T05:21:08.043720+0000","last_active":"2024-03-28T05:22:38.766601+0000","last_peered":"2024-03-28T05:22:38.766601+0000","last_clean":"2024-03-28T05:22:38.766601+0000","last_became_active":"2024-03-28T05:21:08.043250+0000","last_became_peered":"2024-03-28T05:21:08.043250+0000","last_unstale":"2024-03-28T05:22:38.766601+0000","last_undegraded":"2024-03-28T05:22:38.766601+0000","last_fullsized":"2024-03-28T05:22:38.766601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:29:10.144086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.772608+0000","last_change":"2024-03-28T05:21:08.053247+0000","last_active":"2024-03-28T05:22:38.772608+0000","last_peered":"2024-03-28T05:22:38.772608+0000","last_clean":"2024-03-28T05:22:38.772608+0000","last_became_active":"2024-03-28T05:21:08.052941+0000","last_became_peered":"2024-03-28T05:21:08.052941+0000","last_unstale":"2024-03-28T05:22:38.772608+0000","last_undegraded":"2024-03-28T05:22:38.772608+0000","last_fullsized":"2024-03-28T05:22:38.772608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:41:22.223858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.759932+0000","last_change":"2024-03-28T05:21:08.051938+0000","last_active":"2024-03-28T05:22:38.759932+0000","last_peered":"2024-03-28T05:22:38.759932+0000","last_clean":"2024-03-28T05:22:38.759932+0000","last_became_active":"2024-03-28T05:21:08.050366+0000","last_became_peered":"2024-03-28T05:21:08.050366+0000","last_unstale":"2024-03-28T05:22:38.759932+0000","last_undegraded":"2024-03-28T05:22:38.759932+0000","last_fullsized":"2024-03-28T05:22:38.759932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:27:29.325009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.747999+0000","last_change":"2024-03-28T05:21:08.043371+0000","last_active":"2024-03-28T05:22:38.747999+0000","last_peered":"2024-03-28T05:22:38.747999+0000","last_clean":"2024-03-28T05:22:38.747999+0000","last_became_active":"2024-03-28T05:21:08.043038+0000","last_became_peered":"2024-03-28T05:21:08.043038+0000","last_unstale":"2024-03-28T05:22:38.747999+0000","last_undegraded":"2024-03-28T05:22:38.747999+0000","last_fullsized":"2024-03-28T05:22:38.747999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:16:14.137248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.740392+0000","last_change":"2024-03-28T05:21:08.048678+0000","last_active":"2024-03-28T05:22:38.740392+0000","last_peered":"2024-03-28T05:22:38.740392+0000","last_clean":"2024-03-28T05:22:38.740392+0000","last_became_active":"2024-03-28T05:21:08.048272+0000","last_became_peered":"2024-03-28T05:21:08.048272+0000","last_unstale":"2024-03-28T05:22:38.740392+0000","last_undegraded":"2024-03-28T05:22:38.740392+0000","last_fullsized":"2024-03-28T05:22:38.740392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:34:29.029249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.772420+0000","last_change":"2024-03-28T05:21:08.041764+0000","last_active":"2024-03-28T05:22:38.772420+0000","last_peered":"2024-03-28T05:22:38.772420+0000","last_clean":"2024-03-28T05:22:38.772420+0000","last_became_active":"2024-03-28T05:21:08.041337+0000","last_became_peered":"2024-03-28T05:21:08.041337+0000","last_unstale":"2024-03-28T05:22:38.772420+0000","last_undegraded":"2024-03-28T05:22:38.772420+0000","last_fullsized":"2024-03-28T05:22:38.772420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:42:29.136829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.764626+0000","last_change":"2024-03-28T05:22:10.518391+0000","last_active":"2024-03-28T05:22:38.764626+0000","last_peered":"2024-03-28T05:22:38.764626+0000","last_clean":"2024-03-28T05:22:38.764626+0000","last_became_active":"2024-03-28T05:22:10.518052+0000","last_became_peered":"2024-03-28T05:22:10.518052+0000","last_unstale":"2024-03-28T05:22:38.764626+0000","last_undegraded":"2024-03-28T05:22:38.764626+0000","last_fullsized":"2024-03-28T05:22:38.764626+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:39:06.337034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.772436+0000","last_change":"2024-03-28T05:21:08.048605+0000","last_active":"2024-03-28T05:22:38.772436+0000","last_peered":"2024-03-28T05:22:38.772436+0000","last_clean":"2024-03-28T05:22:38.772436+0000","last_became_active":"2024-03-28T05:21:08.048279+0000","last_became_peered":"2024-03-28T05:21:08.048279+0000","last_unstale":"2024-03-28T05:22:38.772436+0000","last_undegraded":"2024-03-28T05:22:38.772436+0000","last_fullsized":"2024-03-28T05:22:38.772436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:34:00.935133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.769056+0000","last_change":"2024-03-28T05:21:08.051357+0000","last_active":"2024-03-28T05:22:38.769056+0000","last_peered":"2024-03-28T05:22:38.769056+0000","last_clean":"2024-03-28T05:22:38.769056+0000","last_became_active":"2024-03-28T05:21:08.051012+0000","last_became_peered":"2024-03-28T05:21:08.051012+0000","last_unstale":"2024-03-28T05:22:38.769056+0000","last_undegraded":"2024-03-28T05:22:38.769056+0000","last_fullsized":"2024-03-28T05:22:38.769056+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:43:42.090619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.769083+0000","last_change":"2024-03-28T05:21:08.038727+0000","last_active":"2024-03-28T05:22:38.769083+0000","last_peered":"2024-03-28T05:22:38.769083+0000","last_clean":"2024-03-28T05:22:38.769083+0000","last_became_active":"2024-03-28T05:21:08.038425+0000","last_became_peered":"2024-03-28T05:21:08.038425+0000","last_unstale":"2024-03-28T05:22:38.769083+0000","last_undegraded":"2024-03-28T05:22:38.769083+0000","last_fullsized":"2024-03-28T05:22:38.769083+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:06:53.786136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.764629+0000","last_change":"2024-03-28T05:21:08.054698+0000","last_active":"2024-03-28T05:22:38.764629+0000","last_peered":"2024-03-28T05:22:38.764629+0000","last_clean":"2024-03-28T05:22:38.764629+0000","last_became_active":"2024-03-28T05:21:08.054310+0000","last_became_peered":"2024-03-28T05:21:08.054310+0000","last_unstale":"2024-03-28T05:22:38.764629+0000","last_undegraded":"2024-03-28T05:22:38.764629+0000","last_fullsized":"2024-03-28T05:22:38.764629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:09:47.996381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.769179+0000","last_change":"2024-03-28T05:21:08.042826+0000","last_active":"2024-03-28T05:22:38.769179+0000","last_peered":"2024-03-28T05:22:38.769179+0000","last_clean":"2024-03-28T05:22:38.769179+0000","last_became_active":"2024-03-28T05:21:08.042487+0000","last_became_peered":"2024-03-28T05:21:08.042487+0000","last_unstale":"2024-03-28T05:22:38.769179+0000","last_undegraded":"2024-03-28T05:22:38.769179+0000","last_fullsized":"2024-03-28T05:22:38.769179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:47:06.611206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.768692+0000","last_change":"2024-03-28T05:21:08.030791+0000","last_active":"2024-03-28T05:22:38.768692+0000","last_peered":"2024-03-28T05:22:38.768692+0000","last_clean":"2024-03-28T05:22:38.768692+0000","last_became_active":"2024-03-28T05:21:08.030447+0000","last_became_peered":"2024-03-28T05:21:08.030447+0000","last_unstale":"2024-03-28T05:22:38.768692+0000","last_undegraded":"2024-03-28T05:22:38.768692+0000","last_fullsized":"2024-03-28T05:22:38.768692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:54:53.454037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.772664+0000","last_change":"2024-03-28T05:21:08.042477+0000","last_active":"2024-03-28T05:22:38.772664+0000","last_peered":"2024-03-28T05:22:38.772664+0000","last_clean":"2024-03-28T05:22:38.772664+0000","last_became_active":"2024-03-28T05:21:08.042164+0000","last_became_peered":"2024-03-28T05:21:08.042164+0000","last_unstale":"2024-03-28T05:22:38.772664+0000","last_undegraded":"2024-03-28T05:22:38.772664+0000","last_fullsized":"2024-03-28T05:22:38.772664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:44:25.566195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.765642+0000","last_change":"2024-03-28T05:21:08.050635+0000","last_active":"2024-03-28T05:22:38.765642+0000","last_peered":"2024-03-28T05:22:38.765642+0000","last_clean":"2024-03-28T05:22:38.765642+0000","last_became_active":"2024-03-28T05:21:08.050217+0000","last_became_peered":"2024-03-28T05:21:08.050217+0000","last_unstale":"2024-03-28T05:22:38.765642+0000","last_undegraded":"2024-03-28T05:22:38.765642+0000","last_fullsized":"2024-03-28T05:22:38.765642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:13:49.252755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.770510+0000","last_change":"2024-03-28T05:21:08.048021+0000","last_active":"2024-03-28T05:22:38.770510+0000","last_peered":"2024-03-28T05:22:38.770510+0000","last_clean":"2024-03-28T05:22:38.770510+0000","last_became_active":"2024-03-28T05:21:08.047381+0000","last_became_peered":"2024-03-28T05:21:08.047381+0000","last_unstale":"2024-03-28T05:22:38.770510+0000","last_undegraded":"2024-03-28T05:22:38.770510+0000","last_fullsized":"2024-03-28T05:22:38.770510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:42:39.718804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.766474+0000","last_change":"2024-03-28T05:21:08.044774+0000","last_active":"2024-03-28T05:22:38.766474+0000","last_peered":"2024-03-28T05:22:38.766474+0000","last_clean":"2024-03-28T05:22:38.766474+0000","last_became_active":"2024-03-28T05:21:08.044470+0000","last_became_peered":"2024-03-28T05:21:08.044470+0000","last_unstale":"2024-03-28T05:22:38.766474+0000","last_undegraded":"2024-03-28T05:22:38.766474+0000","last_fullsized":"2024-03-28T05:22:38.766474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:26:11.354288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.771590+0000","last_change":"2024-03-28T05:21:08.042871+0000","last_active":"2024-03-28T05:22:38.771590+0000","last_peered":"2024-03-28T05:22:38.771590+0000","last_clean":"2024-03-28T05:22:38.771590+0000","last_became_active":"2024-03-28T05:21:08.042506+0000","last_became_peered":"2024-03-28T05:21:08.042506+0000","last_unstale":"2024-03-28T05:22:38.771590+0000","last_undegraded":"2024-03-28T05:22:38.771590+0000","last_fullsized":"2024-03-28T05:22:38.771590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:34:50.424380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.751454+0000","last_change":"2024-03-28T05:21:08.045056+0000","last_active":"2024-03-28T05:22:38.751454+0000","last_peered":"2024-03-28T05:22:38.751454+0000","last_clean":"2024-03-28T05:22:38.751454+0000","last_became_active":"2024-03-28T05:21:08.044719+0000","last_became_peered":"2024-03-28T05:21:08.044719+0000","last_unstale":"2024-03-28T05:22:38.751454+0000","last_undegraded":"2024-03-28T05:22:38.751454+0000","last_fullsized":"2024-03-28T05:22:38.751454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:23:12.479829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.508781+0000","last_change":"2024-03-28T05:21:12.295239+0000","last_active":"2024-03-28T05:22:10.508781+0000","last_peered":"2024-03-28T05:22:10.508781+0000","last_clean":"2024-03-28T05:22:10.508781+0000","last_became_active":"2024-03-28T05:21:12.294843+0000","last_became_peered":"2024-03-28T05:21:12.294843+0000","last_unstale":"2024-03-28T05:22:10.508781+0000","last_undegraded":"2024-03-28T05:22:10.508781+0000","last_fullsized":"2024-03-28T05:22:10.508781+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:36:35.291983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.513954+0000","last_change":"2024-03-28T05:21:07.270833+0000","last_active":"2024-03-28T05:22:10.513954+0000","last_peered":"2024-03-28T05:22:10.513954+0000","last_clean":"2024-03-28T05:22:10.513954+0000","last_became_active":"2024-03-28T05:21:07.270288+0000","last_became_peered":"2024-03-28T05:21:07.270288+0000","last_unstale":"2024-03-28T05:22:10.513954+0000","last_undegraded":"2024-03-28T05:22:10.513954+0000","last_fullsized":"2024-03-28T05:22:10.513954+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:41:05.494851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.741663+0000","last_change":"2024-03-28T05:22:10.515698+0000","last_active":"2024-03-28T05:22:38.741663+0000","last_peered":"2024-03-28T05:22:38.741663+0000","last_clean":"2024-03-28T05:22:38.741663+0000","last_became_active":"2024-03-28T05:22:10.515275+0000","last_became_peered":"2024-03-28T05:22:10.515275+0000","last_unstale":"2024-03-28T05:22:38.741663+0000","last_undegraded":"2024-03-28T05:22:38.741663+0000","last_fullsized":"2024-03-28T05:22:38.741663+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:16:03.215899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.511056+0000","last_change":"2024-03-28T05:21:12.295599+0000","last_active":"2024-03-28T05:22:10.511056+0000","last_peered":"2024-03-28T05:22:10.511056+0000","last_clean":"2024-03-28T05:22:10.511056+0000","last_became_active":"2024-03-28T05:21:12.295161+0000","last_became_peered":"2024-03-28T05:21:12.295161+0000","last_unstale":"2024-03-28T05:22:10.511056+0000","last_undegraded":"2024-03-28T05:22:10.511056+0000","last_fullsized":"2024-03-28T05:22:10.511056+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:28:39.097018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.276535+0000","last_change":"2024-03-28T05:21:07.264532+0000","last_active":"2024-03-28T05:22:11.276535+0000","last_peered":"2024-03-28T05:22:11.276535+0000","last_clean":"2024-03-28T05:22:11.276535+0000","last_became_active":"2024-03-28T05:21:07.264121+0000","last_became_peered":"2024-03-28T05:21:07.264121+0000","last_unstale":"2024-03-28T05:22:11.276535+0000","last_undegraded":"2024-03-28T05:22:11.276535+0000","last_fullsized":"2024-03-28T05:22:11.276535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:40:01.927165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.771177+0000","last_change":"2024-03-28T05:21:08.045547+0000","last_active":"2024-03-28T05:22:38.771177+0000","last_peered":"2024-03-28T05:22:38.771177+0000","last_clean":"2024-03-28T05:22:38.771177+0000","last_became_active":"2024-03-28T05:21:08.045243+0000","last_became_peered":"2024-03-28T05:21:08.045243+0000","last_unstale":"2024-03-28T05:22:38.771177+0000","last_undegraded":"2024-03-28T05:22:38.771177+0000","last_fullsized":"2024-03-28T05:22:38.771177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:24:51.178938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.511119+0000","last_change":"2024-03-28T05:21:12.301461+0000","last_active":"2024-03-28T05:22:10.511119+0000","last_peered":"2024-03-28T05:22:10.511119+0000","last_clean":"2024-03-28T05:22:10.511119+0000","last_became_active":"2024-03-28T05:21:12.300765+0000","last_became_peered":"2024-03-28T05:21:12.300765+0000","last_unstale":"2024-03-28T05:22:10.511119+0000","last_undegraded":"2024-03-28T05:22:10.511119+0000","last_fullsized":"2024-03-28T05:22:10.511119+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:10:38.609978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.365258+0000","last_change":"2024-03-28T05:21:07.269209+0000","last_active":"2024-03-28T05:22:11.365258+0000","last_peered":"2024-03-28T05:22:11.365258+0000","last_clean":"2024-03-28T05:22:11.365258+0000","last_became_active":"2024-03-28T05:21:07.268068+0000","last_became_peered":"2024-03-28T05:21:07.268068+0000","last_unstale":"2024-03-28T05:22:11.365258+0000","last_undegraded":"2024-03-28T05:22:11.365258+0000","last_fullsized":"2024-03-28T05:22:11.365258+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:44:53.135439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.769265+0000","last_change":"2024-03-28T05:21:08.045233+0000","last_active":"2024-03-28T05:22:38.769265+0000","last_peered":"2024-03-28T05:22:38.769265+0000","last_clean":"2024-03-28T05:22:38.769265+0000","last_became_active":"2024-03-28T05:21:08.044878+0000","last_became_peered":"2024-03-28T05:21:08.044878+0000","last_unstale":"2024-03-28T05:22:38.769265+0000","last_undegraded":"2024-03-28T05:22:38.769265+0000","last_fullsized":"2024-03-28T05:22:38.769265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:15:50.830519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.511121+0000","last_change":"2024-03-28T05:21:12.296326+0000","last_active":"2024-03-28T05:22:10.511121+0000","last_peered":"2024-03-28T05:22:10.511121+0000","last_clean":"2024-03-28T05:22:10.511121+0000","last_became_active":"2024-03-28T05:21:12.295863+0000","last_became_peered":"2024-03-28T05:21:12.295863+0000","last_unstale":"2024-03-28T05:22:10.511121+0000","last_undegraded":"2024-03-28T05:22:10.511121+0000","last_fullsized":"2024-03-28T05:22:10.511121+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:26:49.340900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.365112+0000","last_change":"2024-03-28T05:21:07.263689+0000","last_active":"2024-03-28T05:22:11.365112+0000","last_peered":"2024-03-28T05:22:11.365112+0000","last_clean":"2024-03-28T05:22:11.365112+0000","last_became_active":"2024-03-28T05:21:07.263410+0000","last_became_peered":"2024-03-28T05:21:07.263410+0000","last_unstale":"2024-03-28T05:22:11.365112+0000","last_undegraded":"2024-03-28T05:22:11.365112+0000","last_fullsized":"2024-03-28T05:22:11.365112+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:51:03.017620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.764612+0000","last_change":"2024-03-28T05:22:11.367486+0000","last_active":"2024-03-28T05:22:38.764612+0000","last_peered":"2024-03-28T05:22:38.764612+0000","last_clean":"2024-03-28T05:22:38.764612+0000","last_became_active":"2024-03-28T05:22:11.367149+0000","last_became_peered":"2024-03-28T05:22:11.367149+0000","last_unstale":"2024-03-28T05:22:38.764612+0000","last_undegraded":"2024-03-28T05:22:38.764612+0000","last_fullsized":"2024-03-28T05:22:38.764612+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:24:37.312732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.508870+0000","last_change":"2024-03-28T05:21:12.284248+0000","last_active":"2024-03-28T05:22:10.508870+0000","last_peered":"2024-03-28T05:22:10.508870+0000","last_clean":"2024-03-28T05:22:10.508870+0000","last_became_active":"2024-03-28T05:21:12.283887+0000","last_became_peered":"2024-03-28T05:21:12.283887+0000","last_unstale":"2024-03-28T05:22:10.508870+0000","last_undegraded":"2024-03-28T05:22:10.508870+0000","last_fullsized":"2024-03-28T05:22:10.508870+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:50:50.871119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.512917+0000","last_change":"2024-03-28T05:21:07.266436+0000","last_active":"2024-03-28T05:22:10.512917+0000","last_peered":"2024-03-28T05:22:10.512917+0000","last_clean":"2024-03-28T05:22:10.512917+0000","last_became_active":"2024-03-28T05:21:07.266115+0000","last_became_peered":"2024-03-28T05:21:07.266115+0000","last_unstale":"2024-03-28T05:22:10.512917+0000","last_undegraded":"2024-03-28T05:22:10.512917+0000","last_fullsized":"2024-03-28T05:22:10.512917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:42:06.865456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.764385+0000","last_change":"2024-03-28T05:21:08.047500+0000","last_active":"2024-03-28T05:22:38.764385+0000","last_peered":"2024-03-28T05:22:38.764385+0000","last_clean":"2024-03-28T05:22:38.764385+0000","last_became_active":"2024-03-28T05:21:08.047210+0000","last_became_peered":"2024-03-28T05:21:08.047210+0000","last_unstale":"2024-03-28T05:22:38.764385+0000","last_undegraded":"2024-03-28T05:22:38.764385+0000","last_fullsized":"2024-03-28T05:22:38.764385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:31:02.393561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.367586+0000","last_change":"2024-03-28T05:22:11.367586+0000","last_active":"2024-03-28T05:22:11.367586+0000","last_peered":"2024-03-28T05:22:11.367586+0000","last_clean":"2024-03-28T05:22:11.367586+0000","last_became_active":"2024-03-28T05:22:11.367122+0000","last_became_peered":"2024-03-28T05:22:11.367122+0000","last_unstale":"2024-03-28T05:22:11.367586+0000","last_undegraded":"2024-03-28T05:22:11.367586+0000","last_fullsized":"2024-03-28T05:22:11.367586+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:59:56.177807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.514783+0000","last_change":"2024-03-28T05:21:07.265869+0000","last_active":"2024-03-28T05:22:10.514783+0000","last_peered":"2024-03-28T05:22:10.514783+0000","last_clean":"2024-03-28T05:22:10.514783+0000","last_became_active":"2024-03-28T05:21:07.265586+0000","last_became_peered":"2024-03-28T05:21:07.265586+0000","last_unstale":"2024-03-28T05:22:10.514783+0000","last_undegraded":"2024-03-28T05:22:10.514783+0000","last_fullsized":"2024-03-28T05:22:10.514783+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:17:39.328864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.766512+0000","last_change":"2024-03-28T05:21:08.046780+0000","last_active":"2024-03-28T05:22:38.766512+0000","last_peered":"2024-03-28T05:22:38.766512+0000","last_clean":"2024-03-28T05:22:38.766512+0000","last_became_active":"2024-03-28T05:21:08.046484+0000","last_became_peered":"2024-03-28T05:21:08.046484+0000","last_unstale":"2024-03-28T05:22:38.766512+0000","last_undegraded":"2024-03-28T05:22:38.766512+0000","last_fullsized":"2024-03-28T05:22:38.766512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:02:08.388581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.516843+0000","last_change":"2024-03-28T05:21:12.288723+0000","last_active":"2024-03-28T05:22:10.516843+0000","last_peered":"2024-03-28T05:22:10.516843+0000","last_clean":"2024-03-28T05:22:10.516843+0000","last_became_active":"2024-03-28T05:21:12.288310+0000","last_became_peered":"2024-03-28T05:21:12.288310+0000","last_unstale":"2024-03-28T05:22:10.516843+0000","last_undegraded":"2024-03-28T05:22:10.516843+0000","last_fullsized":"2024-03-28T05:22:10.516843+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:15:39.953732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.513832+0000","last_change":"2024-03-28T05:21:07.257433+0000","last_active":"2024-03-28T05:22:10.513832+0000","last_peered":"2024-03-28T05:22:10.513832+0000","last_clean":"2024-03-28T05:22:10.513832+0000","last_became_active":"2024-03-28T05:21:07.257099+0000","last_became_peered":"2024-03-28T05:21:07.257099+0000","last_unstale":"2024-03-28T05:22:10.513832+0000","last_undegraded":"2024-03-28T05:22:10.513832+0000","last_fullsized":"2024-03-28T05:22:10.513832+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:59:24.580079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.761171+0000","last_change":"2024-03-28T05:21:08.047963+0000","last_active":"2024-03-28T05:22:38.761171+0000","last_peered":"2024-03-28T05:22:38.761171+0000","last_clean":"2024-03-28T05:22:38.761171+0000","last_became_active":"2024-03-28T05:21:08.046403+0000","last_became_peered":"2024-03-28T05:21:08.046403+0000","last_unstale":"2024-03-28T05:22:38.761171+0000","last_undegraded":"2024-03-28T05:22:38.761171+0000","last_fullsized":"2024-03-28T05:22:38.761171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:58:45.280549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.777471+0000","last_change":"2024-03-28T05:21:12.284618+0000","last_active":"2024-03-28T05:22:10.777471+0000","last_peered":"2024-03-28T05:22:10.777471+0000","last_clean":"2024-03-28T05:22:10.777471+0000","last_became_active":"2024-03-28T05:21:12.284185+0000","last_became_peered":"2024-03-28T05:21:12.284185+0000","last_unstale":"2024-03-28T05:22:10.777471+0000","last_undegraded":"2024-03-28T05:22:10.777471+0000","last_fullsized":"2024-03-28T05:22:10.777471+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:15:48.715607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.512753+0000","last_change":"2024-03-28T05:21:07.265493+0000","last_active":"2024-03-28T05:22:10.512753+0000","last_peered":"2024-03-28T05:22:10.512753+0000","last_clean":"2024-03-28T05:22:10.512753+0000","last_became_active":"2024-03-28T05:21:07.265182+0000","last_became_peered":"2024-03-28T05:21:07.265182+0000","last_unstale":"2024-03-28T05:22:10.512753+0000","last_undegraded":"2024-03-28T05:22:10.512753+0000","last_fullsized":"2024-03-28T05:22:10.512753+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:55:00.151367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.771646+0000","last_change":"2024-03-28T05:21:08.041161+0000","last_active":"2024-03-28T05:22:38.771646+0000","last_peered":"2024-03-28T05:22:38.771646+0000","last_clean":"2024-03-28T05:22:38.771646+0000","last_became_active":"2024-03-28T05:21:08.040797+0000","last_became_peered":"2024-03-28T05:21:08.040797+0000","last_unstale":"2024-03-28T05:22:38.771646+0000","last_undegraded":"2024-03-28T05:22:38.771646+0000","last_fullsized":"2024-03-28T05:22:38.771646+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:05:03.387916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.277351+0000","last_change":"2024-03-28T05:21:12.293759+0000","last_active":"2024-03-28T05:22:11.277351+0000","last_peered":"2024-03-28T05:22:11.277351+0000","last_clean":"2024-03-28T05:22:11.277351+0000","last_became_active":"2024-03-28T05:21:12.293355+0000","last_became_peered":"2024-03-28T05:21:12.293355+0000","last_unstale":"2024-03-28T05:22:11.277351+0000","last_undegraded":"2024-03-28T05:22:11.277351+0000","last_fullsized":"2024-03-28T05:22:11.277351+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:29:52.424113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.514423+0000","last_change":"2024-03-28T05:21:07.267179+0000","last_active":"2024-03-28T05:22:10.514423+0000","last_peered":"2024-03-28T05:22:10.514423+0000","last_clean":"2024-03-28T05:22:10.514423+0000","last_became_active":"2024-03-28T05:21:07.266837+0000","last_became_peered":"2024-03-28T05:21:07.266837+0000","last_unstale":"2024-03-28T05:22:10.514423+0000","last_undegraded":"2024-03-28T05:22:10.514423+0000","last_fullsized":"2024-03-28T05:22:10.514423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:58:17.424542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.769698+0000","last_change":"2024-03-28T05:21:08.045789+0000","last_active":"2024-03-28T05:22:38.769698+0000","last_peered":"2024-03-28T05:22:38.769698+0000","last_clean":"2024-03-28T05:22:38.769698+0000","last_became_active":"2024-03-28T05:21:08.045441+0000","last_became_peered":"2024-03-28T05:21:08.045441+0000","last_unstale":"2024-03-28T05:22:38.769698+0000","last_undegraded":"2024-03-28T05:22:38.769698+0000","last_fullsized":"2024-03-28T05:22:38.769698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:15:49.968174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.511944+0000","last_change":"2024-03-28T05:21:12.290300+0000","last_active":"2024-03-28T05:22:10.511944+0000","last_peered":"2024-03-28T05:22:10.511944+0000","last_clean":"2024-03-28T05:22:10.511944+0000","last_became_active":"2024-03-28T05:21:12.289817+0000","last_became_peered":"2024-03-28T05:21:12.289817+0000","last_unstale":"2024-03-28T05:22:10.511944+0000","last_undegraded":"2024-03-28T05:22:10.511944+0000","last_fullsized":"2024-03-28T05:22:10.511944+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:53:50.705017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.516093+0000","last_change":"2024-03-28T05:21:07.261302+0000","last_active":"2024-03-28T05:22:10.516093+0000","last_peered":"2024-03-28T05:22:10.516093+0000","last_clean":"2024-03-28T05:22:10.516093+0000","last_became_active":"2024-03-28T05:21:07.260991+0000","last_became_peered":"2024-03-28T05:21:07.260991+0000","last_unstale":"2024-03-28T05:22:10.516093+0000","last_undegraded":"2024-03-28T05:22:10.516093+0000","last_fullsized":"2024-03-28T05:22:10.516093+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:57:58.411737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.768439+0000","last_change":"2024-03-28T05:21:08.039043+0000","last_active":"2024-03-28T05:22:38.768439+0000","last_peered":"2024-03-28T05:22:38.768439+0000","last_clean":"2024-03-28T05:22:38.768439+0000","last_became_active":"2024-03-28T05:21:08.038734+0000","last_became_peered":"2024-03-28T05:21:08.038734+0000","last_unstale":"2024-03-28T05:22:38.768439+0000","last_undegraded":"2024-03-28T05:22:38.768439+0000","last_fullsized":"2024-03-28T05:22:38.768439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:26:40.155570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.511263+0000","last_change":"2024-03-28T05:21:12.290037+0000","last_active":"2024-03-28T05:22:10.511263+0000","last_peered":"2024-03-28T05:22:10.511263+0000","last_clean":"2024-03-28T05:22:10.511263+0000","last_became_active":"2024-03-28T05:21:12.289526+0000","last_became_peered":"2024-03-28T05:21:12.289526+0000","last_unstale":"2024-03-28T05:22:10.511263+0000","last_undegraded":"2024-03-28T05:22:10.511263+0000","last_fullsized":"2024-03-28T05:22:10.511263+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:06:03.705716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.512912+0000","last_change":"2024-03-28T05:21:07.263945+0000","last_active":"2024-03-28T05:22:10.512912+0000","last_peered":"2024-03-28T05:22:10.512912+0000","last_clean":"2024-03-28T05:22:10.512912+0000","last_became_active":"2024-03-28T05:21:07.263648+0000","last_became_peered":"2024-03-28T05:21:07.263648+0000","last_unstale":"2024-03-28T05:22:10.512912+0000","last_undegraded":"2024-03-28T05:22:10.512912+0000","last_fullsized":"2024-03-28T05:22:10.512912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:37:28.797262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.751357+0000","last_change":"2024-03-28T05:21:08.041653+0000","last_active":"2024-03-28T05:22:38.751357+0000","last_peered":"2024-03-28T05:22:38.751357+0000","last_clean":"2024-03-28T05:22:38.751357+0000","last_became_active":"2024-03-28T05:21:08.041349+0000","last_became_peered":"2024-03-28T05:21:08.041349+0000","last_unstale":"2024-03-28T05:22:38.751357+0000","last_undegraded":"2024-03-28T05:22:38.751357+0000","last_fullsized":"2024-03-28T05:22:38.751357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:51:23.329132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.366467+0000","last_change":"2024-03-28T05:22:11.366467+0000","last_active":"2024-03-28T05:22:11.366467+0000","last_peered":"2024-03-28T05:22:11.366467+0000","last_clean":"2024-03-28T05:22:11.366467+0000","last_became_active":"2024-03-28T05:22:11.366031+0000","last_became_peered":"2024-03-28T05:22:11.366031+0000","last_unstale":"2024-03-28T05:22:11.366467+0000","last_undegraded":"2024-03-28T05:22:11.366467+0000","last_fullsized":"2024-03-28T05:22:11.366467+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:52:17.493615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"91'34","reported_seq":215,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:44.480998+0000","last_change":"2024-03-28T05:21:07.263460+0000","last_active":"2024-03-28T05:22:44.480998+0000","last_peered":"2024-03-28T05:22:44.480998+0000","last_clean":"2024-03-28T05:22:44.480998+0000","last_became_active":"2024-03-28T05:21:07.263118+0000","last_became_peered":"2024-03-28T05:21:07.263118+0000","last_unstale":"2024-03-28T05:22:44.480998+0000","last_undegraded":"2024-03-28T05:22:44.480998+0000","last_fullsized":"2024-03-28T05:22:44.480998+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:56:18.271666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63091,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":155,"num_read_kb":140,"num_write":34,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.749586+0000","last_change":"2024-03-28T05:21:08.041307+0000","last_active":"2024-03-28T05:22:38.749586+0000","last_peered":"2024-03-28T05:22:38.749586+0000","last_clean":"2024-03-28T05:22:38.749586+0000","last_became_active":"2024-03-28T05:21:08.040961+0000","last_became_peered":"2024-03-28T05:21:08.040961+0000","last_unstale":"2024-03-28T05:22:38.749586+0000","last_undegraded":"2024-03-28T05:22:38.749586+0000","last_fullsized":"2024-03-28T05:22:38.749586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:12:40.318091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.513550+0000","last_change":"2024-03-28T05:21:12.287222+0000","last_active":"2024-03-28T05:22:10.513550+0000","last_peered":"2024-03-28T05:22:10.513550+0000","last_clean":"2024-03-28T05:22:10.513550+0000","last_became_active":"2024-03-28T05:21:12.286744+0000","last_became_peered":"2024-03-28T05:21:12.286744+0000","last_unstale":"2024-03-28T05:22:10.513550+0000","last_undegraded":"2024-03-28T05:22:10.513550+0000","last_fullsized":"2024-03-28T05:22:10.513550+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:20:45.698054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.508819+0000","last_change":"2024-03-28T05:21:07.262932+0000","last_active":"2024-03-28T05:22:10.508819+0000","last_peered":"2024-03-28T05:22:10.508819+0000","last_clean":"2024-03-28T05:22:10.508819+0000","last_became_active":"2024-03-28T05:21:07.262497+0000","last_became_peered":"2024-03-28T05:21:07.262497+0000","last_unstale":"2024-03-28T05:22:10.508819+0000","last_undegraded":"2024-03-28T05:22:10.508819+0000","last_fullsized":"2024-03-28T05:22:10.508819+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:56:58.628442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.762747+0000","last_change":"2024-03-28T05:22:10.519929+0000","last_active":"2024-03-28T05:22:38.762747+0000","last_peered":"2024-03-28T05:22:38.762747+0000","last_clean":"2024-03-28T05:22:38.762747+0000","last_became_active":"2024-03-28T05:22:10.519458+0000","last_became_peered":"2024-03-28T05:22:10.519458+0000","last_unstale":"2024-03-28T05:22:38.762747+0000","last_undegraded":"2024-03-28T05:22:38.762747+0000","last_fullsized":"2024-03-28T05:22:38.762747+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:51:16.364386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.516570+0000","last_change":"2024-03-28T05:21:12.294339+0000","last_active":"2024-03-28T05:22:10.516570+0000","last_peered":"2024-03-28T05:22:10.516570+0000","last_clean":"2024-03-28T05:22:10.516570+0000","last_became_active":"2024-03-28T05:21:12.293939+0000","last_became_peered":"2024-03-28T05:21:12.293939+0000","last_unstale":"2024-03-28T05:22:10.516570+0000","last_undegraded":"2024-03-28T05:22:10.516570+0000","last_fullsized":"2024-03-28T05:22:10.516570+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:38:59.851650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.513125+0000","last_change":"2024-03-28T05:21:07.262414+0000","last_active":"2024-03-28T05:22:10.513125+0000","last_peered":"2024-03-28T05:22:10.513125+0000","last_clean":"2024-03-28T05:22:10.513125+0000","last_became_active":"2024-03-28T05:21:07.262121+0000","last_became_peered":"2024-03-28T05:21:07.262121+0000","last_unstale":"2024-03-28T05:22:10.513125+0000","last_undegraded":"2024-03-28T05:22:10.513125+0000","last_fullsized":"2024-03-28T05:22:10.513125+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:15:52.832449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.769149+0000","last_change":"2024-03-28T05:21:08.049263+0000","last_active":"2024-03-28T05:22:38.769149+0000","last_peered":"2024-03-28T05:22:38.769149+0000","last_clean":"2024-03-28T05:22:38.769149+0000","last_became_active":"2024-03-28T05:21:08.048912+0000","last_became_peered":"2024-03-28T05:21:08.048912+0000","last_unstale":"2024-03-28T05:22:38.769149+0000","last_undegraded":"2024-03-28T05:22:38.769149+0000","last_fullsized":"2024-03-28T05:22:38.769149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:36:06.562731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.513666+0000","last_change":"2024-03-28T05:21:12.290345+0000","last_active":"2024-03-28T05:22:10.513666+0000","last_peered":"2024-03-28T05:22:10.513666+0000","last_clean":"2024-03-28T05:22:10.513666+0000","last_became_active":"2024-03-28T05:21:12.289945+0000","last_became_peered":"2024-03-28T05:21:12.289945+0000","last_unstale":"2024-03-28T05:22:10.513666+0000","last_undegraded":"2024-03-28T05:22:10.513666+0000","last_fullsized":"2024-03-28T05:22:10.513666+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:41:43.024707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.511151+0000","last_change":"2024-03-28T05:21:07.262588+0000","last_active":"2024-03-28T05:22:10.511151+0000","last_peered":"2024-03-28T05:22:10.511151+0000","last_clean":"2024-03-28T05:22:10.511151+0000","last_became_active":"2024-03-28T05:21:07.262237+0000","last_became_peered":"2024-03-28T05:21:07.262237+0000","last_unstale":"2024-03-28T05:22:10.511151+0000","last_undegraded":"2024-03-28T05:22:10.511151+0000","last_fullsized":"2024-03-28T05:22:10.511151+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:49:08.207031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.767552+0000","last_change":"2024-03-28T05:21:08.041473+0000","last_active":"2024-03-28T05:22:38.767552+0000","last_peered":"2024-03-28T05:22:38.767552+0000","last_clean":"2024-03-28T05:22:38.767552+0000","last_became_active":"2024-03-28T05:21:08.041171+0000","last_became_peered":"2024-03-28T05:21:08.041171+0000","last_unstale":"2024-03-28T05:22:38.767552+0000","last_undegraded":"2024-03-28T05:22:38.767552+0000","last_fullsized":"2024-03-28T05:22:38.767552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:56:26.652129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.514194+0000","last_change":"2024-03-28T05:21:12.296288+0000","last_active":"2024-03-28T05:22:10.514194+0000","last_peered":"2024-03-28T05:22:10.514194+0000","last_clean":"2024-03-28T05:22:10.514194+0000","last_became_active":"2024-03-28T05:21:12.295896+0000","last_became_peered":"2024-03-28T05:21:12.295896+0000","last_unstale":"2024-03-28T05:22:10.514194+0000","last_undegraded":"2024-03-28T05:22:10.514194+0000","last_fullsized":"2024-03-28T05:22:10.514194+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:05:05.779622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.364912+0000","last_change":"2024-03-28T05:21:07.262448+0000","last_active":"2024-03-28T05:22:11.364912+0000","last_peered":"2024-03-28T05:22:11.364912+0000","last_clean":"2024-03-28T05:22:11.364912+0000","last_became_active":"2024-03-28T05:21:07.261966+0000","last_became_peered":"2024-03-28T05:21:07.261966+0000","last_unstale":"2024-03-28T05:22:11.364912+0000","last_undegraded":"2024-03-28T05:22:11.364912+0000","last_fullsized":"2024-03-28T05:22:11.364912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:45:37.904160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.769963+0000","last_change":"2024-03-28T05:21:08.045929+0000","last_active":"2024-03-28T05:22:38.769963+0000","last_peered":"2024-03-28T05:22:38.769963+0000","last_clean":"2024-03-28T05:22:38.769963+0000","last_became_active":"2024-03-28T05:21:08.045640+0000","last_became_peered":"2024-03-28T05:21:08.045640+0000","last_unstale":"2024-03-28T05:22:38.769963+0000","last_undegraded":"2024-03-28T05:22:38.769963+0000","last_fullsized":"2024-03-28T05:22:38.769963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:38:01.546332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.513279+0000","last_change":"2024-03-28T05:21:12.292516+0000","last_active":"2024-03-28T05:22:10.513279+0000","last_peered":"2024-03-28T05:22:10.513279+0000","last_clean":"2024-03-28T05:22:10.513279+0000","last_became_active":"2024-03-28T05:21:12.292129+0000","last_became_peered":"2024-03-28T05:21:12.292129+0000","last_unstale":"2024-03-28T05:22:10.513279+0000","last_undegraded":"2024-03-28T05:22:10.513279+0000","last_fullsized":"2024-03-28T05:22:10.513279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:32:25.086143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.364999+0000","last_change":"2024-03-28T05:21:07.267442+0000","last_active":"2024-03-28T05:22:11.364999+0000","last_peered":"2024-03-28T05:22:11.364999+0000","last_clean":"2024-03-28T05:22:11.364999+0000","last_became_active":"2024-03-28T05:21:07.267140+0000","last_became_peered":"2024-03-28T05:21:07.267140+0000","last_unstale":"2024-03-28T05:22:11.364999+0000","last_undegraded":"2024-03-28T05:22:11.364999+0000","last_fullsized":"2024-03-28T05:22:11.364999+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:43:05.543211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.763002+0000","last_change":"2024-03-28T05:21:08.040304+0000","last_active":"2024-03-28T05:22:38.763002+0000","last_peered":"2024-03-28T05:22:38.763002+0000","last_clean":"2024-03-28T05:22:38.763002+0000","last_became_active":"2024-03-28T05:21:08.039976+0000","last_became_peered":"2024-03-28T05:21:08.039976+0000","last_unstale":"2024-03-28T05:22:38.763002+0000","last_undegraded":"2024-03-28T05:22:38.763002+0000","last_fullsized":"2024-03-28T05:22:38.763002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:57:33.062202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.365355+0000","last_change":"2024-03-28T05:21:12.289177+0000","last_active":"2024-03-28T05:22:11.365355+0000","last_peered":"2024-03-28T05:22:11.365355+0000","last_clean":"2024-03-28T05:22:11.365355+0000","last_became_active":"2024-03-28T05:21:12.288615+0000","last_became_peered":"2024-03-28T05:21:12.288615+0000","last_unstale":"2024-03-28T05:22:11.365355+0000","last_undegraded":"2024-03-28T05:22:11.365355+0000","last_fullsized":"2024-03-28T05:22:11.365355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:27:51.819911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.513577+0000","last_change":"2024-03-28T05:21:07.272325+0000","last_active":"2024-03-28T05:22:10.513577+0000","last_peered":"2024-03-28T05:22:10.513577+0000","last_clean":"2024-03-28T05:22:10.513577+0000","last_became_active":"2024-03-28T05:21:07.271965+0000","last_became_peered":"2024-03-28T05:21:07.271965+0000","last_unstale":"2024-03-28T05:22:10.513577+0000","last_undegraded":"2024-03-28T05:22:10.513577+0000","last_fullsized":"2024-03-28T05:22:10.513577+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:20:24.723627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.772490+0000","last_change":"2024-03-28T05:21:08.047122+0000","last_active":"2024-03-28T05:22:38.772490+0000","last_peered":"2024-03-28T05:22:38.772490+0000","last_clean":"2024-03-28T05:22:38.772490+0000","last_became_active":"2024-03-28T05:21:08.046828+0000","last_became_peered":"2024-03-28T05:21:08.046828+0000","last_unstale":"2024-03-28T05:22:38.772490+0000","last_undegraded":"2024-03-28T05:22:38.772490+0000","last_fullsized":"2024-03-28T05:22:38.772490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:07:16.940600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.514274+0000","last_change":"2024-03-28T05:21:12.293325+0000","last_active":"2024-03-28T05:22:10.514274+0000","last_peered":"2024-03-28T05:22:10.514274+0000","last_clean":"2024-03-28T05:22:10.514274+0000","last_became_active":"2024-03-28T05:21:12.292876+0000","last_became_peered":"2024-03-28T05:21:12.292876+0000","last_unstale":"2024-03-28T05:22:10.514274+0000","last_undegraded":"2024-03-28T05:22:10.514274+0000","last_fullsized":"2024-03-28T05:22:10.514274+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:35:25.173848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.515530+0000","last_change":"2024-03-28T05:21:07.262435+0000","last_active":"2024-03-28T05:22:10.515530+0000","last_peered":"2024-03-28T05:22:10.515530+0000","last_clean":"2024-03-28T05:22:10.515530+0000","last_became_active":"2024-03-28T05:21:07.262019+0000","last_became_peered":"2024-03-28T05:21:07.262019+0000","last_unstale":"2024-03-28T05:22:10.515530+0000","last_undegraded":"2024-03-28T05:22:10.515530+0000","last_fullsized":"2024-03-28T05:22:10.515530+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:41:31.348300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.770285+0000","last_change":"2024-03-28T05:21:08.043614+0000","last_active":"2024-03-28T05:22:38.770285+0000","last_peered":"2024-03-28T05:22:38.770285+0000","last_clean":"2024-03-28T05:22:38.770285+0000","last_became_active":"2024-03-28T05:21:08.043187+0000","last_became_peered":"2024-03-28T05:21:08.043187+0000","last_unstale":"2024-03-28T05:22:38.770285+0000","last_undegraded":"2024-03-28T05:22:38.770285+0000","last_fullsized":"2024-03-28T05:22:38.770285+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:55:08.972707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.514419+0000","last_change":"2024-03-28T05:21:12.289381+0000","last_active":"2024-03-28T05:22:10.514419+0000","last_peered":"2024-03-28T05:22:10.514419+0000","last_clean":"2024-03-28T05:22:10.514419+0000","last_became_active":"2024-03-28T05:21:12.288987+0000","last_became_peered":"2024-03-28T05:21:12.288987+0000","last_unstale":"2024-03-28T05:22:10.514419+0000","last_undegraded":"2024-03-28T05:22:10.514419+0000","last_fullsized":"2024-03-28T05:22:10.514419+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:31:46.296664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.769526+0000","last_change":"2024-03-28T05:21:08.047077+0000","last_active":"2024-03-28T05:22:38.769526+0000","last_peered":"2024-03-28T05:22:38.769526+0000","last_clean":"2024-03-28T05:22:38.769526+0000","last_became_active":"2024-03-28T05:21:08.046783+0000","last_became_peered":"2024-03-28T05:21:08.046783+0000","last_unstale":"2024-03-28T05:22:38.769526+0000","last_undegraded":"2024-03-28T05:22:38.769526+0000","last_fullsized":"2024-03-28T05:22:38.769526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:52:07.823247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.513733+0000","last_change":"2024-03-28T05:21:07.271028+0000","last_active":"2024-03-28T05:22:10.513733+0000","last_peered":"2024-03-28T05:22:10.513733+0000","last_clean":"2024-03-28T05:22:10.513733+0000","last_became_active":"2024-03-28T05:21:07.270404+0000","last_became_peered":"2024-03-28T05:21:07.270404+0000","last_unstale":"2024-03-28T05:22:10.513733+0000","last_undegraded":"2024-03-28T05:22:10.513733+0000","last_fullsized":"2024-03-28T05:22:10.513733+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:01:23.875231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.372135+0000","last_change":"2024-03-28T05:22:11.372135+0000","last_active":"2024-03-28T05:22:11.372135+0000","last_peered":"2024-03-28T05:22:11.372135+0000","last_clean":"2024-03-28T05:22:11.372135+0000","last_became_active":"2024-03-28T05:22:11.368172+0000","last_became_peered":"2024-03-28T05:22:11.368172+0000","last_unstale":"2024-03-28T05:22:11.372135+0000","last_undegraded":"2024-03-28T05:22:11.372135+0000","last_fullsized":"2024-03-28T05:22:11.372135+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:01:05.027315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.762687+0000","last_change":"2024-03-28T05:21:08.044169+0000","last_active":"2024-03-28T05:22:38.762687+0000","last_peered":"2024-03-28T05:22:38.762687+0000","last_clean":"2024-03-28T05:22:38.762687+0000","last_became_active":"2024-03-28T05:21:08.043836+0000","last_became_peered":"2024-03-28T05:21:08.043836+0000","last_unstale":"2024-03-28T05:22:38.762687+0000","last_undegraded":"2024-03-28T05:22:38.762687+0000","last_fullsized":"2024-03-28T05:22:38.762687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:07:41.128111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.511230+0000","last_change":"2024-03-28T05:21:07.267368+0000","last_active":"2024-03-28T05:22:10.511230+0000","last_peered":"2024-03-28T05:22:10.511230+0000","last_clean":"2024-03-28T05:22:10.511230+0000","last_became_active":"2024-03-28T05:21:07.266995+0000","last_became_peered":"2024-03-28T05:21:07.266995+0000","last_unstale":"2024-03-28T05:22:10.511230+0000","last_undegraded":"2024-03-28T05:22:10.511230+0000","last_fullsized":"2024-03-28T05:22:10.511230+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:24:40.146950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":361,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.515336+0000","last_change":"2024-03-28T05:19:57.662704+0000","last_active":"2024-03-28T05:22:10.515336+0000","last_peered":"2024-03-28T05:22:10.515336+0000","last_clean":"2024-03-28T05:22:10.515336+0000","last_became_active":"2024-03-28T05:19:57.662332+0000","last_became_peered":"2024-03-28T05:19:57.662332+0000","last_unstale":"2024-03-28T05:22:10.515336+0000","last_undegraded":"2024-03-28T05:22:10.515336+0000","last_fullsized":"2024-03-28T05:22:10.515336+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:16:33.255097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:16:33.255097+0000","last_clean_scrub_stamp":"2024-03-28T05:16:33.255097+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:40:47.655643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.519139+0000","last_change":"2024-03-28T05:22:10.519139+0000","last_active":"2024-03-28T05:22:10.519139+0000","last_peered":"2024-03-28T05:22:10.519139+0000","last_clean":"2024-03-28T05:22:10.519139+0000","last_became_active":"2024-03-28T05:22:10.518661+0000","last_became_peered":"2024-03-28T05:22:10.518661+0000","last_unstale":"2024-03-28T05:22:10.519139+0000","last_undegraded":"2024-03-28T05:22:10.519139+0000","last_fullsized":"2024-03-28T05:22:10.519139+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:22:53.757709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.764075+0000","last_change":"2024-03-28T05:21:08.043036+0000","last_active":"2024-03-28T05:22:38.764075+0000","last_peered":"2024-03-28T05:22:38.764075+0000","last_clean":"2024-03-28T05:22:38.764075+0000","last_became_active":"2024-03-28T05:21:08.042467+0000","last_became_peered":"2024-03-28T05:21:08.042467+0000","last_unstale":"2024-03-28T05:22:38.764075+0000","last_undegraded":"2024-03-28T05:22:38.764075+0000","last_fullsized":"2024-03-28T05:22:38.764075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:16:41.718497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.515328+0000","last_change":"2024-03-28T05:21:07.260270+0000","last_active":"2024-03-28T05:22:10.515328+0000","last_peered":"2024-03-28T05:22:10.515328+0000","last_clean":"2024-03-28T05:22:10.515328+0000","last_became_active":"2024-03-28T05:21:07.259727+0000","last_became_peered":"2024-03-28T05:21:07.259727+0000","last_unstale":"2024-03-28T05:22:10.515328+0000","last_undegraded":"2024-03-28T05:22:10.515328+0000","last_fullsized":"2024-03-28T05:22:10.515328+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:19:15.883172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.516594+0000","last_change":"2024-03-28T05:21:12.294439+0000","last_active":"2024-03-28T05:22:10.516594+0000","last_peered":"2024-03-28T05:22:10.516594+0000","last_clean":"2024-03-28T05:22:10.516594+0000","last_became_active":"2024-03-28T05:21:12.294044+0000","last_became_peered":"2024-03-28T05:21:12.294044+0000","last_unstale":"2024-03-28T05:22:10.516594+0000","last_undegraded":"2024-03-28T05:22:10.516594+0000","last_fullsized":"2024-03-28T05:22:10.516594+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:22:51.218266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.365140+0000","last_change":"2024-03-28T05:21:07.272735+0000","last_active":"2024-03-28T05:22:11.365140+0000","last_peered":"2024-03-28T05:22:11.365140+0000","last_clean":"2024-03-28T05:22:11.365140+0000","last_became_active":"2024-03-28T05:21:07.272407+0000","last_became_peered":"2024-03-28T05:21:07.272407+0000","last_unstale":"2024-03-28T05:22:11.365140+0000","last_undegraded":"2024-03-28T05:22:11.365140+0000","last_fullsized":"2024-03-28T05:22:11.365140+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:55:02.165183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.760296+0000","last_change":"2024-03-28T05:21:08.042577+0000","last_active":"2024-03-28T05:22:38.760296+0000","last_peered":"2024-03-28T05:22:38.760296+0000","last_clean":"2024-03-28T05:22:38.760296+0000","last_became_active":"2024-03-28T05:21:08.042248+0000","last_became_peered":"2024-03-28T05:21:08.042248+0000","last_unstale":"2024-03-28T05:22:38.760296+0000","last_undegraded":"2024-03-28T05:22:38.760296+0000","last_fullsized":"2024-03-28T05:22:38.760296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:31:15.856177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.277103+0000","last_change":"2024-03-28T05:21:12.294141+0000","last_active":"2024-03-28T05:22:11.277103+0000","last_peered":"2024-03-28T05:22:11.277103+0000","last_clean":"2024-03-28T05:22:11.277103+0000","last_became_active":"2024-03-28T05:21:12.293746+0000","last_became_peered":"2024-03-28T05:21:12.293746+0000","last_unstale":"2024-03-28T05:22:11.277103+0000","last_undegraded":"2024-03-28T05:22:11.277103+0000","last_fullsized":"2024-03-28T05:22:11.277103+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:06:18.871631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.514555+0000","last_change":"2024-03-28T05:21:07.275515+0000","last_active":"2024-03-28T05:22:10.514555+0000","last_peered":"2024-03-28T05:22:10.514555+0000","last_clean":"2024-03-28T05:22:10.514555+0000","last_became_active":"2024-03-28T05:21:07.275188+0000","last_became_peered":"2024-03-28T05:21:07.275188+0000","last_unstale":"2024-03-28T05:22:10.514555+0000","last_undegraded":"2024-03-28T05:22:10.514555+0000","last_fullsized":"2024-03-28T05:22:10.514555+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:34:29.351759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.752739+0000","last_change":"2024-03-28T05:21:08.051019+0000","last_active":"2024-03-28T05:22:38.752739+0000","last_peered":"2024-03-28T05:22:38.752739+0000","last_clean":"2024-03-28T05:22:38.752739+0000","last_became_active":"2024-03-28T05:21:08.050696+0000","last_became_peered":"2024-03-28T05:21:08.050696+0000","last_unstale":"2024-03-28T05:22:38.752739+0000","last_undegraded":"2024-03-28T05:22:38.752739+0000","last_fullsized":"2024-03-28T05:22:38.752739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:06:23.318153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.508602+0000","last_change":"2024-03-28T05:21:12.285980+0000","last_active":"2024-03-28T05:22:10.508602+0000","last_peered":"2024-03-28T05:22:10.508602+0000","last_clean":"2024-03-28T05:22:10.508602+0000","last_became_active":"2024-03-28T05:21:12.285578+0000","last_became_peered":"2024-03-28T05:21:12.285578+0000","last_unstale":"2024-03-28T05:22:10.508602+0000","last_undegraded":"2024-03-28T05:22:10.508602+0000","last_fullsized":"2024-03-28T05:22:10.508602+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:33:58.377753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.511800+0000","last_change":"2024-03-28T05:21:07.256475+0000","last_active":"2024-03-28T05:22:10.511800+0000","last_peered":"2024-03-28T05:22:10.511800+0000","last_clean":"2024-03-28T05:22:10.511800+0000","last_became_active":"2024-03-28T05:21:07.256125+0000","last_became_peered":"2024-03-28T05:21:07.256125+0000","last_unstale":"2024-03-28T05:22:10.511800+0000","last_undegraded":"2024-03-28T05:22:10.511800+0000","last_fullsized":"2024-03-28T05:22:10.511800+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:38:42.263704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.766697+0000","last_change":"2024-03-28T05:21:08.044417+0000","last_active":"2024-03-28T05:22:38.766697+0000","last_peered":"2024-03-28T05:22:38.766697+0000","last_clean":"2024-03-28T05:22:38.766697+0000","last_became_active":"2024-03-28T05:21:08.043931+0000","last_became_peered":"2024-03-28T05:21:08.043931+0000","last_unstale":"2024-03-28T05:22:38.766697+0000","last_undegraded":"2024-03-28T05:22:38.766697+0000","last_fullsized":"2024-03-28T05:22:38.766697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:43:55.013625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.511492+0000","last_change":"2024-03-28T05:21:12.300641+0000","last_active":"2024-03-28T05:22:10.511492+0000","last_peered":"2024-03-28T05:22:10.511492+0000","last_clean":"2024-03-28T05:22:10.511492+0000","last_became_active":"2024-03-28T05:21:12.300013+0000","last_became_peered":"2024-03-28T05:21:12.300013+0000","last_unstale":"2024-03-28T05:22:10.511492+0000","last_undegraded":"2024-03-28T05:22:10.511492+0000","last_fullsized":"2024-03-28T05:22:10.511492+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:38:42.474798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.798497+0000","last_change":"2024-03-28T05:21:07.261548+0000","last_active":"2024-03-28T05:22:10.798497+0000","last_peered":"2024-03-28T05:22:10.798497+0000","last_clean":"2024-03-28T05:22:10.798497+0000","last_became_active":"2024-03-28T05:21:07.261201+0000","last_became_peered":"2024-03-28T05:21:07.261201+0000","last_unstale":"2024-03-28T05:22:10.798497+0000","last_undegraded":"2024-03-28T05:22:10.798497+0000","last_fullsized":"2024-03-28T05:22:10.798497+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:35:48.899064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.723556+0000","last_change":"2024-03-28T05:21:08.045168+0000","last_active":"2024-03-28T05:22:38.723556+0000","last_peered":"2024-03-28T05:22:38.723556+0000","last_clean":"2024-03-28T05:22:38.723556+0000","last_became_active":"2024-03-28T05:21:08.044654+0000","last_became_peered":"2024-03-28T05:21:08.044654+0000","last_unstale":"2024-03-28T05:22:38.723556+0000","last_undegraded":"2024-03-28T05:22:38.723556+0000","last_fullsized":"2024-03-28T05:22:38.723556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:48:14.592798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.513786+0000","last_change":"2024-03-28T05:21:12.285593+0000","last_active":"2024-03-28T05:22:10.513786+0000","last_peered":"2024-03-28T05:22:10.513786+0000","last_clean":"2024-03-28T05:22:10.513786+0000","last_became_active":"2024-03-28T05:21:12.285202+0000","last_became_peered":"2024-03-28T05:21:12.285202+0000","last_unstale":"2024-03-28T05:22:10.513786+0000","last_undegraded":"2024-03-28T05:22:10.513786+0000","last_fullsized":"2024-03-28T05:22:10.513786+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:00:40.198255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.276436+0000","last_change":"2024-03-28T05:21:07.271352+0000","last_active":"2024-03-28T05:22:11.276436+0000","last_peered":"2024-03-28T05:22:11.276436+0000","last_clean":"2024-03-28T05:22:11.276436+0000","last_became_active":"2024-03-28T05:21:07.271038+0000","last_became_peered":"2024-03-28T05:21:07.271038+0000","last_unstale":"2024-03-28T05:22:11.276436+0000","last_undegraded":"2024-03-28T05:22:11.276436+0000","last_fullsized":"2024-03-28T05:22:11.276436+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:37:41.742550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.760926+0000","last_change":"2024-03-28T05:21:08.043503+0000","last_active":"2024-03-28T05:22:38.760926+0000","last_peered":"2024-03-28T05:22:38.760926+0000","last_clean":"2024-03-28T05:22:38.760926+0000","last_became_active":"2024-03-28T05:21:08.043173+0000","last_became_peered":"2024-03-28T05:21:08.043173+0000","last_unstale":"2024-03-28T05:22:38.760926+0000","last_undegraded":"2024-03-28T05:22:38.760926+0000","last_fullsized":"2024-03-28T05:22:38.760926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:23:30.576539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.511427+0000","last_change":"2024-03-28T05:21:12.300600+0000","last_active":"2024-03-28T05:22:10.511427+0000","last_peered":"2024-03-28T05:22:10.511427+0000","last_clean":"2024-03-28T05:22:10.511427+0000","last_became_active":"2024-03-28T05:21:12.299931+0000","last_became_peered":"2024-03-28T05:21:12.299931+0000","last_unstale":"2024-03-28T05:22:10.511427+0000","last_undegraded":"2024-03-28T05:22:10.511427+0000","last_fullsized":"2024-03-28T05:22:10.511427+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:29:22.405410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.508724+0000","last_change":"2024-03-28T05:21:07.265708+0000","last_active":"2024-03-28T05:22:10.508724+0000","last_peered":"2024-03-28T05:22:10.508724+0000","last_clean":"2024-03-28T05:22:10.508724+0000","last_became_active":"2024-03-28T05:21:07.265416+0000","last_became_peered":"2024-03-28T05:21:07.265416+0000","last_unstale":"2024-03-28T05:22:10.508724+0000","last_undegraded":"2024-03-28T05:22:10.508724+0000","last_fullsized":"2024-03-28T05:22:10.508724+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:14:05.003658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.772076+0000","last_change":"2024-03-28T05:21:08.051458+0000","last_active":"2024-03-28T05:22:38.772076+0000","last_peered":"2024-03-28T05:22:38.772076+0000","last_clean":"2024-03-28T05:22:38.772076+0000","last_became_active":"2024-03-28T05:21:08.050997+0000","last_became_peered":"2024-03-28T05:21:08.050997+0000","last_unstale":"2024-03-28T05:22:38.772076+0000","last_undegraded":"2024-03-28T05:22:38.772076+0000","last_fullsized":"2024-03-28T05:22:38.772076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T06:33:41.046841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.511465+0000","last_change":"2024-03-28T05:21:12.285977+0000","last_active":"2024-03-28T05:22:10.511465+0000","last_peered":"2024-03-28T05:22:10.511465+0000","last_clean":"2024-03-28T05:22:10.511465+0000","last_became_active":"2024-03-28T05:21:12.285439+0000","last_became_peered":"2024-03-28T05:21:12.285439+0000","last_unstale":"2024-03-28T05:22:10.511465+0000","last_undegraded":"2024-03-28T05:22:10.511465+0000","last_fullsized":"2024-03-28T05:22:10.511465+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:07:34.409001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:11.364930+0000","last_change":"2024-03-28T05:21:07.268187+0000","last_active":"2024-03-28T05:22:11.364930+0000","last_peered":"2024-03-28T05:22:11.364930+0000","last_clean":"2024-03-28T05:22:11.364930+0000","last_became_active":"2024-03-28T05:21:07.266791+0000","last_became_peered":"2024-03-28T05:21:07.266791+0000","last_unstale":"2024-03-28T05:22:11.364930+0000","last_undegraded":"2024-03-28T05:22:11.364930+0000","last_fullsized":"2024-03-28T05:22:11.364930+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:22:03.683043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.769329+0000","last_change":"2024-03-28T05:21:08.054067+0000","last_active":"2024-03-28T05:22:38.769329+0000","last_peered":"2024-03-28T05:22:38.769329+0000","last_clean":"2024-03-28T05:22:38.769329+0000","last_became_active":"2024-03-28T05:21:08.053759+0000","last_became_peered":"2024-03-28T05:21:08.053759+0000","last_unstale":"2024-03-28T05:22:38.769329+0000","last_undegraded":"2024-03-28T05:22:38.769329+0000","last_fullsized":"2024-03-28T05:22:38.769329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:47:39.329564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.798532+0000","last_change":"2024-03-28T05:21:12.294547+0000","last_active":"2024-03-28T05:22:10.798532+0000","last_peered":"2024-03-28T05:22:10.798532+0000","last_clean":"2024-03-28T05:22:10.798532+0000","last_became_active":"2024-03-28T05:21:12.294146+0000","last_became_peered":"2024-03-28T05:21:12.294146+0000","last_unstale":"2024-03-28T05:22:10.798532+0000","last_undegraded":"2024-03-28T05:22:10.798532+0000","last_fullsized":"2024-03-28T05:22:10.798532+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:24:28.939378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.509253+0000","last_change":"2024-03-28T05:21:07.260749+0000","last_active":"2024-03-28T05:22:10.509253+0000","last_peered":"2024-03-28T05:22:10.509253+0000","last_clean":"2024-03-28T05:22:10.509253+0000","last_became_active":"2024-03-28T05:21:07.260439+0000","last_became_peered":"2024-03-28T05:21:07.260439+0000","last_unstale":"2024-03-28T05:22:10.509253+0000","last_undegraded":"2024-03-28T05:22:10.509253+0000","last_fullsized":"2024-03-28T05:22:10.509253+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:24:04.257080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.771298+0000","last_change":"2024-03-28T05:21:08.051079+0000","last_active":"2024-03-28T05:22:38.771298+0000","last_peered":"2024-03-28T05:22:38.771298+0000","last_clean":"2024-03-28T05:22:38.771298+0000","last_became_active":"2024-03-28T05:21:08.050790+0000","last_became_peered":"2024-03-28T05:21:08.050790+0000","last_unstale":"2024-03-28T05:22:38.771298+0000","last_undegraded":"2024-03-28T05:22:38.771298+0000","last_fullsized":"2024-03-28T05:22:38.771298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:04:39.205068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.777624+0000","last_change":"2024-03-28T05:21:12.291964+0000","last_active":"2024-03-28T05:22:10.777624+0000","last_peered":"2024-03-28T05:22:10.777624+0000","last_clean":"2024-03-28T05:22:10.777624+0000","last_became_active":"2024-03-28T05:21:12.291553+0000","last_became_peered":"2024-03-28T05:21:12.291553+0000","last_unstale":"2024-03-28T05:22:10.777624+0000","last_undegraded":"2024-03-28T05:22:10.777624+0000","last_fullsized":"2024-03-28T05:22:10.777624+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:10:34.180033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.513418+0000","last_change":"2024-03-28T05:21:07.263204+0000","last_active":"2024-03-28T05:22:10.513418+0000","last_peered":"2024-03-28T05:22:10.513418+0000","last_clean":"2024-03-28T05:22:10.513418+0000","last_became_active":"2024-03-28T05:21:07.262919+0000","last_became_peered":"2024-03-28T05:21:07.262919+0000","last_unstale":"2024-03-28T05:22:10.513418+0000","last_undegraded":"2024-03-28T05:22:10.513418+0000","last_fullsized":"2024-03-28T05:22:10.513418+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:53:19.505448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.767552+0000","last_change":"2024-03-28T05:21:08.044499+0000","last_active":"2024-03-28T05:22:38.767552+0000","last_peered":"2024-03-28T05:22:38.767552+0000","last_clean":"2024-03-28T05:22:38.767552+0000","last_became_active":"2024-03-28T05:21:08.043371+0000","last_became_peered":"2024-03-28T05:21:08.043371+0000","last_unstale":"2024-03-28T05:22:38.767552+0000","last_undegraded":"2024-03-28T05:22:38.767552+0000","last_fullsized":"2024-03-28T05:22:38.767552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:44:35.901625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.512587+0000","last_change":"2024-03-28T05:21:12.282443+0000","last_active":"2024-03-28T05:22:10.512587+0000","last_peered":"2024-03-28T05:22:10.512587+0000","last_clean":"2024-03-28T05:22:10.512587+0000","last_became_active":"2024-03-28T05:21:12.282065+0000","last_became_peered":"2024-03-28T05:21:12.282065+0000","last_unstale":"2024-03-28T05:22:10.512587+0000","last_undegraded":"2024-03-28T05:22:10.512587+0000","last_fullsized":"2024-03-28T05:22:10.512587+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:11.256337+0000","last_clean_scrub_stamp":"2024-03-28T05:21:11.256337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:26:55.366554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:10.514296+0000","last_change":"2024-03-28T05:21:07.266038+0000","last_active":"2024-03-28T05:22:10.514296+0000","last_peered":"2024-03-28T05:22:10.514296+0000","last_clean":"2024-03-28T05:22:10.514296+0000","last_became_active":"2024-03-28T05:21:07.265728+0000","last_became_peered":"2024-03-28T05:21:07.265728+0000","last_unstale":"2024-03-28T05:22:10.514296+0000","last_undegraded":"2024-03-28T05:22:10.514296+0000","last_fullsized":"2024-03-28T05:22:10.514296+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:06.233078+0000","last_clean_scrub_stamp":"2024-03-28T05:21:06.233078+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T09:06:05.539565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.769497+0000","last_change":"2024-03-28T05:21:08.047626+0000","last_active":"2024-03-28T05:22:38.769497+0000","last_peered":"2024-03-28T05:22:38.769497+0000","last_clean":"2024-03-28T05:22:38.769497+0000","last_became_active":"2024-03-28T05:21:08.047251+0000","last_became_peered":"2024-03-28T05:21:08.047251+0000","last_unstale":"2024-03-28T05:22:38.769497+0000","last_undegraded":"2024-03-28T05:22:38.769497+0000","last_fullsized":"2024-03-28T05:22:38.769497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:25:36.043024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.763577+0000","last_change":"2024-03-28T05:21:08.042216+0000","last_active":"2024-03-28T05:22:38.763577+0000","last_peered":"2024-03-28T05:22:38.763577+0000","last_clean":"2024-03-28T05:22:38.763577+0000","last_became_active":"2024-03-28T05:21:08.041830+0000","last_became_peered":"2024-03-28T05:21:08.041830+0000","last_unstale":"2024-03-28T05:22:38.763577+0000","last_undegraded":"2024-03-28T05:22:38.763577+0000","last_fullsized":"2024-03-28T05:22:38.763577+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:56:19.785472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.770350+0000","last_change":"2024-03-28T05:22:10.516542+0000","last_active":"2024-03-28T05:22:38.770350+0000","last_peered":"2024-03-28T05:22:38.770350+0000","last_clean":"2024-03-28T05:22:38.770350+0000","last_became_active":"2024-03-28T05:22:10.516129+0000","last_became_peered":"2024-03-28T05:22:10.516129+0000","last_unstale":"2024-03-28T05:22:38.770350+0000","last_undegraded":"2024-03-28T05:22:38.770350+0000","last_fullsized":"2024-03-28T05:22:38.770350+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:14:49.040098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.761258+0000","last_change":"2024-03-28T05:21:08.043471+0000","last_active":"2024-03-28T05:22:38.761258+0000","last_peered":"2024-03-28T05:22:38.761258+0000","last_clean":"2024-03-28T05:22:38.761258+0000","last_became_active":"2024-03-28T05:21:08.043144+0000","last_became_peered":"2024-03-28T05:21:08.043144+0000","last_unstale":"2024-03-28T05:22:38.761258+0000","last_undegraded":"2024-03-28T05:22:38.761258+0000","last_fullsized":"2024-03-28T05:22:38.761258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T08:18:40.580589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.765708+0000","last_change":"2024-03-28T05:21:08.050105+0000","last_active":"2024-03-28T05:22:38.765708+0000","last_peered":"2024-03-28T05:22:38.765708+0000","last_clean":"2024-03-28T05:22:38.765708+0000","last_became_active":"2024-03-28T05:21:08.047056+0000","last_became_peered":"2024-03-28T05:21:08.047056+0000","last_unstale":"2024-03-28T05:22:38.765708+0000","last_undegraded":"2024-03-28T05:22:38.765708+0000","last_fullsized":"2024-03-28T05:22:38.765708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T12:59:54.341517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.763511+0000","last_change":"2024-03-28T05:22:10.778600+0000","last_active":"2024-03-28T05:22:38.763511+0000","last_peered":"2024-03-28T05:22:38.763511+0000","last_clean":"2024-03-28T05:22:38.763511+0000","last_became_active":"2024-03-28T05:22:10.778212+0000","last_became_peered":"2024-03-28T05:22:10.778212+0000","last_unstale":"2024-03-28T05:22:38.763511+0000","last_undegraded":"2024-03-28T05:22:38.763511+0000","last_fullsized":"2024-03-28T05:22:38.763511+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:18:00.460013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.763287+0000","last_change":"2024-03-28T05:21:08.045349+0000","last_active":"2024-03-28T05:22:38.763287+0000","last_peered":"2024-03-28T05:22:38.763287+0000","last_clean":"2024-03-28T05:22:38.763287+0000","last_became_active":"2024-03-28T05:21:08.044477+0000","last_became_peered":"2024-03-28T05:21:08.044477+0000","last_unstale":"2024-03-28T05:22:38.763287+0000","last_undegraded":"2024-03-28T05:22:38.763287+0000","last_fullsized":"2024-03-28T05:22:38.763287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T07:38:34.732765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.763133+0000","last_change":"2024-03-28T05:21:08.044801+0000","last_active":"2024-03-28T05:22:38.763133+0000","last_peered":"2024-03-28T05:22:38.763133+0000","last_clean":"2024-03-28T05:22:38.763133+0000","last_became_active":"2024-03-28T05:21:08.042208+0000","last_became_peered":"2024-03-28T05:21:08.042208+0000","last_unstale":"2024-03-28T05:22:38.763133+0000","last_undegraded":"2024-03-28T05:22:38.763133+0000","last_fullsized":"2024-03-28T05:22:38.763133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:49:15.663953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.745244+0000","last_change":"2024-03-28T05:21:08.047843+0000","last_active":"2024-03-28T05:22:38.745244+0000","last_peered":"2024-03-28T05:22:38.745244+0000","last_clean":"2024-03-28T05:22:38.745244+0000","last_became_active":"2024-03-28T05:21:08.047466+0000","last_became_peered":"2024-03-28T05:21:08.047466+0000","last_unstale":"2024-03-28T05:22:38.745244+0000","last_undegraded":"2024-03-28T05:22:38.745244+0000","last_fullsized":"2024-03-28T05:22:38.745244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T05:25:03.424233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.761650+0000","last_change":"2024-03-28T05:21:08.040786+0000","last_active":"2024-03-28T05:22:38.761650+0000","last_peered":"2024-03-28T05:22:38.761650+0000","last_clean":"2024-03-28T05:22:38.761650+0000","last_became_active":"2024-03-28T05:21:08.040495+0000","last_became_peered":"2024-03-28T05:21:08.040495+0000","last_unstale":"2024-03-28T05:22:38.761650+0000","last_undegraded":"2024-03-28T05:22:38.761650+0000","last_fullsized":"2024-03-28T05:22:38.761650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T10:06:31.626221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.751685+0000","last_change":"2024-03-28T05:21:08.050245+0000","last_active":"2024-03-28T05:22:38.751685+0000","last_peered":"2024-03-28T05:22:38.751685+0000","last_clean":"2024-03-28T05:22:38.751685+0000","last_became_active":"2024-03-28T05:21:08.049932+0000","last_became_peered":"2024-03-28T05:21:08.049932+0000","last_unstale":"2024-03-28T05:22:38.751685+0000","last_undegraded":"2024-03-28T05:22:38.751685+0000","last_fullsized":"2024-03-28T05:22:38.751685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:59:03.545226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.770895+0000","last_change":"2024-03-28T05:21:08.049262+0000","last_active":"2024-03-28T05:22:38.770895+0000","last_peered":"2024-03-28T05:22:38.770895+0000","last_clean":"2024-03-28T05:22:38.770895+0000","last_became_active":"2024-03-28T05:21:08.048578+0000","last_became_peered":"2024-03-28T05:21:08.048578+0000","last_unstale":"2024-03-28T05:22:38.770895+0000","last_undegraded":"2024-03-28T05:22:38.770895+0000","last_fullsized":"2024-03-28T05:22:38.770895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T13:26:38.614430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.765317+0000","last_change":"2024-03-28T05:21:08.046612+0000","last_active":"2024-03-28T05:22:38.765317+0000","last_peered":"2024-03-28T05:22:38.765317+0000","last_clean":"2024-03-28T05:22:38.765317+0000","last_became_active":"2024-03-28T05:21:08.046312+0000","last_became_peered":"2024-03-28T05:21:08.046312+0000","last_unstale":"2024-03-28T05:22:38.765317+0000","last_undegraded":"2024-03-28T05:22:38.765317+0000","last_fullsized":"2024-03-28T05:22:38.765317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T14:20:08.379620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-28T05:22:38.748578+0000","last_change":"2024-03-28T05:21:08.045205+0000","last_active":"2024-03-28T05:22:38.748578+0000","last_peered":"2024-03-28T05:22:38.748578+0000","last_clean":"2024-03-28T05:22:38.748578+0000","last_became_active":"2024-03-28T05:21:08.044901+0000","last_became_peered":"2024-03-28T05:21:08.044901+0000","last_unstale":"2024-03-28T05:22:38.748578+0000","last_undegraded":"2024-03-28T05:22:38.748578+0000","last_fullsized":"2024-03-28T05:22:38.748578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T05:21:07.235230+0000","last_clean_scrub_stamp":"2024-03-28T05:21:07.235230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T11:31:11.045766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63889,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":161,"num_read_kb":146,"num_write":85,"num_write_kb":94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":191667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383712,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":500233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645349,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28672,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716480,"statfs":{"total":95995035648,"available":95965675520,"internally_reserved":0,"allocated":1695744,"data_stored":1149790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808875,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":504911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070511,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1032192,"data_stored":504933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332148,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1622016,"data_stored":1095279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495673,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":496137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1089536,"data_stored":559228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":496227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":559262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182472,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":496241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920836,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":500255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757310,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1622016,"data_stored":1086687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":63125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":63285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T05:22:48.999 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-28T05:22:48.999 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-28T05:22:48.999 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-28T05:22:48.999 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-28T05:22:49.247 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:22:49.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:49 smithi023 ceph-mon[31954]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T05:22:49.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:49 smithi113 ceph-mon[37164]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T05:22:49.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:49 smithi098 ceph-mon[36087]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T05:22:50.435 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:50 smithi023 ceph-mon[31954]: pgmap v341: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:22:50.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:50 smithi113 ceph-mon[37164]: pgmap v341: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:22:50.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:50 smithi098 ceph-mon[36087]: pgmap v341: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:22:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.978+0000 7fb8dcd04700 1 -- 172.21.15.23:0/1861670115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8d8100780 msgr2=0x7fb8d8100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.978+0000 7fb8dcd04700 1 --2- 172.21.15.23:0/1861670115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8d8100780 0x7fb8d8100c00 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8009990 tx=0x7fb8c8009ca0 comp rx=0 tx=0).stop 2024-03-28T05:22:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.978+0000 7fb8dcd04700 1 -- 172.21.15.23:0/1861670115 shutdown_connections 2024-03-28T05:22:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.978+0000 7fb8dcd04700 1 --2- 172.21.15.23:0/1861670115 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d8101140 0x7fb8d810f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.978+0000 7fb8dcd04700 1 --2- 172.21.15.23:0/1861670115 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8d8107200 0x7fb8d81075e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.978+0000 7fb8dcd04700 1 --2- 172.21.15.23:0/1861670115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8d8100780 0x7fb8d8100c00 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.978+0000 7fb8dcd04700 1 -- 172.21.15.23:0/1861670115 >> 172.21.15.23:0/1861670115 conn(0x7fb8d8077690 msgr2=0x7fb8d80feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:50.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.978+0000 7fb8dcd04700 1 -- 172.21.15.23:0/1861670115 shutdown_connections 2024-03-28T05:22:50.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.978+0000 7fb8dcd04700 1 -- 172.21.15.23:0/1861670115 wait complete. 2024-03-28T05:22:50.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.979+0000 7fb8dcd04700 1 Processor -- start 2024-03-28T05:22:50.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.979+0000 7fb8dcd04700 1 -- start start 2024-03-28T05:22:50.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.979+0000 7fb8dcd04700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d8100780 0x7fb8d8101960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:50.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.979+0000 7fb8dcd04700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8d8101140 0x7fb8d8101ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:50.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.979+0000 7fb8dcd04700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8d8107200 0x7fb8d81023e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:50.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.979+0000 7fb8dcd04700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d81125e0 con 0x7fb8d8107200 2024-03-28T05:22:50.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.979+0000 7fb8dcd04700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d8112460 con 0x7fb8d8100780 2024-03-28T05:22:50.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.979+0000 7fb8dcd04700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d8112760 con 0x7fb8d8101140 2024-03-28T05:22:50.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.979+0000 7fb8d6d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8d8107200 0x7fb8d81023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:50.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.979+0000 7fb8d5d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8d8101140 0x7fb8d8101ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:50.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.979+0000 7fb8d6d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8d8107200 0x7fb8d81023e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43840/0 (socket says 172.21.15.23:43840) 2024-03-28T05:22:50.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.979+0000 7fb8d6d9d700 1 -- 172.21.15.23:0/1045749483 learned_addr learned my addr 172.21.15.23:0/1045749483 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:50.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.980+0000 7fb8d659c700 1 --2- 172.21.15.23:0/1045749483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d8100780 0x7fb8d8101960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:50.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.980+0000 7fb8d5d9b700 1 -- 172.21.15.23:0/1045749483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d8100780 msgr2=0x7fb8d8101960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:50.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.980+0000 7fb8d5d9b700 1 --2- 172.21.15.23:0/1045749483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d8100780 0x7fb8d8101960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:50.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.980+0000 7fb8d5d9b700 1 -- 172.21.15.23:0/1045749483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8d8107200 msgr2=0x7fb8d81023e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:50.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.980+0000 7fb8d5d9b700 1 --2- 172.21.15.23:0/1045749483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8d8107200 0x7fb8d81023e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:50.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.980+0000 7fb8d5d9b700 1 -- 172.21.15.23:0/1045749483 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d81a6f40 con 0x7fb8d8101140 2024-03-28T05:22:50.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.980+0000 7fb8d6d9d700 1 --2- 172.21.15.23:0/1045749483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8d8107200 0x7fb8d81023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:22:50.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.980+0000 7fb8d659c700 1 --2- 172.21.15.23:0/1045749483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d8100780 0x7fb8d8101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:22:50.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.980+0000 7fb8d5d9b700 1 --2- 172.21.15.23:0/1045749483 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8d8101140 0x7fb8d8101ea0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb8c80059d0 tx=0x7fb8c800bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:50.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.980+0000 7fb8c77fe700 1 -- 172.21.15.23:0/1045749483 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c8027070 con 0x7fb8d8101140 2024-03-28T05:22:50.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.980+0000 7fb8dcd04700 1 -- 172.21.15.23:0/1045749483 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8d81a71d0 con 0x7fb8d8101140 2024-03-28T05:22:50.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.980+0000 7fb8c77fe700 1 -- 172.21.15.23:0/1045749483 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb8c8003d70 con 0x7fb8d8101140 2024-03-28T05:22:50.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.980+0000 7fb8c77fe700 1 -- 172.21.15.23:0/1045749483 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c800f460 con 0x7fb8d8101140 2024-03-28T05:22:50.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.980+0000 7fb8dcd04700 1 -- 172.21.15.23:0/1045749483 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8d81a7680 con 0x7fb8d8101140 2024-03-28T05:22:50.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.981+0000 7fb8dcd04700 1 -- 172.21.15.23:0/1045749483 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8d810a760 con 0x7fb8d8101140 2024-03-28T05:22:50.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.982+0000 7fb8c77fe700 1 -- 172.21.15.23:0/1045749483 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb8c802a070 con 0x7fb8d8101140 2024-03-28T05:22:50.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.983+0000 7fb8c77fe700 1 --2- 172.21.15.23:0/1045749483 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb8c00a2890 0x7fb8c00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:50.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.983+0000 7fb8d659c700 1 --2- 172.21.15.23:0/1045749483 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb8c00a2890 0x7fb8c00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:50.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.983+0000 7fb8c77fe700 1 -- 172.21.15.23:0/1045749483 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fb8c80cff00 con 0x7fb8d8101140 2024-03-28T05:22:50.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.983+0000 7fb8d659c700 1 --2- 172.21.15.23:0/1045749483 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb8c00a2890 0x7fb8c00a4d50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc009c90 tx=0x7fb8cc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:50.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:50.986+0000 7fb8c77fe700 1 -- 172.21.15.23:0/1045749483 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb8c8013070 con 0x7fb8d8101140 2024-03-28T05:22:51.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.239+0000 7fb8dcd04700 1 -- 172.21.15.23:0/1045749483 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb8d8102e90 con 0x7fb8d8101140 2024-03-28T05:22:51.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.240+0000 7fb8c77fe700 1 -- 172.21.15.23:0/1045749483 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fb8c809cc70 con 0x7fb8d8101140 2024-03-28T05:22:51.238 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:22:51.239 INFO:teuthology.orchestra.run.smithi023.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-28T05:22:51.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.242+0000 7fb8dcd04700 1 -- 172.21.15.23:0/1045749483 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb8c00a2890 msgr2=0x7fb8c00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:51.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.242+0000 7fb8dcd04700 1 --2- 172.21.15.23:0/1045749483 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb8c00a2890 0x7fb8c00a4d50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc009c90 tx=0x7fb8cc009260 comp rx=0 tx=0).stop 2024-03-28T05:22:51.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.242+0000 7fb8dcd04700 1 -- 172.21.15.23:0/1045749483 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8d8101140 msgr2=0x7fb8d8101ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:51.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.242+0000 7fb8dcd04700 1 --2- 172.21.15.23:0/1045749483 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8d8101140 0x7fb8d8101ea0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb8c80059d0 tx=0x7fb8c800bab0 comp rx=0 tx=0).stop 2024-03-28T05:22:51.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.242+0000 7fb8dcd04700 1 -- 172.21.15.23:0/1045749483 shutdown_connections 2024-03-28T05:22:51.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.242+0000 7fb8dcd04700 1 --2- 172.21.15.23:0/1045749483 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb8c00a2890 0x7fb8c00a4d50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:51.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.242+0000 7fb8dcd04700 1 --2- 172.21.15.23:0/1045749483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8d8107200 0x7fb8d81023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:51.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.242+0000 7fb8dcd04700 1 --2- 172.21.15.23:0/1045749483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d8100780 0x7fb8d8101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:51.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.242+0000 7fb8dcd04700 1 --2- 172.21.15.23:0/1045749483 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8d8101140 0x7fb8d8101ea0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:51.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.242+0000 7fb8dcd04700 1 -- 172.21.15.23:0/1045749483 >> 172.21.15.23:0/1045749483 conn(0x7fb8d8077690 msgr2=0x7fb8d81050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:51.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.242+0000 7fb8dcd04700 1 -- 172.21.15.23:0/1045749483 shutdown_connections 2024-03-28T05:22:51.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.242+0000 7fb8dcd04700 1 -- 172.21.15.23:0/1045749483 wait complete. 2024-03-28T05:22:51.882 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-28T05:22:51.882 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T05:22:51.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.989+0000 7fe88cd52640 1 -- 172.21.15.23:0/270989701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8881017d0 msgr2=0x7fe888101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:51.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.989+0000 7fe88cd52640 1 --2- 172.21.15.23:0/270989701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8881017d0 0x7fe888101c50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009900 tx=0x7fe87c02bc30 comp rx=0 tx=0).stop 2024-03-28T05:22:51.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.989+0000 7fe88cd52640 1 -- 172.21.15.23:0/270989701 shutdown_connections 2024-03-28T05:22:51.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.989+0000 7fe88cd52640 1 --2- 172.21.15.23:0/270989701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe888102190 0x7fe888108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:51.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.989+0000 7fe88cd52640 1 --2- 172.21.15.23:0/270989701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8881017d0 0x7fe888101c50 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:51.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.989+0000 7fe88cd52640 1 --2- 172.21.15.23:0/270989701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8881005e0 0x7fe888100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:51.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.989+0000 7fe88cd52640 1 -- 172.21.15.23:0/270989701 >> 172.21.15.23:0/270989701 conn(0x7fe8880fbb50 msgr2=0x7fe8880fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:51.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.989+0000 7fe88cd52640 1 -- 172.21.15.23:0/270989701 shutdown_connections 2024-03-28T05:22:51.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.989+0000 7fe88cd52640 1 -- 172.21.15.23:0/270989701 wait complete. 2024-03-28T05:22:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.990+0000 7fe88cd52640 1 Processor -- start 2024-03-28T05:22:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.990+0000 7fe88cd52640 1 -- start start 2024-03-28T05:22:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.990+0000 7fe88cd52640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8881005e0 0x7fe88806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.990+0000 7fe88cd52640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8881017d0 0x7fe88819c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.990+0000 7fe88cd52640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe888102190 0x7fe88819edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.990+0000 7fe88cd52640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88810c240 con 0x7fe8881017d0 2024-03-28T05:22:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.990+0000 7fe88cd52640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88810c0c0 con 0x7fe888102190 2024-03-28T05:22:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.990+0000 7fe88cd52640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88810c3c0 con 0x7fe8881005e0 2024-03-28T05:22:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.990+0000 7fe885d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8881017d0 0x7fe88819c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.990+0000 7fe88659c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8881005e0 0x7fe88806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.990+0000 7fe885d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8881017d0 0x7fe88819c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43868/0 (socket says 172.21.15.23:43868) 2024-03-28T05:22:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.990+0000 7fe885d9b640 1 -- 172.21.15.23:0/88578077 learned_addr learned my addr 172.21.15.23:0/88578077 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.990+0000 7fe886d9d640 1 --2- 172.21.15.23:0/88578077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe888102190 0x7fe88819edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.991+0000 7fe885d9b640 1 -- 172.21.15.23:0/88578077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8881005e0 msgr2=0x7fe88806a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.991+0000 7fe885d9b640 1 --2- 172.21.15.23:0/88578077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8881005e0 0x7fe88806a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.991+0000 7fe885d9b640 1 -- 172.21.15.23:0/88578077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe888102190 msgr2=0x7fe88819edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.991+0000 7fe885d9b640 1 --2- 172.21.15.23:0/88578077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe888102190 0x7fe88819edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.991+0000 7fe885d9b640 1 -- 172.21.15.23:0/88578077 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe88819f680 con 0x7fe8881017d0 2024-03-28T05:22:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.991+0000 7fe88659c640 1 --2- 172.21.15.23:0/88578077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8881005e0 0x7fe88806a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:22:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.991+0000 7fe885d9b640 1 --2- 172.21.15.23:0/88578077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8881017d0 0x7fe88819c780 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe87c006010 tx=0x7fe87c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.991+0000 7fe8777fe640 1 -- 172.21.15.23:0/88578077 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c041070 con 0x7fe8881017d0 2024-03-28T05:22:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.991+0000 7fe8777fe640 1 -- 172.21.15.23:0/88578077 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe87c0054e0 con 0x7fe8881017d0 2024-03-28T05:22:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.991+0000 7fe88cd52640 1 -- 172.21.15.23:0/88578077 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe88819f910 con 0x7fe8881017d0 2024-03-28T05:22:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.991+0000 7fe8777fe640 1 -- 172.21.15.23:0/88578077 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c034440 con 0x7fe8881017d0 2024-03-28T05:22:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.991+0000 7fe88cd52640 1 -- 172.21.15.23:0/88578077 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe88819fe00 con 0x7fe8881017d0 2024-03-28T05:22:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.992+0000 7fe88cd52640 1 -- 172.21.15.23:0/88578077 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8881078a0 con 0x7fe8881017d0 2024-03-28T05:22:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.993+0000 7fe8777fe640 1 -- 172.21.15.23:0/88578077 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe87c044070 con 0x7fe8881017d0 2024-03-28T05:22:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.994+0000 7fe8777fe640 1 --2- 172.21.15.23:0/88578077 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe8700a2800 0x7fe8700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.994+0000 7fe88659c640 1 --2- 172.21.15.23:0/88578077 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe8700a2800 0x7fe8700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.994+0000 7fe8777fe640 1 -- 172.21.15.23:0/88578077 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fe87c0f0a00 con 0x7fe8881017d0 2024-03-28T05:22:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.994+0000 7fe88659c640 1 --2- 172.21.15.23:0/88578077 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe8700a2800 0x7fe8700a4cc0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe888101630 tx=0x7fe87800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:51.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:51.998+0000 7fe8777fe640 1 -- 172.21.15.23:0/88578077 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe87c0bd0f0 con 0x7fe8881017d0 2024-03-28T05:22:52.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.185+0000 7fe88cd52640 1 -- 172.21.15.23:0/88578077 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8880622a0 con 0x7fe8881017d0 2024-03-28T05:22:52.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.187+0000 7fe8777fe640 1 -- 172.21.15.23:0/88578077 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24554 (secure 0 0 0) 0x7fe87c0c1190 con 0x7fe8881017d0 2024-03-28T05:22:52.186 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:22:52.186 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":91,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","created":"2024-03-28T05:13:13.453653+0000","modified":"2024-03-28T05:22:10.494031+0000","last_up_change":"2024-03-28T05:20:20.221527+0000","last_in_change":"2024-03-28T05:20:00.365478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T05:16:32.986974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T05:21:06.103894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T05:21:06.827190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T05:21:10.507418+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"18150942-59a4-4c3b-83d4-c1fc3ccf1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6803","nonce":1219956881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6805","nonce":1219956881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6809","nonce":1219956881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6807","nonce":1219956881}]},"public_addr":"172.21.15.23:6803/1219956881","cluster_addr":"172.21.15.23:6805/1219956881","heartbeat_back_addr":"172.21.15.23:6809/1219956881","heartbeat_front_addr":"172.21.15.23:6807/1219956881","state":["exists","up"]},{"osd":1,"uuid":"ba6d1a95-5135-4670-95f9-7878c0e7bd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6801","nonce":555491234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6803","nonce":555491234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6807","nonce":555491234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6805","nonce":555491234}]},"public_addr":"172.21.15.98:6801/555491234","cluster_addr":"172.21.15.98:6803/555491234","heartbeat_back_addr":"172.21.15.98:6807/555491234","heartbeat_front_addr":"172.21.15.98:6805/555491234","state":["exists","up"]},{"osd":2,"uuid":"873bdd60-2a47-403b-b1f4-ee9226a95067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6801","nonce":1535856914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6803","nonce":1535856914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6807","nonce":1535856914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6805","nonce":1535856914}]},"public_addr":"172.21.15.113:6801/1535856914","cluster_addr":"172.21.15.113:6803/1535856914","heartbeat_back_addr":"172.21.15.113:6807/1535856914","heartbeat_front_addr":"172.21.15.113:6805/1535856914","state":["exists","up"]},{"osd":3,"uuid":"83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6811","nonce":2949415020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6813","nonce":2949415020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6817","nonce":2949415020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6815","nonce":2949415020}]},"public_addr":"172.21.15.23:6811/2949415020","cluster_addr":"172.21.15.23:6813/2949415020","heartbeat_back_addr":"172.21.15.23:6817/2949415020","heartbeat_front_addr":"172.21.15.23:6815/2949415020","state":["exists","up"]},{"osd":4,"uuid":"76a90bac-b221-40a3-8238-40e8049cec36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6809","nonce":2615902819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6811","nonce":2615902819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6815","nonce":2615902819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6813","nonce":2615902819}]},"public_addr":"172.21.15.98:6809/2615902819","cluster_addr":"172.21.15.98:6811/2615902819","heartbeat_back_addr":"172.21.15.98:6815/2615902819","heartbeat_front_addr":"172.21.15.98:6813/2615902819","state":["exists","up"]},{"osd":5,"uuid":"b53fdb55-a8df-40a8-bd1b-8bcef634e26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6809","nonce":625271528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6811","nonce":625271528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6815","nonce":625271528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6813","nonce":625271528}]},"public_addr":"172.21.15.113:6809/625271528","cluster_addr":"172.21.15.113:6811/625271528","heartbeat_back_addr":"172.21.15.113:6815/625271528","heartbeat_front_addr":"172.21.15.113:6813/625271528","state":["exists","up"]},{"osd":6,"uuid":"f00283d2-8d67-4e97-a575-9a9d3b55d5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6819","nonce":4188464176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6821","nonce":4188464176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6825","nonce":4188464176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6823","nonce":4188464176}]},"public_addr":"172.21.15.23:6819/4188464176","cluster_addr":"172.21.15.23:6821/4188464176","heartbeat_back_addr":"172.21.15.23:6825/4188464176","heartbeat_front_addr":"172.21.15.23:6823/4188464176","state":["exists","up"]},{"osd":7,"uuid":"edc008a7-b07b-40a8-88b7-1614ba38a775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6817","nonce":1985546028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6819","nonce":1985546028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6823","nonce":1985546028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6821","nonce":1985546028}]},"public_addr":"172.21.15.98:6817/1985546028","cluster_addr":"172.21.15.98:6819/1985546028","heartbeat_back_addr":"172.21.15.98:6823/1985546028","heartbeat_front_addr":"172.21.15.98:6821/1985546028","state":["exists","up"]},{"osd":8,"uuid":"56e7b2d1-81f0-47b2-8638-f29832bed8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6817","nonce":3534876081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6819","nonce":3534876081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6823","nonce":3534876081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6821","nonce":3534876081}]},"public_addr":"172.21.15.113:6817/3534876081","cluster_addr":"172.21.15.113:6819/3534876081","heartbeat_back_addr":"172.21.15.113:6823/3534876081","heartbeat_front_addr":"172.21.15.113:6821/3534876081","state":["exists","up"]},{"osd":9,"uuid":"a2e90caa-6e9c-42bf-9f18-81f7f47880cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6827","nonce":672264207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6829","nonce":672264207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6833","nonce":672264207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6831","nonce":672264207}]},"public_addr":"172.21.15.23:6827/672264207","cluster_addr":"172.21.15.23:6829/672264207","heartbeat_back_addr":"172.21.15.23:6833/672264207","heartbeat_front_addr":"172.21.15.23:6831/672264207","state":["exists","up"]},{"osd":10,"uuid":"0eda917f-020c-437f-95ae-b26203f6c68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6825","nonce":2699920595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6827","nonce":2699920595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6831","nonce":2699920595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6829","nonce":2699920595}]},"public_addr":"172.21.15.98:6825/2699920595","cluster_addr":"172.21.15.98:6827/2699920595","heartbeat_back_addr":"172.21.15.98:6831/2699920595","heartbeat_front_addr":"172.21.15.98:6829/2699920595","state":["exists","up"]},{"osd":11,"uuid":"aae78501-78a8-469a-be20-3006f5ec5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6825","nonce":2113644020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6827","nonce":2113644020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6831","nonce":2113644020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6829","nonce":2113644020}]},"public_addr":"172.21.15.113:6825/2113644020","cluster_addr":"172.21.15.113:6827/2113644020","heartbeat_back_addr":"172.21.15.113:6831/2113644020","heartbeat_front_addr":"172.21.15.113:6829/2113644020","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:15:43.527475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:07.006538+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:30.594176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:55.810999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:20.595811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:45.474138+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:10.500409+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:35.787638+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:01.001188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:26.761115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:52.952752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:20:18.559832+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":7}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:6800/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1876451381":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2253559862":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2653562762":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/3337463830":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6800/1735221490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/2784771635":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2389902490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6801/1735221490":"2024-03-29T05:13:42.579607+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T05:22:52.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.189+0000 7fe88cd52640 1 -- 172.21.15.23:0/88578077 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe8700a2800 msgr2=0x7fe8700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:52.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.189+0000 7fe88cd52640 1 --2- 172.21.15.23:0/88578077 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe8700a2800 0x7fe8700a4cc0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe888101630 tx=0x7fe87800b390 comp rx=0 tx=0).stop 2024-03-28T05:22:52.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.189+0000 7fe88cd52640 1 -- 172.21.15.23:0/88578077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8881017d0 msgr2=0x7fe88819c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:52.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.189+0000 7fe88cd52640 1 --2- 172.21.15.23:0/88578077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8881017d0 0x7fe88819c780 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe87c006010 tx=0x7fe87c005dd0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.190+0000 7fe88cd52640 1 -- 172.21.15.23:0/88578077 shutdown_connections 2024-03-28T05:22:52.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.190+0000 7fe88cd52640 1 --2- 172.21.15.23:0/88578077 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe8700a2800 0x7fe8700a4cc0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.190+0000 7fe88cd52640 1 --2- 172.21.15.23:0/88578077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe888102190 0x7fe88819edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.190+0000 7fe88cd52640 1 --2- 172.21.15.23:0/88578077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8881017d0 0x7fe88819c780 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.190+0000 7fe88cd52640 1 --2- 172.21.15.23:0/88578077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8881005e0 0x7fe88806a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.190+0000 7fe88cd52640 1 -- 172.21.15.23:0/88578077 >> 172.21.15.23:0/88578077 conn(0x7fe8880fbb50 msgr2=0x7fe888104a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:52.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.190+0000 7fe88cd52640 1 -- 172.21.15.23:0/88578077 shutdown_connections 2024-03-28T05:22:52.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.190+0000 7fe88cd52640 1 -- 172.21.15.23:0/88578077 wait complete. 2024-03-28T05:22:52.200 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T05:16:32.986974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T05:21:06.103894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T05:21:06.827190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T05:21:10.507418+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T05:22:52.201 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T05:22:52.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.309+0000 7ff8a77d4640 1 -- 172.21.15.23:0/133975692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8a01005c0 msgr2=0x7ff8a01009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:52.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.309+0000 7ff8a77d4640 1 --2- 172.21.15.23:0/133975692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8a01005c0 0x7ff8a01009e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff89c009900 tx=0x7ff89c02bc30 comp rx=0 tx=0).stop 2024-03-28T05:22:52.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.309+0000 7ff8a77d4640 1 -- 172.21.15.23:0/133975692 shutdown_connections 2024-03-28T05:22:52.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.309+0000 7ff8a77d4640 1 --2- 172.21.15.23:0/133975692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8a0102170 0x7ff8a0108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.309+0000 7ff8a77d4640 1 --2- 172.21.15.23:0/133975692 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8a01017b0 0x7ff8a0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.309+0000 7ff8a77d4640 1 --2- 172.21.15.23:0/133975692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8a01005c0 0x7ff8a01009e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.309+0000 7ff8a77d4640 1 -- 172.21.15.23:0/133975692 >> 172.21.15.23:0/133975692 conn(0x7ff8a00fbb50 msgr2=0x7ff8a00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:52.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.309+0000 7ff8a77d4640 1 -- 172.21.15.23:0/133975692 shutdown_connections 2024-03-28T05:22:52.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.309+0000 7ff8a77d4640 1 -- 172.21.15.23:0/133975692 wait complete. 2024-03-28T05:22:52.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.309+0000 7ff8a77d4640 1 Processor -- start 2024-03-28T05:22:52.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.309+0000 7ff8a77d4640 1 -- start start 2024-03-28T05:22:52.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.310+0000 7ff8a77d4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8a01005c0 0x7ff8a006a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:52.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.310+0000 7ff8a77d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8a01017b0 0x7ff8a019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:52.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.310+0000 7ff8a77d4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8a0102170 0x7ff8a019eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:52.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.310+0000 7ff8a77d4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a010c230 con 0x7ff8a01005c0 2024-03-28T05:22:52.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.310+0000 7ff8a77d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a010c0b0 con 0x7ff8a01017b0 2024-03-28T05:22:52.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.310+0000 7ff8a77d4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a010c3b0 con 0x7ff8a0102170 2024-03-28T05:22:52.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.310+0000 7ff8a5570640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8a01005c0 0x7ff8a006a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:52.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.310+0000 7ff8a5570640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8a01005c0 0x7ff8a006a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43882/0 (socket says 172.21.15.23:43882) 2024-03-28T05:22:52.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.310+0000 7ff8a5570640 1 -- 172.21.15.23:0/1567539600 learned_addr learned my addr 172.21.15.23:0/1567539600 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:52.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.310+0000 7ff8a4d6f640 1 --2- 172.21.15.23:0/1567539600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8a01017b0 0x7ff8a019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:52.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.310+0000 7ff8a5d71640 1 --2- 172.21.15.23:0/1567539600 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8a0102170 0x7ff8a019eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:52.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.310+0000 7ff8a5570640 1 -- 172.21.15.23:0/1567539600 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8a0102170 msgr2=0x7ff8a019eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:52.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.310+0000 7ff8a5570640 1 --2- 172.21.15.23:0/1567539600 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8a0102170 0x7ff8a019eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.310+0000 7ff8a5570640 1 -- 172.21.15.23:0/1567539600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8a01017b0 msgr2=0x7ff8a019c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:52.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.310+0000 7ff8a5570640 1 --2- 172.21.15.23:0/1567539600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8a01017b0 0x7ff8a019c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.310+0000 7ff8a5570640 1 -- 172.21.15.23:0/1567539600 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a019f6b0 con 0x7ff8a01005c0 2024-03-28T05:22:52.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.310+0000 7ff8a5570640 1 --2- 172.21.15.23:0/1567539600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8a01005c0 0x7ff8a006a820 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff89c000c00 tx=0x7ff89c02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:52.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.310+0000 7ff8967fc640 1 -- 172.21.15.23:0/1567539600 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c041070 con 0x7ff8a01005c0 2024-03-28T05:22:52.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.310+0000 7ff8967fc640 1 -- 172.21.15.23:0/1567539600 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff89c004450 con 0x7ff8a01005c0 2024-03-28T05:22:52.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.310+0000 7ff8a77d4640 1 -- 172.21.15.23:0/1567539600 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a019f8e0 con 0x7ff8a01005c0 2024-03-28T05:22:52.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.310+0000 7ff8967fc640 1 -- 172.21.15.23:0/1567539600 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c034440 con 0x7ff8a01005c0 2024-03-28T05:22:52.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.310+0000 7ff8a77d4640 1 -- 172.21.15.23:0/1567539600 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a019fdd0 con 0x7ff8a01005c0 2024-03-28T05:22:52.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.311+0000 7ff8a77d4640 1 -- 172.21.15.23:0/1567539600 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a0107900 con 0x7ff8a01005c0 2024-03-28T05:22:52.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.312+0000 7ff8967fc640 1 -- 172.21.15.23:0/1567539600 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff89c044070 con 0x7ff8a01005c0 2024-03-28T05:22:52.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.313+0000 7ff8967fc640 1 --2- 172.21.15.23:0/1567539600 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff88c0a2800 0x7ff88c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:52.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.313+0000 7ff8a4d6f640 1 --2- 172.21.15.23:0/1567539600 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff88c0a2800 0x7ff88c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:52.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.313+0000 7ff8967fc640 1 -- 172.21.15.23:0/1567539600 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7ff89c0f08a0 con 0x7ff8a01005c0 2024-03-28T05:22:52.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.313+0000 7ff8a4d6f640 1 --2- 172.21.15.23:0/1567539600 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff88c0a2800 0x7ff88c0a4cc0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff8a006a590 tx=0x7ff890005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:52.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.317+0000 7ff8967fc640 1 -- 172.21.15.23:0/1567539600 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff89c0bd020 con 0x7ff8a01005c0 2024-03-28T05:22:52.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:52 smithi113 ceph-mon[37164]: pgmap v342: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:22:52.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:52 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1045749483' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T05:22:52.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:52 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/88578077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:22:52.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.500+0000 7ff8a77d4640 1 -- 172.21.15.23:0/1567539600 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff8a002d070 con 0x7ff8a01005c0 2024-03-28T05:22:52.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.500+0000 7ff8967fc640 1 -- 172.21.15.23:0/1567539600 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff89c0c10c0 con 0x7ff8a01005c0 2024-03-28T05:22:52.499 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-28T05:22:52.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.502+0000 7ff8a77d4640 1 -- 172.21.15.23:0/1567539600 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff88c0a2800 msgr2=0x7ff88c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:52.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.502+0000 7ff8a77d4640 1 --2- 172.21.15.23:0/1567539600 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff88c0a2800 0x7ff88c0a4cc0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff8a006a590 tx=0x7ff890005f90 comp rx=0 tx=0).stop 2024-03-28T05:22:52.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.502+0000 7ff8a77d4640 1 -- 172.21.15.23:0/1567539600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8a01005c0 msgr2=0x7ff8a006a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:52.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.502+0000 7ff8a77d4640 1 --2- 172.21.15.23:0/1567539600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8a01005c0 0x7ff8a006a820 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff89c000c00 tx=0x7ff89c02c640 comp rx=0 tx=0).stop 2024-03-28T05:22:52.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.502+0000 7ff8a77d4640 1 -- 172.21.15.23:0/1567539600 shutdown_connections 2024-03-28T05:22:52.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.502+0000 7ff8a77d4640 1 --2- 172.21.15.23:0/1567539600 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff88c0a2800 0x7ff88c0a4cc0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.502+0000 7ff8a77d4640 1 --2- 172.21.15.23:0/1567539600 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff8a0102170 0x7ff8a019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.502+0000 7ff8a77d4640 1 --2- 172.21.15.23:0/1567539600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8a01017b0 0x7ff8a019c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.502+0000 7ff8a77d4640 1 --2- 172.21.15.23:0/1567539600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8a01005c0 0x7ff8a006a820 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.502+0000 7ff8a77d4640 1 -- 172.21.15.23:0/1567539600 >> 172.21.15.23:0/1567539600 conn(0x7ff8a00fbb50 msgr2=0x7ff8a0104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:52.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.502+0000 7ff8a77d4640 1 -- 172.21.15.23:0/1567539600 shutdown_connections 2024-03-28T05:22:52.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.502+0000 7ff8a77d4640 1 -- 172.21.15.23:0/1567539600 wait complete. 2024-03-28T05:22:52.511 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:52 smithi023 ceph-mon[31954]: pgmap v342: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:22:52.511 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:52 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1045749483' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T05:22:52.511 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:52 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/88578077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:22:52.512 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T05:22:52.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:52 smithi098 ceph-mon[36087]: pgmap v342: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:22:52.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:52 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1045749483' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T05:22:52.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:52 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/88578077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:22:52.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.619+0000 7f788c87e640 1 -- 172.21.15.23:0/304397047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78881005c0 msgr2=0x7f78881009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:52.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.619+0000 7f788c87e640 1 --2- 172.21.15.23:0/304397047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78881005c0 0x7f78881009e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7870009900 tx=0x7f787002bc30 comp rx=0 tx=0).stop 2024-03-28T05:22:52.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.619+0000 7f788c87e640 1 -- 172.21.15.23:0/304397047 shutdown_connections 2024-03-28T05:22:52.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.619+0000 7f788c87e640 1 --2- 172.21.15.23:0/304397047 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7888102170 0x7f7888108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.619+0000 7f788c87e640 1 --2- 172.21.15.23:0/304397047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78881017b0 0x7f7888101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.619+0000 7f788c87e640 1 --2- 172.21.15.23:0/304397047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78881005c0 0x7f78881009e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.619+0000 7f788c87e640 1 -- 172.21.15.23:0/304397047 >> 172.21.15.23:0/304397047 conn(0x7f78880fbb50 msgr2=0x7f78880fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:52.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.619+0000 7f788c87e640 1 -- 172.21.15.23:0/304397047 shutdown_connections 2024-03-28T05:22:52.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.619+0000 7f788c87e640 1 -- 172.21.15.23:0/304397047 wait complete. 2024-03-28T05:22:52.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.620+0000 7f788c87e640 1 Processor -- start 2024-03-28T05:22:52.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.620+0000 7f788c87e640 1 -- start start 2024-03-28T05:22:52.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.620+0000 7f788c87e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78881005c0 0x7f7888197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:52.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.620+0000 7f788c87e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78881017b0 0x7f7888197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:52.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.620+0000 7f788c87e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7888102170 0x7f788819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:52.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.620+0000 7f788c87e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788810c230 con 0x7f78881005c0 2024-03-28T05:22:52.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.620+0000 7f788c87e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788810c0b0 con 0x7f78881017b0 2024-03-28T05:22:52.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.620+0000 7f788c87e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788810c3b0 con 0x7f7888102170 2024-03-28T05:22:52.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.620+0000 7f788659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78881005c0 0x7f7888197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:52.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.620+0000 7f788659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78881005c0 0x7f7888197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43900/0 (socket says 172.21.15.23:43900) 2024-03-28T05:22:52.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.620+0000 7f788659c640 1 -- 172.21.15.23:0/4189273490 learned_addr learned my addr 172.21.15.23:0/4189273490 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:52.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.620+0000 7f7886d9d640 1 --2- 172.21.15.23:0/4189273490 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7888102170 0x7f788819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:52.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.620+0000 7f7885d9b640 1 --2- 172.21.15.23:0/4189273490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78881017b0 0x7f7888197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:52.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.620+0000 7f788659c640 1 -- 172.21.15.23:0/4189273490 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7888102170 msgr2=0x7f788819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:52.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.620+0000 7f788659c640 1 --2- 172.21.15.23:0/4189273490 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7888102170 0x7f788819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.620+0000 7f788659c640 1 -- 172.21.15.23:0/4189273490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78881017b0 msgr2=0x7f7888197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:52.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.620+0000 7f788659c640 1 --2- 172.21.15.23:0/4189273490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78881017b0 0x7f7888197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.620+0000 7f788659c640 1 -- 172.21.15.23:0/4189273490 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788819f620 con 0x7f78881005c0 2024-03-28T05:22:52.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.620+0000 7f7885d9b640 1 --2- 172.21.15.23:0/4189273490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78881017b0 0x7f7888197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:22:52.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.620+0000 7f788659c640 1 --2- 172.21.15.23:0/4189273490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78881005c0 0x7f7888197a50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7870000c00 tx=0x7f7870005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:52.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.621+0000 7f787f7fe640 1 -- 172.21.15.23:0/4189273490 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7870041070 con 0x7f78881005c0 2024-03-28T05:22:52.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.621+0000 7f787f7fe640 1 -- 172.21.15.23:0/4189273490 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78700044d0 con 0x7f78881005c0 2024-03-28T05:22:52.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.621+0000 7f787f7fe640 1 -- 172.21.15.23:0/4189273490 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7870034440 con 0x7f78881005c0 2024-03-28T05:22:52.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.621+0000 7f788c87e640 1 -- 172.21.15.23:0/4189273490 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f788819f8b0 con 0x7f78881005c0 2024-03-28T05:22:52.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.621+0000 7f788c87e640 1 -- 172.21.15.23:0/4189273490 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f788819fdf0 con 0x7f78881005c0 2024-03-28T05:22:52.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.622+0000 7f788c87e640 1 -- 172.21.15.23:0/4189273490 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f788819fff0 con 0x7f78881005c0 2024-03-28T05:22:52.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.623+0000 7f787f7fe640 1 -- 172.21.15.23:0/4189273490 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7870044070 con 0x7f78881005c0 2024-03-28T05:22:52.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.623+0000 7f787f7fe640 1 --2- 172.21.15.23:0/4189273490 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f78740a2880 0x7f78740a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:52.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.623+0000 7f7885d9b640 1 --2- 172.21.15.23:0/4189273490 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f78740a2880 0x7f78740a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:52.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.623+0000 7f787f7fe640 1 -- 172.21.15.23:0/4189273490 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f78700f0ad0 con 0x7f78881005c0 2024-03-28T05:22:52.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.624+0000 7f7885d9b640 1 --2- 172.21.15.23:0/4189273490 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f78740a2880 0x7f78740a4d40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7888198f10 tx=0x7f7878005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:52.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.627+0000 7f787f7fe640 1 -- 172.21.15.23:0/4189273490 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f78700bd810 con 0x7f78881005c0 2024-03-28T05:22:52.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.810+0000 7f788c87e640 1 -- 172.21.15.23:0/4189273490 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f78881a01c0 con 0x7f78881005c0 2024-03-28T05:22:52.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.811+0000 7f787f7fe640 1 -- 172.21.15.23:0/4189273490 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f78700bd110 con 0x7f78881005c0 2024-03-28T05:22:52.810 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T05:22:52.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.813+0000 7f788c87e640 1 -- 172.21.15.23:0/4189273490 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f78740a2880 msgr2=0x7f78740a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:52.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.813+0000 7f788c87e640 1 --2- 172.21.15.23:0/4189273490 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f78740a2880 0x7f78740a4d40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7888198f10 tx=0x7f7878005e50 comp rx=0 tx=0).stop 2024-03-28T05:22:52.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.813+0000 7f788c87e640 1 -- 172.21.15.23:0/4189273490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78881005c0 msgr2=0x7f7888197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:52.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.813+0000 7f788c87e640 1 --2- 172.21.15.23:0/4189273490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78881005c0 0x7f7888197a50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7870000c00 tx=0x7f7870005d50 comp rx=0 tx=0).stop 2024-03-28T05:22:52.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.813+0000 7f788c87e640 1 -- 172.21.15.23:0/4189273490 shutdown_connections 2024-03-28T05:22:52.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.813+0000 7f788c87e640 1 --2- 172.21.15.23:0/4189273490 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f78740a2880 0x7f78740a4d40 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.813+0000 7f788c87e640 1 --2- 172.21.15.23:0/4189273490 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7888102170 0x7f788819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.813+0000 7f788c87e640 1 --2- 172.21.15.23:0/4189273490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78881017b0 0x7f7888197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.813+0000 7f788c87e640 1 --2- 172.21.15.23:0/4189273490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78881005c0 0x7f7888197a50 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.813+0000 7f788c87e640 1 -- 172.21.15.23:0/4189273490 >> 172.21.15.23:0/4189273490 conn(0x7f78880fbb50 msgr2=0x7f78880fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:52.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.813+0000 7f788c87e640 1 -- 172.21.15.23:0/4189273490 shutdown_connections 2024-03-28T05:22:52.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.813+0000 7f788c87e640 1 -- 172.21.15.23:0/4189273490 wait complete. 2024-03-28T05:22:52.822 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T05:22:52.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.929+0000 7f8eade7e640 1 -- 172.21.15.23:0/2015808116 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ea8102120 msgr2=0x7f8ea8104510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:52.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.929+0000 7f8eade7e640 1 --2- 172.21.15.23:0/2015808116 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ea8102120 0x7f8ea8104510 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8e98009900 tx=0x7f8e9802bc30 comp rx=0 tx=0).stop 2024-03-28T05:22:52.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.929+0000 7f8eade7e640 1 -- 172.21.15.23:0/2015808116 shutdown_connections 2024-03-28T05:22:52.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.929+0000 7f8eade7e640 1 --2- 172.21.15.23:0/2015808116 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ea8107380 0x7f8ea81097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.929+0000 7f8eade7e640 1 --2- 172.21.15.23:0/2015808116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea8104a50 0x7f8ea8106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.929+0000 7f8eade7e640 1 --2- 172.21.15.23:0/2015808116 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ea8102120 0x7f8ea8104510 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.929+0000 7f8eade7e640 1 -- 172.21.15.23:0/2015808116 >> 172.21.15.23:0/2015808116 conn(0x7f8ea80fbb30 msgr2=0x7f8ea80fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:52.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.929+0000 7f8eade7e640 1 -- 172.21.15.23:0/2015808116 shutdown_connections 2024-03-28T05:22:52.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.929+0000 7f8eade7e640 1 -- 172.21.15.23:0/2015808116 wait complete. 2024-03-28T05:22:52.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.930+0000 7f8eade7e640 1 Processor -- start 2024-03-28T05:22:52.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.930+0000 7f8eade7e640 1 -- start start 2024-03-28T05:22:52.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.930+0000 7f8eade7e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ea8102120 0x7f8ea81979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:52.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.930+0000 7f8eade7e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ea8104a50 0x7f8ea8197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:52.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.930+0000 7f8eade7e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea8107380 0x7f8ea819ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:52.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.930+0000 7f8eade7e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea810d320 con 0x7f8ea8104a50 2024-03-28T05:22:52.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.930+0000 7f8eade7e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea810d1a0 con 0x7f8ea8107380 2024-03-28T05:22:52.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.930+0000 7f8eade7e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea810d4a0 con 0x7f8ea8102120 2024-03-28T05:22:52.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.930+0000 7f8ea6ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ea8104a50 0x7f8ea8197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:52.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.930+0000 7f8ea77fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ea8102120 0x7f8ea81979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:52.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.930+0000 7f8ea6ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ea8104a50 0x7f8ea8197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34134/0 (socket says 172.21.15.23:34134) 2024-03-28T05:22:52.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.930+0000 7f8ea6ffd640 1 -- 172.21.15.23:0/468017425 learned_addr learned my addr 172.21.15.23:0/468017425 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:52.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.931+0000 7f8ea7fff640 1 --2- 172.21.15.23:0/468017425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea8107380 0x7f8ea819ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:52.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.931+0000 7f8ea77fe640 1 -- 172.21.15.23:0/468017425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea8107380 msgr2=0x7f8ea819ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:52.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.931+0000 7f8ea77fe640 1 --2- 172.21.15.23:0/468017425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea8107380 0x7f8ea819ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.931+0000 7f8ea77fe640 1 -- 172.21.15.23:0/468017425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ea8104a50 msgr2=0x7f8ea8197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:52.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.931+0000 7f8ea77fe640 1 --2- 172.21.15.23:0/468017425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ea8104a50 0x7f8ea8197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:52.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.931+0000 7f8ea77fe640 1 -- 172.21.15.23:0/468017425 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea819f5a0 con 0x7f8ea8102120 2024-03-28T05:22:52.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.931+0000 7f8ea6ffd640 1 --2- 172.21.15.23:0/468017425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ea8104a50 0x7f8ea8197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:22:52.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.931+0000 7f8ea77fe640 1 --2- 172.21.15.23:0/468017425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ea8102120 0x7f8ea81979f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8e98009230 tx=0x7f8e9802c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:52.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.931+0000 7f8ea4ff9640 1 -- 172.21.15.23:0/468017425 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98041070 con 0x7f8ea8102120 2024-03-28T05:22:52.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.931+0000 7f8eade7e640 1 -- 172.21.15.23:0/468017425 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea819f830 con 0x7f8ea8102120 2024-03-28T05:22:52.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.931+0000 7f8ea4ff9640 1 -- 172.21.15.23:0/468017425 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e98004450 con 0x7f8ea8102120 2024-03-28T05:22:52.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.931+0000 7f8ea4ff9640 1 -- 172.21.15.23:0/468017425 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98034440 con 0x7f8ea8102120 2024-03-28T05:22:52.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.931+0000 7f8eade7e640 1 -- 172.21.15.23:0/468017425 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea819fd70 con 0x7f8ea8102120 2024-03-28T05:22:52.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.932+0000 7f8eade7e640 1 -- 172.21.15.23:0/468017425 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea80fd620 con 0x7f8ea8102120 2024-03-28T05:22:52.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.933+0000 7f8ea4ff9640 1 -- 172.21.15.23:0/468017425 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8e98044070 con 0x7f8ea8102120 2024-03-28T05:22:52.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.934+0000 7f8ea4ff9640 1 --2- 172.21.15.23:0/468017425 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8e900a2800 0x7f8e900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:52.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.934+0000 7f8ea6ffd640 1 --2- 172.21.15.23:0/468017425 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8e900a2800 0x7f8e900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:52.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.934+0000 7f8ea4ff9640 1 -- 172.21.15.23:0/468017425 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f8e980f0910 con 0x7f8ea8102120 2024-03-28T05:22:52.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.934+0000 7f8ea6ffd640 1 --2- 172.21.15.23:0/468017425 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8e900a2800 0x7f8e900a4cc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8198e90 tx=0x7f8e9c00a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:52.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:52.937+0000 7f8ea4ff9640 1 -- 172.21.15.23:0/468017425 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8e980bd5f0 con 0x7f8ea8102120 2024-03-28T05:22:53.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.123+0000 7f8eade7e640 1 -- 172.21.15.23:0/468017425 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8ea802d200 con 0x7f8ea8102120 2024-03-28T05:22:53.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.124+0000 7f8ea4ff9640 1 -- 172.21.15.23:0/468017425 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8e980bcef0 con 0x7f8ea8102120 2024-03-28T05:22:53.122 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-28T05:22:53.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.126+0000 7f8eade7e640 1 -- 172.21.15.23:0/468017425 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8e900a2800 msgr2=0x7f8e900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:53.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.126+0000 7f8eade7e640 1 --2- 172.21.15.23:0/468017425 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8e900a2800 0x7f8e900a4cc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8198e90 tx=0x7f8e9c00a380 comp rx=0 tx=0).stop 2024-03-28T05:22:53.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.126+0000 7f8eade7e640 1 -- 172.21.15.23:0/468017425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ea8102120 msgr2=0x7f8ea81979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:53.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.126+0000 7f8eade7e640 1 --2- 172.21.15.23:0/468017425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ea8102120 0x7f8ea81979f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8e98009230 tx=0x7f8e9802c6c0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.126+0000 7f8eade7e640 1 -- 172.21.15.23:0/468017425 shutdown_connections 2024-03-28T05:22:53.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.126+0000 7f8eade7e640 1 --2- 172.21.15.23:0/468017425 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8e900a2800 0x7f8e900a4cc0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.126+0000 7f8eade7e640 1 --2- 172.21.15.23:0/468017425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea8107380 0x7f8ea819ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.126+0000 7f8eade7e640 1 --2- 172.21.15.23:0/468017425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ea8104a50 0x7f8ea8197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.126+0000 7f8eade7e640 1 --2- 172.21.15.23:0/468017425 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ea8102120 0x7f8ea81979f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.126+0000 7f8eade7e640 1 -- 172.21.15.23:0/468017425 >> 172.21.15.23:0/468017425 conn(0x7f8ea80fbb30 msgr2=0x7f8ea8102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:53.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.126+0000 7f8eade7e640 1 -- 172.21.15.23:0/468017425 shutdown_connections 2024-03-28T05:22:53.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.126+0000 7f8eade7e640 1 -- 172.21.15.23:0/468017425 wait complete. 2024-03-28T05:22:53.136 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T05:22:53.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.243+0000 7fdbd1a4b640 1 -- 172.21.15.23:0/500982595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdbcc102140 msgr2=0x7fdbcc104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:53.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.243+0000 7fdbd1a4b640 1 --2- 172.21.15.23:0/500982595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdbcc102140 0x7fdbcc104530 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc009900 tx=0x7fdbbc02bc30 comp rx=0 tx=0).stop 2024-03-28T05:22:53.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.243+0000 7fdbd1a4b640 1 -- 172.21.15.23:0/500982595 shutdown_connections 2024-03-28T05:22:53.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.243+0000 7fdbd1a4b640 1 --2- 172.21.15.23:0/500982595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdbcc1073a0 0x7fdbcc1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.243+0000 7fdbd1a4b640 1 --2- 172.21.15.23:0/500982595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbcc104a70 0x7fdbcc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.243+0000 7fdbd1a4b640 1 --2- 172.21.15.23:0/500982595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdbcc102140 0x7fdbcc104530 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.243+0000 7fdbd1a4b640 1 -- 172.21.15.23:0/500982595 >> 172.21.15.23:0/500982595 conn(0x7fdbcc0fbb50 msgr2=0x7fdbcc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:53.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.243+0000 7fdbd1a4b640 1 -- 172.21.15.23:0/500982595 shutdown_connections 2024-03-28T05:22:53.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.243+0000 7fdbd1a4b640 1 -- 172.21.15.23:0/500982595 wait complete. 2024-03-28T05:22:53.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.244+0000 7fdbd1a4b640 1 Processor -- start 2024-03-28T05:22:53.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.244+0000 7fdbd1a4b640 1 -- start start 2024-03-28T05:22:53.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.244+0000 7fdbd1a4b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbcc102140 0x7fdbcc1957e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:53.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.244+0000 7fdbd1a4b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdbcc104a70 0x7fdbcc195d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:53.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.244+0000 7fdbd1a4b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdbcc1073a0 0x7fdbcc19cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:53.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.244+0000 7fdbd1a4b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbcc10b180 con 0x7fdbcc1073a0 2024-03-28T05:22:53.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.244+0000 7fdbd1a4b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbcc10b000 con 0x7fdbcc102140 2024-03-28T05:22:53.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.244+0000 7fdbd1a4b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbcc10b300 con 0x7fdbcc104a70 2024-03-28T05:22:53.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.244+0000 7fdbcb7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdbcc1073a0 0x7fdbcc19cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:53.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.244+0000 7fdbcb7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdbcc1073a0 0x7fdbcc19cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34154/0 (socket says 172.21.15.23:34154) 2024-03-28T05:22:53.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.244+0000 7fdbcb7fe640 1 -- 172.21.15.23:0/859764330 learned_addr learned my addr 172.21.15.23:0/859764330 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:53.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.244+0000 7fdbc3fff640 1 --2- 172.21.15.23:0/859764330 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdbcc104a70 0x7fdbcc195d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:53.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.244+0000 7fdbcaffd640 1 --2- 172.21.15.23:0/859764330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbcc102140 0x7fdbcc1957e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:53.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.244+0000 7fdbcb7fe640 1 -- 172.21.15.23:0/859764330 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdbcc104a70 msgr2=0x7fdbcc195d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:53.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.244+0000 7fdbcb7fe640 1 --2- 172.21.15.23:0/859764330 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdbcc104a70 0x7fdbcc195d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.244+0000 7fdbcb7fe640 1 -- 172.21.15.23:0/859764330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbcc102140 msgr2=0x7fdbcc1957e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:53.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.244+0000 7fdbcb7fe640 1 --2- 172.21.15.23:0/859764330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbcc102140 0x7fdbcc1957e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.244+0000 7fdbcb7fe640 1 -- 172.21.15.23:0/859764330 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbcc19d400 con 0x7fdbcc1073a0 2024-03-28T05:22:53.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.245+0000 7fdbcb7fe640 1 --2- 172.21.15.23:0/859764330 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdbcc1073a0 0x7fdbcc19cd90 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400edb0 tx=0x7fdbc4012490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:53.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.245+0000 7fdbd0a49640 1 -- 172.21.15.23:0/859764330 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc401c030 con 0x7fdbcc1073a0 2024-03-28T05:22:53.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.245+0000 7fdbd0a49640 1 -- 172.21.15.23:0/859764330 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbc4012db0 con 0x7fdbcc1073a0 2024-03-28T05:22:53.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.245+0000 7fdbd1a4b640 1 -- 172.21.15.23:0/859764330 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbcc19d6f0 con 0x7fdbcc1073a0 2024-03-28T05:22:53.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.245+0000 7fdbd0a49640 1 -- 172.21.15.23:0/859764330 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4018400 con 0x7fdbcc1073a0 2024-03-28T05:22:53.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.245+0000 7fdbd1a4b640 1 -- 172.21.15.23:0/859764330 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbcc19dc90 con 0x7fdbcc1073a0 2024-03-28T05:22:53.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.246+0000 7fdbd1a4b640 1 -- 172.21.15.23:0/859764330 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbcc0fd640 con 0x7fdbcc1073a0 2024-03-28T05:22:53.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.247+0000 7fdbd0a49640 1 -- 172.21.15.23:0/859764330 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdbc401f070 con 0x7fdbcc1073a0 2024-03-28T05:22:53.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.247+0000 7fdbd0a49640 1 --2- 172.21.15.23:0/859764330 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fdbb80a2880 0x7fdbb80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:53.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.247+0000 7fdbcaffd640 1 --2- 172.21.15.23:0/859764330 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fdbb80a2880 0x7fdbb80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:53.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.247+0000 7fdbd0a49640 1 -- 172.21.15.23:0/859764330 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fdbc40d2d10 con 0x7fdbcc1073a0 2024-03-28T05:22:53.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.248+0000 7fdbcaffd640 1 --2- 172.21.15.23:0/859764330 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fdbb80a2880 0x7fdbb80a4d40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc006010 tx=0x7fdbbc000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:53.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.252+0000 7fdbd0a49640 1 -- 172.21.15.23:0/859764330 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdbc409fa50 con 0x7fdbcc1073a0 2024-03-28T05:22:53.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.435+0000 7fdbd1a4b640 1 -- 172.21.15.23:0/859764330 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fdbcc0622a0 con 0x7fdbcc1073a0 2024-03-28T05:22:53.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.436+0000 7fdbd0a49640 1 -- 172.21.15.23:0/859764330 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7fdbc409f350 con 0x7fdbcc1073a0 2024-03-28T05:22:53.434 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T05:22:53.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.438+0000 7fdbd1a4b640 1 -- 172.21.15.23:0/859764330 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fdbb80a2880 msgr2=0x7fdbb80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:53.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.438+0000 7fdbd1a4b640 1 --2- 172.21.15.23:0/859764330 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fdbb80a2880 0x7fdbb80a4d40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc006010 tx=0x7fdbbc000bc0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.438+0000 7fdbd1a4b640 1 -- 172.21.15.23:0/859764330 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdbcc1073a0 msgr2=0x7fdbcc19cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:53.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.438+0000 7fdbd1a4b640 1 --2- 172.21.15.23:0/859764330 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdbcc1073a0 0x7fdbcc19cd90 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400edb0 tx=0x7fdbc4012490 comp rx=0 tx=0).stop 2024-03-28T05:22:53.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.438+0000 7fdbd1a4b640 1 -- 172.21.15.23:0/859764330 shutdown_connections 2024-03-28T05:22:53.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.438+0000 7fdbd1a4b640 1 --2- 172.21.15.23:0/859764330 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fdbb80a2880 0x7fdbb80a4d40 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.438+0000 7fdbd1a4b640 1 --2- 172.21.15.23:0/859764330 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdbcc1073a0 0x7fdbcc19cd90 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.438+0000 7fdbd1a4b640 1 --2- 172.21.15.23:0/859764330 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdbcc104a70 0x7fdbcc195d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.438+0000 7fdbd1a4b640 1 --2- 172.21.15.23:0/859764330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbcc102140 0x7fdbcc1957e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.438+0000 7fdbd1a4b640 1 -- 172.21.15.23:0/859764330 >> 172.21.15.23:0/859764330 conn(0x7fdbcc0fbb50 msgr2=0x7fdbcc102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:53.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.438+0000 7fdbd1a4b640 1 -- 172.21.15.23:0/859764330 shutdown_connections 2024-03-28T05:22:53.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.438+0000 7fdbd1a4b640 1 -- 172.21.15.23:0/859764330 wait complete. 2024-03-28T05:22:53.446 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:53 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1567539600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:22:53.446 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:53 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/4189273490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:22:53.446 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:53 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/468017425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:22:53.448 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:22:53.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:53 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1567539600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:22:53.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:53 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/4189273490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:22:53.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:53 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/468017425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:22:53.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.552+0000 7f8785627640 1 -- 172.21.15.23:0/1113951578 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87801021f0 msgr2=0x7f8780108a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:53.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.552+0000 7f8785627640 1 --2- 172.21.15.23:0/1113951578 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87801021f0 0x7f8780108a80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8778009ca0 tx=0x7f877802c0f0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.552+0000 7f8785627640 1 -- 172.21.15.23:0/1113951578 shutdown_connections 2024-03-28T05:22:53.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.552+0000 7f8785627640 1 --2- 172.21.15.23:0/1113951578 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87801021f0 0x7f8780108a80 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.552+0000 7f8785627640 1 --2- 172.21.15.23:0/1113951578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8780101830 0x7f8780101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.552+0000 7f8785627640 1 --2- 172.21.15.23:0/1113951578 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8780100640 0x7f8780100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.552+0000 7f8785627640 1 -- 172.21.15.23:0/1113951578 >> 172.21.15.23:0/1113951578 conn(0x7f87800fbbd0 msgr2=0x7f87800fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:53.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.552+0000 7f8785627640 1 -- 172.21.15.23:0/1113951578 shutdown_connections 2024-03-28T05:22:53.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.552+0000 7f8785627640 1 -- 172.21.15.23:0/1113951578 wait complete. 2024-03-28T05:22:53.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.553+0000 7f8785627640 1 Processor -- start 2024-03-28T05:22:53.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.553+0000 7f8785627640 1 -- start start 2024-03-28T05:22:53.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.553+0000 7f8785627640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8780100640 0x7f8780197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:53.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.553+0000 7f8785627640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8780101830 0x7f8780197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:53.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.553+0000 7f8785627640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87801021f0 0x7f878019ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:53.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.553+0000 7f8785627640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878010c250 con 0x7f8780101830 2024-03-28T05:22:53.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.553+0000 7f8785627640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878010c0d0 con 0x7f87801021f0 2024-03-28T05:22:53.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.553+0000 7f8785627640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878010c3d0 con 0x7f8780100640 2024-03-28T05:22:53.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.553+0000 7f8777fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8780101830 0x7f8780197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:53.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.553+0000 7f8777fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8780101830 0x7f8780197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34170/0 (socket says 172.21.15.23:34170) 2024-03-28T05:22:53.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.553+0000 7f8777fff640 1 -- 172.21.15.23:0/3206504573 learned_addr learned my addr 172.21.15.23:0/3206504573 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:53.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.553+0000 7f877effd640 1 --2- 172.21.15.23:0/3206504573 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8780100640 0x7f8780197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:53.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.553+0000 7f877f7fe640 1 --2- 172.21.15.23:0/3206504573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87801021f0 0x7f878019ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:53.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.553+0000 7f8777fff640 1 -- 172.21.15.23:0/3206504573 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8780100640 msgr2=0x7f8780197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:53.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.553+0000 7f8777fff640 1 --2- 172.21.15.23:0/3206504573 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8780100640 0x7f8780197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.553+0000 7f8777fff640 1 -- 172.21.15.23:0/3206504573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87801021f0 msgr2=0x7f878019ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:53.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.553+0000 7f8777fff640 1 --2- 172.21.15.23:0/3206504573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87801021f0 0x7f878019ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.553+0000 7f8777fff640 1 -- 172.21.15.23:0/3206504573 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878019f5b0 con 0x7f8780101830 2024-03-28T05:22:53.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.553+0000 7f8777fff640 1 --2- 172.21.15.23:0/3206504573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8780101830 0x7f8780197ed0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f876800d950 tx=0x7f876800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:53.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.554+0000 7f877cff9640 1 -- 172.21.15.23:0/3206504573 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8768017070 con 0x7f8780101830 2024-03-28T05:22:53.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.554+0000 7f8785627640 1 -- 172.21.15.23:0/3206504573 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f878019f8a0 con 0x7f8780101830 2024-03-28T05:22:53.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.554+0000 7f877cff9640 1 -- 172.21.15.23:0/3206504573 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8768009c30 con 0x7f8780101830 2024-03-28T05:22:53.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.554+0000 7f877cff9640 1 -- 172.21.15.23:0/3206504573 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f876800f6b0 con 0x7f8780101830 2024-03-28T05:22:53.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.554+0000 7f8785627640 1 -- 172.21.15.23:0/3206504573 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f878019fe40 con 0x7f8780101830 2024-03-28T05:22:53.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.555+0000 7f8785627640 1 -- 172.21.15.23:0/3206504573 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87801a0070 con 0x7f8780101830 2024-03-28T05:22:53.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.556+0000 7f877cff9640 1 -- 172.21.15.23:0/3206504573 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f87680109d0 con 0x7f8780101830 2024-03-28T05:22:53.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.556+0000 7f877cff9640 1 --2- 172.21.15.23:0/3206504573 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f876c0a2800 0x7f876c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:53.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.556+0000 7f877effd640 1 --2- 172.21.15.23:0/3206504573 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f876c0a2800 0x7f876c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:53.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.556+0000 7f877cff9640 1 -- 172.21.15.23:0/3206504573 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f87680cea30 con 0x7f8780101830 2024-03-28T05:22:53.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.557+0000 7f877effd640 1 --2- 172.21.15.23:0/3206504573 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f876c0a2800 0x7f876c0a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8780101690 tx=0x7f8770005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:53.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.561+0000 7f877cff9640 1 -- 172.21.15.23:0/3206504573 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8768009de0 con 0x7f8780101830 2024-03-28T05:22:53.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:53 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1567539600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:22:53.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:53 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/4189273490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:22:53.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:53 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/468017425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:22:53.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.783+0000 7f8785627640 1 -- 172.21.15.23:0/3206504573 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87801a0400 con 0x7f8780101830 2024-03-28T05:22:53.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.783+0000 7f877cff9640 1 -- 172.21.15.23:0/3206504573 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11324 (secure 0 0 0) 0x7f876809b770 con 0x7f8780101830 2024-03-28T05:22:53.782 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:22:53.782 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/733913400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":733913400},{"type":"v1","addr":"172.21.15.23:6835","nonce":733913400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1972901387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1972901387},{"type":"v1","addr":"172.21.15.23:6837","nonce":1972901387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/2390673788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2390673788},{"type":"v1","addr":"172.21.15.23:6839","nonce":2390673788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/289263156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":289263156},{"type":"v1","addr":"172.21.15.23:6841","nonce":289263156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3313932732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3313932732},{"type":"v1","addr":"172.21.15.98:6835","nonce":3313932732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2695340024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2695340024},{"type":"v1","addr":"172.21.15.113:6835","nonce":2695340024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2750367469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2750367469},{"type":"v1","addr":"172.21.15.98:6837","nonce":2750367469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/3222726274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":3222726274},{"type":"v1","addr":"172.21.15.113:6837","nonce":3222726274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T05:21:07.616553+0000","modified":"2024-03-28T05:21:32.732132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/144113007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":144113007},{"type":"v1","addr":"172.21.15.98:6839","nonce":144113007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.113:6839/212060489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":212060489},{"type":"v1","addr":"172.21.15.113:6839","nonce":212060489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24487,"qdb_cluster":[24487]},"id":1}]} 2024-03-28T05:22:53.782 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 20 2024-03-28T05:22:53.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.785+0000 7f8785627640 1 -- 172.21.15.23:0/3206504573 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f876c0a2800 msgr2=0x7f876c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:53.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.785+0000 7f8785627640 1 --2- 172.21.15.23:0/3206504573 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f876c0a2800 0x7f876c0a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8780101690 tx=0x7f8770005f90 comp rx=0 tx=0).stop 2024-03-28T05:22:53.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.785+0000 7f8785627640 1 -- 172.21.15.23:0/3206504573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8780101830 msgr2=0x7f8780197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:53.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.785+0000 7f8785627640 1 --2- 172.21.15.23:0/3206504573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8780101830 0x7f8780197ed0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f876800d950 tx=0x7f876800de10 comp rx=0 tx=0).stop 2024-03-28T05:22:53.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.786+0000 7f8785627640 1 -- 172.21.15.23:0/3206504573 shutdown_connections 2024-03-28T05:22:53.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.786+0000 7f8785627640 1 --2- 172.21.15.23:0/3206504573 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f876c0a2800 0x7f876c0a4cc0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.786+0000 7f8785627640 1 --2- 172.21.15.23:0/3206504573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87801021f0 0x7f878019ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.786+0000 7f8785627640 1 --2- 172.21.15.23:0/3206504573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8780101830 0x7f8780197ed0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.786+0000 7f8785627640 1 --2- 172.21.15.23:0/3206504573 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8780100640 0x7f8780197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.786+0000 7f8785627640 1 -- 172.21.15.23:0/3206504573 >> 172.21.15.23:0/3206504573 conn(0x7f87800fbbd0 msgr2=0x7f87800fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:53.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.786+0000 7f8785627640 1 -- 172.21.15.23:0/3206504573 shutdown_connections 2024-03-28T05:22:53.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.786+0000 7f8785627640 1 -- 172.21.15.23:0/3206504573 wait complete. 2024-03-28T05:22:53.796 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T05:22:53.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.900+0000 7fa62ea12640 1 -- 172.21.15.23:0/1544017916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa628101830 msgr2=0x7fa628101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:53.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.900+0000 7fa62ea12640 1 --2- 172.21.15.23:0/1544017916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa628101830 0x7fa628101cb0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa618009900 tx=0x7fa61802bc30 comp rx=0 tx=0).stop 2024-03-28T05:22:53.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.900+0000 7fa62ea12640 1 -- 172.21.15.23:0/1544017916 shutdown_connections 2024-03-28T05:22:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.900+0000 7fa62ea12640 1 --2- 172.21.15.23:0/1544017916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6281021f0 0x7fa628108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.900+0000 7fa62ea12640 1 --2- 172.21.15.23:0/1544017916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa628101830 0x7fa628101cb0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.900+0000 7fa62ea12640 1 --2- 172.21.15.23:0/1544017916 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa628100640 0x7fa628100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.900+0000 7fa62ea12640 1 -- 172.21.15.23:0/1544017916 >> 172.21.15.23:0/1544017916 conn(0x7fa6280fbbd0 msgr2=0x7fa6280fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.900+0000 7fa62ea12640 1 -- 172.21.15.23:0/1544017916 shutdown_connections 2024-03-28T05:22:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.900+0000 7fa62ea12640 1 -- 172.21.15.23:0/1544017916 wait complete. 2024-03-28T05:22:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.901+0000 7fa62ea12640 1 Processor -- start 2024-03-28T05:22:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.901+0000 7fa62ea12640 1 -- start start 2024-03-28T05:22:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.901+0000 7fa62ea12640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa628100640 0x7fa6281968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.901+0000 7fa62ea12640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa628101830 0x7fa628196de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.901+0000 7fa62ea12640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6281021f0 0x7fa62819de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.901+0000 7fa62ea12640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62810c230 con 0x7fa628101830 2024-03-28T05:22:53.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.901+0000 7fa62ea12640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62810c0b0 con 0x7fa6281021f0 2024-03-28T05:22:53.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.901+0000 7fa62ea12640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62810c3b0 con 0x7fa628100640 2024-03-28T05:22:53.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.901+0000 7fa6277fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa628101830 0x7fa628196de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:53.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.901+0000 7fa6277fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa628101830 0x7fa628196de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34174/0 (socket says 172.21.15.23:34174) 2024-03-28T05:22:53.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.901+0000 7fa6277fe640 1 -- 172.21.15.23:0/3350082910 learned_addr learned my addr 172.21.15.23:0/3350082910 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:53.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.901+0000 7fa627fff640 1 --2- 172.21.15.23:0/3350082910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa628100640 0x7fa6281968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:53.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.901+0000 7fa62cfaf640 1 --2- 172.21.15.23:0/3350082910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6281021f0 0x7fa62819de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:53.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.902+0000 7fa6277fe640 1 -- 172.21.15.23:0/3350082910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa628100640 msgr2=0x7fa6281968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:53.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.902+0000 7fa6277fe640 1 --2- 172.21.15.23:0/3350082910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa628100640 0x7fa6281968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.902+0000 7fa6277fe640 1 -- 172.21.15.23:0/3350082910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6281021f0 msgr2=0x7fa62819de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:53.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.902+0000 7fa6277fe640 1 --2- 172.21.15.23:0/3350082910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6281021f0 0x7fa62819de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:53.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.902+0000 7fa6277fe640 1 -- 172.21.15.23:0/3350082910 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62819e4c0 con 0x7fa628101830 2024-03-28T05:22:53.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.902+0000 7fa627fff640 1 --2- 172.21.15.23:0/3350082910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa628100640 0x7fa6281968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:22:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.902+0000 7fa6277fe640 1 --2- 172.21.15.23:0/3350082910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa628101830 0x7fa628196de0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa618009230 tx=0x7fa618005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.902+0000 7fa6257fa640 1 -- 172.21.15.23:0/3350082910 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa618041070 con 0x7fa628101830 2024-03-28T05:22:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.902+0000 7fa6257fa640 1 -- 172.21.15.23:0/3350082910 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6180044d0 con 0x7fa628101830 2024-03-28T05:22:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.902+0000 7fa62ea12640 1 -- 172.21.15.23:0/3350082910 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa62819e750 con 0x7fa628101830 2024-03-28T05:22:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.902+0000 7fa6257fa640 1 -- 172.21.15.23:0/3350082910 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa618034440 con 0x7fa628101830 2024-03-28T05:22:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.902+0000 7fa62ea12640 1 -- 172.21.15.23:0/3350082910 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa62819ec40 con 0x7fa628101830 2024-03-28T05:22:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.903+0000 7fa62ea12640 1 -- 172.21.15.23:0/3350082910 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa628107960 con 0x7fa628101830 2024-03-28T05:22:53.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.904+0000 7fa6257fa640 1 -- 172.21.15.23:0/3350082910 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa618044070 con 0x7fa628101830 2024-03-28T05:22:53.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.904+0000 7fa6257fa640 1 --2- 172.21.15.23:0/3350082910 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa6100a2800 0x7fa6100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:53.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.905+0000 7fa627fff640 1 --2- 172.21.15.23:0/3350082910 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa6100a2800 0x7fa6100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:53.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.905+0000 7fa6257fa640 1 -- 172.21.15.23:0/3350082910 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fa6180f09c0 con 0x7fa628101830 2024-03-28T05:22:53.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.905+0000 7fa627fff640 1 --2- 172.21.15.23:0/3350082910 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa6100a2800 0x7fa6100a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa628101690 tx=0x7fa61c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:53.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:53.909+0000 7fa6257fa640 1 -- 172.21.15.23:0/3350082910 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa6180f2e20 con 0x7fa628101830 2024-03-28T05:22:54.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.099+0000 7fa62ea12640 1 -- 172.21.15.23:0/3350082910 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6280622a0 con 0x7fa628101830 2024-03-28T05:22:54.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.101+0000 7fa6257fa640 1 -- 172.21.15.23:0/3350082910 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24554 (secure 0 0 0) 0x7fa6180bd0b0 con 0x7fa628101830 2024-03-28T05:22:54.099 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:22:54.099 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":91,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","created":"2024-03-28T05:13:13.453653+0000","modified":"2024-03-28T05:22:10.494031+0000","last_up_change":"2024-03-28T05:20:20.221527+0000","last_in_change":"2024-03-28T05:20:00.365478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T05:16:32.986974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T05:21:06.103894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T05:21:06.827190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T05:21:10.507418+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"18150942-59a4-4c3b-83d4-c1fc3ccf1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6803","nonce":1219956881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6805","nonce":1219956881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6809","nonce":1219956881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6807","nonce":1219956881}]},"public_addr":"172.21.15.23:6803/1219956881","cluster_addr":"172.21.15.23:6805/1219956881","heartbeat_back_addr":"172.21.15.23:6809/1219956881","heartbeat_front_addr":"172.21.15.23:6807/1219956881","state":["exists","up"]},{"osd":1,"uuid":"ba6d1a95-5135-4670-95f9-7878c0e7bd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6801","nonce":555491234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6803","nonce":555491234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6807","nonce":555491234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6805","nonce":555491234}]},"public_addr":"172.21.15.98:6801/555491234","cluster_addr":"172.21.15.98:6803/555491234","heartbeat_back_addr":"172.21.15.98:6807/555491234","heartbeat_front_addr":"172.21.15.98:6805/555491234","state":["exists","up"]},{"osd":2,"uuid":"873bdd60-2a47-403b-b1f4-ee9226a95067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6801","nonce":1535856914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6803","nonce":1535856914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6807","nonce":1535856914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6805","nonce":1535856914}]},"public_addr":"172.21.15.113:6801/1535856914","cluster_addr":"172.21.15.113:6803/1535856914","heartbeat_back_addr":"172.21.15.113:6807/1535856914","heartbeat_front_addr":"172.21.15.113:6805/1535856914","state":["exists","up"]},{"osd":3,"uuid":"83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6811","nonce":2949415020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6813","nonce":2949415020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6817","nonce":2949415020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6815","nonce":2949415020}]},"public_addr":"172.21.15.23:6811/2949415020","cluster_addr":"172.21.15.23:6813/2949415020","heartbeat_back_addr":"172.21.15.23:6817/2949415020","heartbeat_front_addr":"172.21.15.23:6815/2949415020","state":["exists","up"]},{"osd":4,"uuid":"76a90bac-b221-40a3-8238-40e8049cec36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6809","nonce":2615902819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6811","nonce":2615902819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6815","nonce":2615902819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6813","nonce":2615902819}]},"public_addr":"172.21.15.98:6809/2615902819","cluster_addr":"172.21.15.98:6811/2615902819","heartbeat_back_addr":"172.21.15.98:6815/2615902819","heartbeat_front_addr":"172.21.15.98:6813/2615902819","state":["exists","up"]},{"osd":5,"uuid":"b53fdb55-a8df-40a8-bd1b-8bcef634e26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6809","nonce":625271528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6811","nonce":625271528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6815","nonce":625271528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6813","nonce":625271528}]},"public_addr":"172.21.15.113:6809/625271528","cluster_addr":"172.21.15.113:6811/625271528","heartbeat_back_addr":"172.21.15.113:6815/625271528","heartbeat_front_addr":"172.21.15.113:6813/625271528","state":["exists","up"]},{"osd":6,"uuid":"f00283d2-8d67-4e97-a575-9a9d3b55d5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6819","nonce":4188464176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6821","nonce":4188464176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6825","nonce":4188464176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6823","nonce":4188464176}]},"public_addr":"172.21.15.23:6819/4188464176","cluster_addr":"172.21.15.23:6821/4188464176","heartbeat_back_addr":"172.21.15.23:6825/4188464176","heartbeat_front_addr":"172.21.15.23:6823/4188464176","state":["exists","up"]},{"osd":7,"uuid":"edc008a7-b07b-40a8-88b7-1614ba38a775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6817","nonce":1985546028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6819","nonce":1985546028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6823","nonce":1985546028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6821","nonce":1985546028}]},"public_addr":"172.21.15.98:6817/1985546028","cluster_addr":"172.21.15.98:6819/1985546028","heartbeat_back_addr":"172.21.15.98:6823/1985546028","heartbeat_front_addr":"172.21.15.98:6821/1985546028","state":["exists","up"]},{"osd":8,"uuid":"56e7b2d1-81f0-47b2-8638-f29832bed8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6817","nonce":3534876081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6819","nonce":3534876081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6823","nonce":3534876081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6821","nonce":3534876081}]},"public_addr":"172.21.15.113:6817/3534876081","cluster_addr":"172.21.15.113:6819/3534876081","heartbeat_back_addr":"172.21.15.113:6823/3534876081","heartbeat_front_addr":"172.21.15.113:6821/3534876081","state":["exists","up"]},{"osd":9,"uuid":"a2e90caa-6e9c-42bf-9f18-81f7f47880cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6827","nonce":672264207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6829","nonce":672264207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6833","nonce":672264207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6831","nonce":672264207}]},"public_addr":"172.21.15.23:6827/672264207","cluster_addr":"172.21.15.23:6829/672264207","heartbeat_back_addr":"172.21.15.23:6833/672264207","heartbeat_front_addr":"172.21.15.23:6831/672264207","state":["exists","up"]},{"osd":10,"uuid":"0eda917f-020c-437f-95ae-b26203f6c68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6825","nonce":2699920595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6827","nonce":2699920595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6831","nonce":2699920595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6829","nonce":2699920595}]},"public_addr":"172.21.15.98:6825/2699920595","cluster_addr":"172.21.15.98:6827/2699920595","heartbeat_back_addr":"172.21.15.98:6831/2699920595","heartbeat_front_addr":"172.21.15.98:6829/2699920595","state":["exists","up"]},{"osd":11,"uuid":"aae78501-78a8-469a-be20-3006f5ec5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6825","nonce":2113644020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6827","nonce":2113644020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6831","nonce":2113644020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6829","nonce":2113644020}]},"public_addr":"172.21.15.113:6825/2113644020","cluster_addr":"172.21.15.113:6827/2113644020","heartbeat_back_addr":"172.21.15.113:6831/2113644020","heartbeat_front_addr":"172.21.15.113:6829/2113644020","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:15:43.527475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:07.006538+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:30.594176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:55.810999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:20.595811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:45.474138+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:10.500409+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:35.787638+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:01.001188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:26.761115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:52.952752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:20:18.559832+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":7}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:6800/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1876451381":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2253559862":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2653562762":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/3337463830":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6800/1735221490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/2784771635":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2389902490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6801/1735221490":"2024-03-29T05:13:42.579607+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T05:22:54.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.103+0000 7fa62ea12640 1 -- 172.21.15.23:0/3350082910 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa6100a2800 msgr2=0x7fa6100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:54.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.103+0000 7fa62ea12640 1 --2- 172.21.15.23:0/3350082910 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa6100a2800 0x7fa6100a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa628101690 tx=0x7fa61c005f90 comp rx=0 tx=0).stop 2024-03-28T05:22:54.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.103+0000 7fa62ea12640 1 -- 172.21.15.23:0/3350082910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa628101830 msgr2=0x7fa628196de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:54.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.103+0000 7fa62ea12640 1 --2- 172.21.15.23:0/3350082910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa628101830 0x7fa628196de0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa618009230 tx=0x7fa618005d50 comp rx=0 tx=0).stop 2024-03-28T05:22:54.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.103+0000 7fa62ea12640 1 -- 172.21.15.23:0/3350082910 shutdown_connections 2024-03-28T05:22:54.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.103+0000 7fa62ea12640 1 --2- 172.21.15.23:0/3350082910 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa6100a2800 0x7fa6100a4cc0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.103+0000 7fa62ea12640 1 --2- 172.21.15.23:0/3350082910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6281021f0 0x7fa62819de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.103+0000 7fa62ea12640 1 --2- 172.21.15.23:0/3350082910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa628101830 0x7fa628196de0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.103+0000 7fa62ea12640 1 --2- 172.21.15.23:0/3350082910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa628100640 0x7fa6281968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.103+0000 7fa62ea12640 1 -- 172.21.15.23:0/3350082910 >> 172.21.15.23:0/3350082910 conn(0x7fa6280fbbd0 msgr2=0x7fa628104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:54.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.103+0000 7fa62ea12640 1 -- 172.21.15.23:0/3350082910 shutdown_connections 2024-03-28T05:22:54.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.103+0000 7fa62ea12640 1 -- 172.21.15.23:0/3350082910 wait complete. 2024-03-28T05:22:54.113 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T05:16:32.986974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T05:21:06.103894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T05:21:06.827190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T05:21:10.507418+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T05:22:54.113 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T05:22:54.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.221+0000 7f15a42b2640 1 -- 172.21.15.23:0/3661022665 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f159c101790 msgr2=0x7f159c101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:54.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.221+0000 7f15a42b2640 1 --2- 172.21.15.23:0/3661022665 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f159c101790 0x7f159c101c10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f158c009900 tx=0x7f158c02bc30 comp rx=0 tx=0).stop 2024-03-28T05:22:54.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.222+0000 7f15a42b2640 1 -- 172.21.15.23:0/3661022665 shutdown_connections 2024-03-28T05:22:54.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.222+0000 7f15a42b2640 1 --2- 172.21.15.23:0/3661022665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f159c102150 0x7f159c1089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.222+0000 7f15a42b2640 1 --2- 172.21.15.23:0/3661022665 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f159c101790 0x7f159c101c10 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.222+0000 7f15a42b2640 1 --2- 172.21.15.23:0/3661022665 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f159c1005a0 0x7f159c1009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.222+0000 7f15a42b2640 1 -- 172.21.15.23:0/3661022665 >> 172.21.15.23:0/3661022665 conn(0x7f159c0fbb30 msgr2=0x7f159c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:54.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.222+0000 7f15a42b2640 1 -- 172.21.15.23:0/3661022665 shutdown_connections 2024-03-28T05:22:54.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.222+0000 7f15a42b2640 1 -- 172.21.15.23:0/3661022665 wait complete. 2024-03-28T05:22:54.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.222+0000 7f15a42b2640 1 Processor -- start 2024-03-28T05:22:54.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.222+0000 7f15a42b2640 1 -- start start 2024-03-28T05:22:54.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.222+0000 7f15a42b2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f159c1005a0 0x7f159c104530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:54.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.222+0000 7f15a42b2640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f159c101790 0x7f159c102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:54.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.222+0000 7f15a42b2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f159c102150 0x7f159c1030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:54.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.222+0000 7f15a42b2640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159c10c230 con 0x7f159c1005a0 2024-03-28T05:22:54.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.222+0000 7f15a42b2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159c10c0b0 con 0x7f159c102150 2024-03-28T05:22:54.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.222+0000 7f15a42b2640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159c10c3b0 con 0x7f159c101790 2024-03-28T05:22:54.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.222+0000 7f15a204e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f159c1005a0 0x7f159c104530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:54.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.222+0000 7f15a204e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f159c1005a0 0x7f159c104530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34206/0 (socket says 172.21.15.23:34206) 2024-03-28T05:22:54.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.222+0000 7f15a204e640 1 -- 172.21.15.23:0/1658221771 learned_addr learned my addr 172.21.15.23:0/1658221771 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:54.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.222+0000 7f15a184d640 1 --2- 172.21.15.23:0/1658221771 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f159c101790 0x7f159c102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:54.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.222+0000 7f15a284f640 1 --2- 172.21.15.23:0/1658221771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f159c102150 0x7f159c1030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:54.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.223+0000 7f15a204e640 1 -- 172.21.15.23:0/1658221771 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f159c101790 msgr2=0x7f159c102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:54.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.223+0000 7f15a204e640 1 --2- 172.21.15.23:0/1658221771 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f159c101790 0x7f159c102b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.223+0000 7f15a204e640 1 -- 172.21.15.23:0/1658221771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f159c102150 msgr2=0x7f159c1030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:54.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.223+0000 7f15a204e640 1 --2- 172.21.15.23:0/1658221771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f159c102150 0x7f159c1030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.223+0000 7f15a204e640 1 -- 172.21.15.23:0/1658221771 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159c103940 con 0x7f159c1005a0 2024-03-28T05:22:54.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.223+0000 7f15a204e640 1 --2- 172.21.15.23:0/1658221771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f159c1005a0 0x7f159c104530 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f159800e9c0 tx=0x7f159800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:54.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.223+0000 7f15937fe640 1 -- 172.21.15.23:0/1658221771 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159800cd80 con 0x7f159c1005a0 2024-03-28T05:22:54.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.223+0000 7f15937fe640 1 -- 172.21.15.23:0/1658221771 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15980035f0 con 0x7f159c1005a0 2024-03-28T05:22:54.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.223+0000 7f15a42b2640 1 -- 172.21.15.23:0/1658221771 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f159c1a1a60 con 0x7f159c1005a0 2024-03-28T05:22:54.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.223+0000 7f15937fe640 1 -- 172.21.15.23:0/1658221771 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1598010690 con 0x7f159c1005a0 2024-03-28T05:22:54.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.223+0000 7f15a42b2640 1 -- 172.21.15.23:0/1658221771 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f159c1a1fd0 con 0x7f159c1005a0 2024-03-28T05:22:54.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.224+0000 7f15a42b2640 1 -- 172.21.15.23:0/1658221771 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f159c1a2260 con 0x7f159c1005a0 2024-03-28T05:22:54.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.225+0000 7f15937fe640 1 -- 172.21.15.23:0/1658221771 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f15980037a0 con 0x7f159c1005a0 2024-03-28T05:22:54.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.226+0000 7f15937fe640 1 --2- 172.21.15.23:0/1658221771 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f15880a2880 0x7f15880a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:54.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.226+0000 7f15a184d640 1 --2- 172.21.15.23:0/1658221771 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f15880a2880 0x7f15880a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:54.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.226+0000 7f15937fe640 1 -- 172.21.15.23:0/1658221771 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f15980cf150 con 0x7f159c1005a0 2024-03-28T05:22:54.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.226+0000 7f15a184d640 1 --2- 172.21.15.23:0/1658221771 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f15880a2880 0x7f15880a4d40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f158c02c7b0 tx=0x7f158c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:54.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.230+0000 7f15937fe640 1 -- 172.21.15.23:0/1658221771 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f159809be90 con 0x7f159c1005a0 2024-03-28T05:22:54.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.414+0000 7f15a42b2640 1 -- 172.21.15.23:0/1658221771 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f159c1a2430 con 0x7f159c1005a0 2024-03-28T05:22:54.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.414+0000 7f15937fe640 1 -- 172.21.15.23:0/1658221771 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f159809b790 con 0x7f159c1005a0 2024-03-28T05:22:54.413 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-28T05:22:54.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.416+0000 7f15a42b2640 1 -- 172.21.15.23:0/1658221771 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f15880a2880 msgr2=0x7f15880a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:54.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.416+0000 7f15a42b2640 1 --2- 172.21.15.23:0/1658221771 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f15880a2880 0x7f15880a4d40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f158c02c7b0 tx=0x7f158c005fd0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.416+0000 7f15a42b2640 1 -- 172.21.15.23:0/1658221771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f159c1005a0 msgr2=0x7f159c104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:54.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.416+0000 7f15a42b2640 1 --2- 172.21.15.23:0/1658221771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f159c1005a0 0x7f159c104530 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f159800e9c0 tx=0x7f159800ee80 comp rx=0 tx=0).stop 2024-03-28T05:22:54.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.416+0000 7f15a42b2640 1 -- 172.21.15.23:0/1658221771 shutdown_connections 2024-03-28T05:22:54.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.416+0000 7f15a42b2640 1 --2- 172.21.15.23:0/1658221771 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f15880a2880 0x7f15880a4d40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.416+0000 7f15a42b2640 1 --2- 172.21.15.23:0/1658221771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f159c102150 0x7f159c1030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.416+0000 7f15a42b2640 1 --2- 172.21.15.23:0/1658221771 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f159c101790 0x7f159c102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.416+0000 7f15a42b2640 1 --2- 172.21.15.23:0/1658221771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f159c1005a0 0x7f159c104530 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.416+0000 7f15a42b2640 1 -- 172.21.15.23:0/1658221771 >> 172.21.15.23:0/1658221771 conn(0x7f159c0fbb30 msgr2=0x7f159c0fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:54.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.416+0000 7f15a42b2640 1 -- 172.21.15.23:0/1658221771 shutdown_connections 2024-03-28T05:22:54.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.416+0000 7f15a42b2640 1 -- 172.21.15.23:0/1658221771 wait complete. 2024-03-28T05:22:54.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:54 smithi023 ceph-mon[31954]: pgmap v343: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-03-28T05:22:54.425 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:54 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/859764330' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:22:54.425 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:54 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3206504573' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:22:54.425 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:54 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3350082910' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:22:54.426 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T05:22:54.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:54 smithi113 ceph-mon[37164]: pgmap v343: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-03-28T05:22:54.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:54 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/859764330' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:22:54.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:54 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3206504573' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:22:54.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:54 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3350082910' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:22:54.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.533+0000 7f4e85c43640 1 -- 172.21.15.23:0/3840426080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e80103260 msgr2=0x7f4e80109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:54.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.533+0000 7f4e85c43640 1 --2- 172.21.15.23:0/3840426080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e80103260 0x7f4e80109af0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4e78009900 tx=0x7f4e7802bc30 comp rx=0 tx=0).stop 2024-03-28T05:22:54.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.533+0000 7f4e85c43640 1 -- 172.21.15.23:0/3840426080 shutdown_connections 2024-03-28T05:22:54.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.533+0000 7f4e85c43640 1 --2- 172.21.15.23:0/3840426080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e80103260 0x7f4e80109af0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.533+0000 7f4e85c43640 1 --2- 172.21.15.23:0/3840426080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e801028a0 0x7f4e80102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.533+0000 7f4e85c43640 1 --2- 172.21.15.23:0/3840426080 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e801016b0 0x7f4e80101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.533+0000 7f4e85c43640 1 -- 172.21.15.23:0/3840426080 >> 172.21.15.23:0/3840426080 conn(0x7f4e800fcc40 msgr2=0x7f4e800ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:54.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.533+0000 7f4e85c43640 1 -- 172.21.15.23:0/3840426080 shutdown_connections 2024-03-28T05:22:54.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.533+0000 7f4e85c43640 1 -- 172.21.15.23:0/3840426080 wait complete. 2024-03-28T05:22:54.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.534+0000 7f4e85c43640 1 Processor -- start 2024-03-28T05:22:54.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.534+0000 7f4e85c43640 1 -- start start 2024-03-28T05:22:54.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.534+0000 7f4e85c43640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e801016b0 0x7f4e801978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.534+0000 7f4e85c43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e801028a0 0x7f4e80197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.534+0000 7f4e85c43640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e80103260 0x7f4e8019ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.534+0000 7f4e85c43640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8010d330 con 0x7f4e80103260 2024-03-28T05:22:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.534+0000 7f4e85c43640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8010d1b0 con 0x7f4e801028a0 2024-03-28T05:22:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.534+0000 7f4e85c43640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8010d4b0 con 0x7f4e801016b0 2024-03-28T05:22:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.534+0000 7f4e7ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e80103260 0x7f4e8019ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.534+0000 7f4e7ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e80103260 0x7f4e8019ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34216/0 (socket says 172.21.15.23:34216) 2024-03-28T05:22:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.534+0000 7f4e7ffff640 1 -- 172.21.15.23:0/3268872044 learned_addr learned my addr 172.21.15.23:0/3268872044 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.534+0000 7f4e7f7fe640 1 --2- 172.21.15.23:0/3268872044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e801016b0 0x7f4e801978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.535+0000 7f4e7effd640 1 --2- 172.21.15.23:0/3268872044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e801028a0 0x7f4e80197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.535+0000 7f4e7ffff640 1 -- 172.21.15.23:0/3268872044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e801016b0 msgr2=0x7f4e801978e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.535+0000 7f4e7ffff640 1 --2- 172.21.15.23:0/3268872044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e801016b0 0x7f4e801978e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.535+0000 7f4e7ffff640 1 -- 172.21.15.23:0/3268872044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e801028a0 msgr2=0x7f4e80197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.535+0000 7f4e7ffff640 1 --2- 172.21.15.23:0/3268872044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e801028a0 0x7f4e80197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.535+0000 7f4e7ffff640 1 -- 172.21.15.23:0/3268872044 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8019f500 con 0x7f4e80103260 2024-03-28T05:22:54.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.535+0000 7f4e7f7fe640 1 --2- 172.21.15.23:0/3268872044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e801016b0 0x7f4e801978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:22:54.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.535+0000 7f4e7effd640 1 --2- 172.21.15.23:0/3268872044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e801028a0 0x7f4e80197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:22:54.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.535+0000 7f4e7ffff640 1 --2- 172.21.15.23:0/3268872044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e80103260 0x7f4e8019ee90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4e7802c7b0 tx=0x7f4e78005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:54.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.535+0000 7f4e7cff9640 1 -- 172.21.15.23:0/3268872044 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e78041070 con 0x7f4e80103260 2024-03-28T05:22:54.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.535+0000 7f4e7cff9640 1 -- 172.21.15.23:0/3268872044 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e780054e0 con 0x7f4e80103260 2024-03-28T05:22:54.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.535+0000 7f4e85c43640 1 -- 172.21.15.23:0/3268872044 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e8019f790 con 0x7f4e80103260 2024-03-28T05:22:54.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.535+0000 7f4e7cff9640 1 -- 172.21.15.23:0/3268872044 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e78034440 con 0x7f4e80103260 2024-03-28T05:22:54.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.535+0000 7f4e85c43640 1 -- 172.21.15.23:0/3268872044 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e8019fc80 con 0x7f4e80103260 2024-03-28T05:22:54.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.536+0000 7f4e85c43640 1 -- 172.21.15.23:0/3268872044 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e80108970 con 0x7f4e80103260 2024-03-28T05:22:54.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.537+0000 7f4e7cff9640 1 -- 172.21.15.23:0/3268872044 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4e78044070 con 0x7f4e80103260 2024-03-28T05:22:54.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.537+0000 7f4e7cff9640 1 --2- 172.21.15.23:0/3268872044 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f4e680a2800 0x7f4e680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:54.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.538+0000 7f4e7f7fe640 1 --2- 172.21.15.23:0/3268872044 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f4e680a2800 0x7f4e680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:54.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.538+0000 7f4e7cff9640 1 -- 172.21.15.23:0/3268872044 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f4e780f0ac0 con 0x7f4e80103260 2024-03-28T05:22:54.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.538+0000 7f4e7f7fe640 1 --2- 172.21.15.23:0/3268872044 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f4e680a2800 0x7f4e680a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4e80102700 tx=0x7f4e70009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:54.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.542+0000 7f4e7cff9640 1 -- 172.21.15.23:0/3268872044 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4e780bd800 con 0x7f4e80103260 2024-03-28T05:22:54.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:54 smithi098 ceph-mon[36087]: pgmap v343: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-03-28T05:22:54.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:54 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/859764330' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:22:54.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:54 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3206504573' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:22:54.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:54 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3350082910' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:22:54.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.725+0000 7f4e85c43640 1 -- 172.21.15.23:0/3268872044 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4e800622a0 con 0x7f4e80103260 2024-03-28T05:22:54.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.725+0000 7f4e7cff9640 1 -- 172.21.15.23:0/3268872044 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4e780bd100 con 0x7f4e80103260 2024-03-28T05:22:54.724 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T05:22:54.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.727+0000 7f4e85c43640 1 -- 172.21.15.23:0/3268872044 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f4e680a2800 msgr2=0x7f4e680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:54.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.727+0000 7f4e85c43640 1 --2- 172.21.15.23:0/3268872044 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f4e680a2800 0x7f4e680a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4e80102700 tx=0x7f4e70009500 comp rx=0 tx=0).stop 2024-03-28T05:22:54.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.727+0000 7f4e85c43640 1 -- 172.21.15.23:0/3268872044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e80103260 msgr2=0x7f4e8019ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:54.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.727+0000 7f4e85c43640 1 --2- 172.21.15.23:0/3268872044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e80103260 0x7f4e8019ee90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4e7802c7b0 tx=0x7f4e78005dd0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.727+0000 7f4e85c43640 1 -- 172.21.15.23:0/3268872044 shutdown_connections 2024-03-28T05:22:54.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.727+0000 7f4e85c43640 1 --2- 172.21.15.23:0/3268872044 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f4e680a2800 0x7f4e680a4cc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.727+0000 7f4e85c43640 1 --2- 172.21.15.23:0/3268872044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e80103260 0x7f4e8019ee90 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.727+0000 7f4e85c43640 1 --2- 172.21.15.23:0/3268872044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e801028a0 0x7f4e80197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.727+0000 7f4e85c43640 1 --2- 172.21.15.23:0/3268872044 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e801016b0 0x7f4e801978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.727+0000 7f4e85c43640 1 -- 172.21.15.23:0/3268872044 >> 172.21.15.23:0/3268872044 conn(0x7f4e800fcc40 msgr2=0x7f4e80105b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:54.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.728+0000 7f4e85c43640 1 -- 172.21.15.23:0/3268872044 shutdown_connections 2024-03-28T05:22:54.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.728+0000 7f4e85c43640 1 -- 172.21.15.23:0/3268872044 wait complete. 2024-03-28T05:22:54.737 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T05:22:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.844+0000 7fcfe771b640 1 -- 172.21.15.23:0/127917196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe01028b0 msgr2=0x7fcfe0102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.844+0000 7fcfe771b640 1 --2- 172.21.15.23:0/127917196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe01028b0 0x7fcfe0102d30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4009900 tx=0x7fcfd402bc30 comp rx=0 tx=0).stop 2024-03-28T05:22:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.844+0000 7fcfe771b640 1 -- 172.21.15.23:0/127917196 shutdown_connections 2024-03-28T05:22:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.844+0000 7fcfe771b640 1 --2- 172.21.15.23:0/127917196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe0103270 0x7fcfe0109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.844+0000 7fcfe771b640 1 --2- 172.21.15.23:0/127917196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe01028b0 0x7fcfe0102d30 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.844+0000 7fcfe771b640 1 --2- 172.21.15.23:0/127917196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfe01016c0 0x7fcfe0101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.844+0000 7fcfe771b640 1 -- 172.21.15.23:0/127917196 >> 172.21.15.23:0/127917196 conn(0x7fcfe00fcc30 msgr2=0x7fcfe00ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.844+0000 7fcfe771b640 1 -- 172.21.15.23:0/127917196 shutdown_connections 2024-03-28T05:22:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.844+0000 7fcfe771b640 1 -- 172.21.15.23:0/127917196 wait complete. 2024-03-28T05:22:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.845+0000 7fcfe771b640 1 Processor -- start 2024-03-28T05:22:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.845+0000 7fcfe771b640 1 -- start start 2024-03-28T05:22:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.845+0000 7fcfe771b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe01016c0 0x7fcfe01979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.845+0000 7fcfe771b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfe01028b0 0x7fcfe0197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.845+0000 7fcfe771b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe0103270 0x7fcfe019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.845+0000 7fcfe771b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe010d340 con 0x7fcfe01016c0 2024-03-28T05:22:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.845+0000 7fcfe771b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe010d1c0 con 0x7fcfe0103270 2024-03-28T05:22:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.845+0000 7fcfe771b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe010d4c0 con 0x7fcfe01028b0 2024-03-28T05:22:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.845+0000 7fcfe54b7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe01016c0 0x7fcfe01979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.845+0000 7fcfe54b7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe01016c0 0x7fcfe01979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34240/0 (socket says 172.21.15.23:34240) 2024-03-28T05:22:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.845+0000 7fcfe54b7640 1 -- 172.21.15.23:0/2882481173 learned_addr learned my addr 172.21.15.23:0/2882481173 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.845+0000 7fcfe4cb6640 1 --2- 172.21.15.23:0/2882481173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfe01028b0 0x7fcfe0197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.845+0000 7fcfe54b7640 1 -- 172.21.15.23:0/2882481173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfe01028b0 msgr2=0x7fcfe0197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.845+0000 7fcfe54b7640 1 --2- 172.21.15.23:0/2882481173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfe01028b0 0x7fcfe0197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.845+0000 7fcfe54b7640 1 -- 172.21.15.23:0/2882481173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe0103270 msgr2=0x7fcfe019ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.845+0000 7fcfe5cb8640 1 --2- 172.21.15.23:0/2882481173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe0103270 0x7fcfe019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.845+0000 7fcfe54b7640 1 --2- 172.21.15.23:0/2882481173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe0103270 0x7fcfe019ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.845+0000 7fcfe54b7640 1 -- 172.21.15.23:0/2882481173 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe019f600 con 0x7fcfe01016c0 2024-03-28T05:22:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.846+0000 7fcfe5cb8640 1 --2- 172.21.15.23:0/2882481173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe0103270 0x7fcfe019ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:22:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.846+0000 7fcfe54b7640 1 --2- 172.21.15.23:0/2882481173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe01016c0 0x7fcfe01979e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800e9c0 tx=0x7fcfd800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.846+0000 7fcfde7fc640 1 -- 172.21.15.23:0/2882481173 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd800cd80 con 0x7fcfe01016c0 2024-03-28T05:22:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.846+0000 7fcfde7fc640 1 -- 172.21.15.23:0/2882481173 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfd80035f0 con 0x7fcfe01016c0 2024-03-28T05:22:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.846+0000 7fcfe771b640 1 -- 172.21.15.23:0/2882481173 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe019f8f0 con 0x7fcfe01016c0 2024-03-28T05:22:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.846+0000 7fcfde7fc640 1 -- 172.21.15.23:0/2882481173 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8010690 con 0x7fcfe01016c0 2024-03-28T05:22:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.846+0000 7fcfe771b640 1 -- 172.21.15.23:0/2882481173 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe019fe10 con 0x7fcfe01016c0 2024-03-28T05:22:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.847+0000 7fcfe771b640 1 -- 172.21.15.23:0/2882481173 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe0108ab0 con 0x7fcfe01016c0 2024-03-28T05:22:54.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.848+0000 7fcfde7fc640 1 -- 172.21.15.23:0/2882481173 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcfd80037a0 con 0x7fcfe01016c0 2024-03-28T05:22:54.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.849+0000 7fcfde7fc640 1 --2- 172.21.15.23:0/2882481173 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fcfc80a2880 0x7fcfc80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:54.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.849+0000 7fcfe4cb6640 1 --2- 172.21.15.23:0/2882481173 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fcfc80a2880 0x7fcfc80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:54.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.849+0000 7fcfde7fc640 1 -- 172.21.15.23:0/2882481173 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fcfd80cf1d0 con 0x7fcfe01016c0 2024-03-28T05:22:54.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.850+0000 7fcfe4cb6640 1 --2- 172.21.15.23:0/2882481173 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fcfc80a2880 0x7fcfc80a4d40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcfd402c7b0 tx=0x7fcfd4005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:54.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:54.853+0000 7fcfde7fc640 1 -- 172.21.15.23:0/2882481173 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcfd809bf10 con 0x7fcfe01016c0 2024-03-28T05:22:55.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.036+0000 7fcfe771b640 1 -- 172.21.15.23:0/2882481173 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcfe002d320 con 0x7fcfe01016c0 2024-03-28T05:22:55.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.037+0000 7fcfde7fc640 1 -- 172.21.15.23:0/2882481173 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcfd809b810 con 0x7fcfe01016c0 2024-03-28T05:22:55.035 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-28T05:22:55.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.039+0000 7fcfe771b640 1 -- 172.21.15.23:0/2882481173 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fcfc80a2880 msgr2=0x7fcfc80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:55.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.039+0000 7fcfe771b640 1 --2- 172.21.15.23:0/2882481173 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fcfc80a2880 0x7fcfc80a4d40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcfd402c7b0 tx=0x7fcfd4005fd0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.039+0000 7fcfe771b640 1 -- 172.21.15.23:0/2882481173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe01016c0 msgr2=0x7fcfe01979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:55.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.039+0000 7fcfe771b640 1 --2- 172.21.15.23:0/2882481173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe01016c0 0x7fcfe01979e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800e9c0 tx=0x7fcfd800ee80 comp rx=0 tx=0).stop 2024-03-28T05:22:55.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.039+0000 7fcfe771b640 1 -- 172.21.15.23:0/2882481173 shutdown_connections 2024-03-28T05:22:55.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.039+0000 7fcfe771b640 1 --2- 172.21.15.23:0/2882481173 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fcfc80a2880 0x7fcfc80a4d40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.039+0000 7fcfe771b640 1 --2- 172.21.15.23:0/2882481173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfe0103270 0x7fcfe019ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.039+0000 7fcfe771b640 1 --2- 172.21.15.23:0/2882481173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfe01028b0 0x7fcfe0197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.039+0000 7fcfe771b640 1 --2- 172.21.15.23:0/2882481173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe01016c0 0x7fcfe01979e0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.039+0000 7fcfe771b640 1 -- 172.21.15.23:0/2882481173 >> 172.21.15.23:0/2882481173 conn(0x7fcfe00fcc30 msgr2=0x7fcfe0105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:55.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.039+0000 7fcfe771b640 1 -- 172.21.15.23:0/2882481173 shutdown_connections 2024-03-28T05:22:55.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.039+0000 7fcfe771b640 1 -- 172.21.15.23:0/2882481173 wait complete. 2024-03-28T05:22:55.048 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T05:22:55.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.155+0000 7f307b2f8640 1 -- 172.21.15.23:0/4141252024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3074103260 msgr2=0x7f3074109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:55.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.155+0000 7f307b2f8640 1 --2- 172.21.15.23:0/4141252024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3074103260 0x7f3074109af0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f306c009ca0 tx=0x7f306c02c0f0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.155+0000 7f307b2f8640 1 -- 172.21.15.23:0/4141252024 shutdown_connections 2024-03-28T05:22:55.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.155+0000 7f307b2f8640 1 --2- 172.21.15.23:0/4141252024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3074103260 0x7f3074109af0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.155+0000 7f307b2f8640 1 --2- 172.21.15.23:0/4141252024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30741028a0 0x7f3074102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.155+0000 7f307b2f8640 1 --2- 172.21.15.23:0/4141252024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30741016b0 0x7f3074101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.155+0000 7f307b2f8640 1 -- 172.21.15.23:0/4141252024 >> 172.21.15.23:0/4141252024 conn(0x7f30740fcc40 msgr2=0x7f30740ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:55.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.155+0000 7f307b2f8640 1 -- 172.21.15.23:0/4141252024 shutdown_connections 2024-03-28T05:22:55.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.155+0000 7f307b2f8640 1 -- 172.21.15.23:0/4141252024 wait complete. 2024-03-28T05:22:55.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.155+0000 7f307b2f8640 1 Processor -- start 2024-03-28T05:22:55.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.155+0000 7f307b2f8640 1 -- start start 2024-03-28T05:22:55.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.156+0000 7f307b2f8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30741016b0 0x7f3074197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:55.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.156+0000 7f307b2f8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30741028a0 0x7f3074197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:55.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.156+0000 7f307b2f8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3074103260 0x7f307419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:55.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.156+0000 7f307b2f8640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307410d330 con 0x7f30741016b0 2024-03-28T05:22:55.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.156+0000 7f307b2f8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307410d1b0 con 0x7f30741028a0 2024-03-28T05:22:55.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.156+0000 7f307b2f8640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307410d4b0 con 0x7f3074103260 2024-03-28T05:22:55.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.156+0000 7f3079094640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30741016b0 0x7f3074197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:55.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.156+0000 7f3079094640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30741016b0 0x7f3074197940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34262/0 (socket says 172.21.15.23:34262) 2024-03-28T05:22:55.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.156+0000 7f3079094640 1 -- 172.21.15.23:0/2992721831 learned_addr learned my addr 172.21.15.23:0/2992721831 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:55.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.156+0000 7f3079895640 1 --2- 172.21.15.23:0/2992721831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3074103260 0x7f307419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:55.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.156+0000 7f3078893640 1 --2- 172.21.15.23:0/2992721831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30741028a0 0x7f3074197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:55.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.156+0000 7f3079094640 1 -- 172.21.15.23:0/2992721831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3074103260 msgr2=0x7f307419eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:55.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.156+0000 7f3079094640 1 --2- 172.21.15.23:0/2992721831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3074103260 0x7f307419eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.156+0000 7f3079094640 1 -- 172.21.15.23:0/2992721831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30741028a0 msgr2=0x7f3074197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:55.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.156+0000 7f3079094640 1 --2- 172.21.15.23:0/2992721831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30741028a0 0x7f3074197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.156+0000 7f3079094640 1 -- 172.21.15.23:0/2992721831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307419f560 con 0x7f30741016b0 2024-03-28T05:22:55.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.156+0000 7f3079094640 1 --2- 172.21.15.23:0/2992721831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30741016b0 0x7f3074197940 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f307000e9c0 tx=0x7f307000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:55.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.156+0000 7f30667fc640 1 -- 172.21.15.23:0/2992721831 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307000cd80 con 0x7f30741016b0 2024-03-28T05:22:55.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.156+0000 7f30667fc640 1 -- 172.21.15.23:0/2992721831 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30700035f0 con 0x7f30741016b0 2024-03-28T05:22:55.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.156+0000 7f307b2f8640 1 -- 172.21.15.23:0/2992721831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f307419f850 con 0x7f30741016b0 2024-03-28T05:22:55.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.156+0000 7f30667fc640 1 -- 172.21.15.23:0/2992721831 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3070010690 con 0x7f30741016b0 2024-03-28T05:22:55.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.156+0000 7f307b2f8640 1 -- 172.21.15.23:0/2992721831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f307419fda0 con 0x7f30741016b0 2024-03-28T05:22:55.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.157+0000 7f307b2f8640 1 -- 172.21.15.23:0/2992721831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3074108970 con 0x7f30741016b0 2024-03-28T05:22:55.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.159+0000 7f30667fc640 1 -- 172.21.15.23:0/2992721831 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f30700037a0 con 0x7f30741016b0 2024-03-28T05:22:55.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.159+0000 7f30667fc640 1 --2- 172.21.15.23:0/2992721831 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f30600a2880 0x7f30600a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:55.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.159+0000 7f3078893640 1 --2- 172.21.15.23:0/2992721831 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f30600a2880 0x7f30600a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:55.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.159+0000 7f30667fc640 1 -- 172.21.15.23:0/2992721831 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f30700cf220 con 0x7f30741016b0 2024-03-28T05:22:55.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.160+0000 7f3078893640 1 --2- 172.21.15.23:0/2992721831 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f30600a2880 0x7f30600a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3068005e10 tx=0x7f3068005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:55.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.163+0000 7f30667fc640 1 -- 172.21.15.23:0/2992721831 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f307009bf60 con 0x7f30741016b0 2024-03-28T05:22:55.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.347+0000 7f307b2f8640 1 -- 172.21.15.23:0/2992721831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f307402d070 con 0x7f30741016b0 2024-03-28T05:22:55.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.347+0000 7f30667fc640 1 -- 172.21.15.23:0/2992721831 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f307009b860 con 0x7f30741016b0 2024-03-28T05:22:55.346 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T05:22:55.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.349+0000 7f307b2f8640 1 -- 172.21.15.23:0/2992721831 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f30600a2880 msgr2=0x7f30600a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:55.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.349+0000 7f307b2f8640 1 --2- 172.21.15.23:0/2992721831 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f30600a2880 0x7f30600a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3068005e10 tx=0x7f3068005da0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.349+0000 7f307b2f8640 1 -- 172.21.15.23:0/2992721831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30741016b0 msgr2=0x7f3074197940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:55.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.349+0000 7f307b2f8640 1 --2- 172.21.15.23:0/2992721831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30741016b0 0x7f3074197940 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f307000e9c0 tx=0x7f307000ee80 comp rx=0 tx=0).stop 2024-03-28T05:22:55.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.349+0000 7f307b2f8640 1 -- 172.21.15.23:0/2992721831 shutdown_connections 2024-03-28T05:22:55.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.349+0000 7f307b2f8640 1 --2- 172.21.15.23:0/2992721831 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f30600a2880 0x7f30600a4d40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.349+0000 7f307b2f8640 1 --2- 172.21.15.23:0/2992721831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3074103260 0x7f307419eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.349+0000 7f307b2f8640 1 --2- 172.21.15.23:0/2992721831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30741028a0 0x7f3074197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.349+0000 7f307b2f8640 1 --2- 172.21.15.23:0/2992721831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30741016b0 0x7f3074197940 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.349+0000 7f307b2f8640 1 -- 172.21.15.23:0/2992721831 >> 172.21.15.23:0/2992721831 conn(0x7f30740fcc40 msgr2=0x7f3074105b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:55.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.349+0000 7f307b2f8640 1 -- 172.21.15.23:0/2992721831 shutdown_connections 2024-03-28T05:22:55.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.349+0000 7f307b2f8640 1 -- 172.21.15.23:0/2992721831 wait complete. 2024-03-28T05:22:55.358 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:55 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1658221771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:22:55.358 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:55 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3268872044' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:22:55.358 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:55 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2882481173' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:22:55.359 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:22:55.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.463+0000 7f8d6632f640 1 -- 172.21.15.23:0/3371361951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d601016c0 msgr2=0x7f8d60101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:55.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.463+0000 7f8d6632f640 1 --2- 172.21.15.23:0/3371361951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d601016c0 0x7f8d60101ae0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8d50009900 tx=0x7f8d5002bc30 comp rx=0 tx=0).stop 2024-03-28T05:22:55.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.463+0000 7f8d6632f640 1 -- 172.21.15.23:0/3371361951 shutdown_connections 2024-03-28T05:22:55.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.463+0000 7f8d6632f640 1 --2- 172.21.15.23:0/3371361951 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d60103270 0x7f8d60109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.463+0000 7f8d6632f640 1 --2- 172.21.15.23:0/3371361951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d601028b0 0x7f8d60102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.463+0000 7f8d6632f640 1 --2- 172.21.15.23:0/3371361951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d601016c0 0x7f8d60101ae0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.463+0000 7f8d6632f640 1 -- 172.21.15.23:0/3371361951 >> 172.21.15.23:0/3371361951 conn(0x7f8d600fcc50 msgr2=0x7f8d600ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:55.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.463+0000 7f8d6632f640 1 -- 172.21.15.23:0/3371361951 shutdown_connections 2024-03-28T05:22:55.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.463+0000 7f8d6632f640 1 -- 172.21.15.23:0/3371361951 wait complete. 2024-03-28T05:22:55.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.463+0000 7f8d6632f640 1 Processor -- start 2024-03-28T05:22:55.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.463+0000 7f8d6632f640 1 -- start start 2024-03-28T05:22:55.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.464+0000 7f8d6632f640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d601016c0 0x7f8d60075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:55.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.464+0000 7f8d6632f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d601028b0 0x7f8d60073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:55.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.464+0000 7f8d6632f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60103270 0x7f8d60073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:55.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.464+0000 7f8d6632f640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6010d310 con 0x7f8d60103270 2024-03-28T05:22:55.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.464+0000 7f8d6632f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6010d190 con 0x7f8d601028b0 2024-03-28T05:22:55.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.464+0000 7f8d6632f640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6010d490 con 0x7f8d601016c0 2024-03-28T05:22:55.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.464+0000 7f8d648cc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60103270 0x7f8d60073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:55.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.464+0000 7f8d648cc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60103270 0x7f8d60073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34280/0 (socket says 172.21.15.23:34280) 2024-03-28T05:22:55.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.464+0000 7f8d648cc640 1 -- 172.21.15.23:0/331071033 learned_addr learned my addr 172.21.15.23:0/331071033 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:55.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.464+0000 7f8d5ffff640 1 --2- 172.21.15.23:0/331071033 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d601016c0 0x7f8d60075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:55.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.464+0000 7f8d5f7fe640 1 --2- 172.21.15.23:0/331071033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d601028b0 0x7f8d60073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:55.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.464+0000 7f8d648cc640 1 -- 172.21.15.23:0/331071033 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d601016c0 msgr2=0x7f8d60075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:55.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.464+0000 7f8d648cc640 1 --2- 172.21.15.23:0/331071033 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d601016c0 0x7f8d60075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.464+0000 7f8d648cc640 1 -- 172.21.15.23:0/331071033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d601028b0 msgr2=0x7f8d60073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:55.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.464+0000 7f8d648cc640 1 --2- 172.21.15.23:0/331071033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d601028b0 0x7f8d60073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.464+0000 7f8d648cc640 1 -- 172.21.15.23:0/331071033 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d60074860 con 0x7f8d60103270 2024-03-28T05:22:55.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.464+0000 7f8d648cc640 1 --2- 172.21.15.23:0/331071033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60103270 0x7f8d60073fd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8d5800eb10 tx=0x7f8d5800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:55.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.465+0000 7f8d5d7fa640 1 -- 172.21.15.23:0/331071033 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d58004b20 con 0x7f8d60103270 2024-03-28T05:22:55.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.465+0000 7f8d5d7fa640 1 -- 172.21.15.23:0/331071033 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d58004cc0 con 0x7f8d60103270 2024-03-28T05:22:55.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.465+0000 7f8d6632f640 1 -- 172.21.15.23:0/331071033 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d601a1ae0 con 0x7f8d60103270 2024-03-28T05:22:55.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.465+0000 7f8d5d7fa640 1 -- 172.21.15.23:0/331071033 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d58010760 con 0x7f8d60103270 2024-03-28T05:22:55.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.465+0000 7f8d6632f640 1 -- 172.21.15.23:0/331071033 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d601a1fe0 con 0x7f8d60103270 2024-03-28T05:22:55.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.466+0000 7f8d6632f640 1 -- 172.21.15.23:0/331071033 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d60108a00 con 0x7f8d60103270 2024-03-28T05:22:55.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.467+0000 7f8d5d7fa640 1 -- 172.21.15.23:0/331071033 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8d58005020 con 0x7f8d60103270 2024-03-28T05:22:55.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.467+0000 7f8d5d7fa640 1 --2- 172.21.15.23:0/331071033 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8d480a2800 0x7f8d480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:55.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.467+0000 7f8d5ffff640 1 --2- 172.21.15.23:0/331071033 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8d480a2800 0x7f8d480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:55.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.467+0000 7f8d5d7fa640 1 -- 172.21.15.23:0/331071033 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f8d580cf450 con 0x7f8d60103270 2024-03-28T05:22:55.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.468+0000 7f8d5ffff640 1 --2- 172.21.15.23:0/331071033 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8d480a2800 0x7f8d480a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8d50000c00 tx=0x7f8d50005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:55.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.472+0000 7f8d5d7fa640 1 -- 172.21.15.23:0/331071033 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8d5809c190 con 0x7f8d60103270 2024-03-28T05:22:55.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:55 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1658221771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:22:55.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:55 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3268872044' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:22:55.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:55 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2882481173' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:22:55.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:55 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1658221771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:22:55.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:55 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3268872044' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:22:55.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:55 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2882481173' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:22:55.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.695+0000 7f8d6632f640 1 -- 172.21.15.23:0/331071033 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d6002d070 con 0x7f8d60103270 2024-03-28T05:22:55.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.696+0000 7f8d5d7fa640 1 -- 172.21.15.23:0/331071033 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11324 (secure 0 0 0) 0x7f8d5809ba90 con 0x7f8d60103270 2024-03-28T05:22:55.694 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:22:55.694 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/733913400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":733913400},{"type":"v1","addr":"172.21.15.23:6835","nonce":733913400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1972901387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1972901387},{"type":"v1","addr":"172.21.15.23:6837","nonce":1972901387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/2390673788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2390673788},{"type":"v1","addr":"172.21.15.23:6839","nonce":2390673788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/289263156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":289263156},{"type":"v1","addr":"172.21.15.23:6841","nonce":289263156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3313932732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3313932732},{"type":"v1","addr":"172.21.15.98:6835","nonce":3313932732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2695340024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2695340024},{"type":"v1","addr":"172.21.15.113:6835","nonce":2695340024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2750367469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2750367469},{"type":"v1","addr":"172.21.15.98:6837","nonce":2750367469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/3222726274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":3222726274},{"type":"v1","addr":"172.21.15.113:6837","nonce":3222726274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T05:21:07.616553+0000","modified":"2024-03-28T05:21:32.732132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/144113007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":144113007},{"type":"v1","addr":"172.21.15.98:6839","nonce":144113007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.113:6839/212060489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":212060489},{"type":"v1","addr":"172.21.15.113:6839","nonce":212060489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24487,"qdb_cluster":[24487]},"id":1}]} 2024-03-28T05:22:55.695 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 20 2024-03-28T05:22:55.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.698+0000 7f8d6632f640 1 -- 172.21.15.23:0/331071033 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8d480a2800 msgr2=0x7f8d480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:55.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.698+0000 7f8d6632f640 1 --2- 172.21.15.23:0/331071033 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8d480a2800 0x7f8d480a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8d50000c00 tx=0x7f8d50005fd0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.698+0000 7f8d6632f640 1 -- 172.21.15.23:0/331071033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60103270 msgr2=0x7f8d60073fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.698+0000 7f8d6632f640 1 --2- 172.21.15.23:0/331071033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60103270 0x7f8d60073fd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8d5800eb10 tx=0x7f8d5800c2d0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.698+0000 7f8d6632f640 1 -- 172.21.15.23:0/331071033 shutdown_connections 2024-03-28T05:22:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.698+0000 7f8d6632f640 1 --2- 172.21.15.23:0/331071033 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8d480a2800 0x7f8d480a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.698+0000 7f8d6632f640 1 --2- 172.21.15.23:0/331071033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60103270 0x7f8d60073fd0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.698+0000 7f8d6632f640 1 --2- 172.21.15.23:0/331071033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d601028b0 0x7f8d60073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.698+0000 7f8d6632f640 1 --2- 172.21.15.23:0/331071033 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d601016c0 0x7f8d60075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.698+0000 7f8d6632f640 1 -- 172.21.15.23:0/331071033 >> 172.21.15.23:0/331071033 conn(0x7f8d600fcc50 msgr2=0x7f8d60105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.698+0000 7f8d6632f640 1 -- 172.21.15.23:0/331071033 shutdown_connections 2024-03-28T05:22:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.698+0000 7f8d6632f640 1 -- 172.21.15.23:0/331071033 wait complete. 2024-03-28T05:22:55.708 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T05:22:55.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.812+0000 7f2990d53640 1 -- 172.21.15.23:0/3070107852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f298c1021f0 msgr2=0x7f298c108a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:55.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.812+0000 7f2990d53640 1 --2- 172.21.15.23:0/3070107852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f298c1021f0 0x7f298c108a80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2984009900 tx=0x7f298402bc30 comp rx=0 tx=0).stop 2024-03-28T05:22:55.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.812+0000 7f2990d53640 1 -- 172.21.15.23:0/3070107852 shutdown_connections 2024-03-28T05:22:55.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.812+0000 7f2990d53640 1 --2- 172.21.15.23:0/3070107852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f298c1021f0 0x7f298c108a80 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.812+0000 7f2990d53640 1 --2- 172.21.15.23:0/3070107852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f298c101830 0x7f298c101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.812+0000 7f2990d53640 1 --2- 172.21.15.23:0/3070107852 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f298c100640 0x7f298c100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.812+0000 7f2990d53640 1 -- 172.21.15.23:0/3070107852 >> 172.21.15.23:0/3070107852 conn(0x7f298c0fbbd0 msgr2=0x7f298c0fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:55.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.812+0000 7f2990d53640 1 -- 172.21.15.23:0/3070107852 shutdown_connections 2024-03-28T05:22:55.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.812+0000 7f2990d53640 1 -- 172.21.15.23:0/3070107852 wait complete. 2024-03-28T05:22:55.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.813+0000 7f2990d53640 1 Processor -- start 2024-03-28T05:22:55.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.813+0000 7f2990d53640 1 -- start start 2024-03-28T05:22:55.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.813+0000 7f2990d53640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f298c100640 0x7f298c10afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:55.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.813+0000 7f2990d53640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f298c101830 0x7f298c109600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:55.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.813+0000 7f2990d53640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f298c1021f0 0x7f298c109b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:55.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.813+0000 7f2990d53640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298c10d3b0 con 0x7f298c1021f0 2024-03-28T05:22:55.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.813+0000 7f2990d53640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298c10d230 con 0x7f298c101830 2024-03-28T05:22:55.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.813+0000 7f2990d53640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298c10d530 con 0x7f298c100640 2024-03-28T05:22:55.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.813+0000 7f298ad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f298c1021f0 0x7f298c109b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:55.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.813+0000 7f298ad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f298c1021f0 0x7f298c109b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34294/0 (socket says 172.21.15.23:34294) 2024-03-28T05:22:55.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.813+0000 7f298ad9d640 1 -- 172.21.15.23:0/2995147278 learned_addr learned my addr 172.21.15.23:0/2995147278 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:55.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.813+0000 7f298a59c640 1 --2- 172.21.15.23:0/2995147278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f298c100640 0x7f298c10afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:55.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.813+0000 7f2989d9b640 1 --2- 172.21.15.23:0/2995147278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f298c101830 0x7f298c109600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:55.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.813+0000 7f298ad9d640 1 -- 172.21.15.23:0/2995147278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f298c100640 msgr2=0x7f298c10afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:55.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.813+0000 7f298ad9d640 1 --2- 172.21.15.23:0/2995147278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f298c100640 0x7f298c10afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.813+0000 7f298ad9d640 1 -- 172.21.15.23:0/2995147278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f298c101830 msgr2=0x7f298c109600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:55.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.813+0000 7f298ad9d640 1 --2- 172.21.15.23:0/2995147278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f298c101830 0x7f298c109600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:55.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.813+0000 7f298ad9d640 1 -- 172.21.15.23:0/2995147278 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f298c10a3d0 con 0x7f298c1021f0 2024-03-28T05:22:55.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.813+0000 7f298ad9d640 1 --2- 172.21.15.23:0/2995147278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f298c1021f0 0x7f298c109b40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f298402c8e0 tx=0x7f298402c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:55.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.814+0000 7f29837fe640 1 -- 172.21.15.23:0/2995147278 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2984043020 con 0x7f298c1021f0 2024-03-28T05:22:55.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.814+0000 7f29837fe640 1 -- 172.21.15.23:0/2995147278 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29840054e0 con 0x7f298c1021f0 2024-03-28T05:22:55.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.814+0000 7f2990d53640 1 -- 172.21.15.23:0/2995147278 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f298c1a1a90 con 0x7f298c1021f0 2024-03-28T05:22:55.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.814+0000 7f29837fe640 1 -- 172.21.15.23:0/2995147278 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2984003ce0 con 0x7f298c1021f0 2024-03-28T05:22:55.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.814+0000 7f2990d53640 1 -- 172.21.15.23:0/2995147278 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f298c1a1fe0 con 0x7f298c1021f0 2024-03-28T05:22:55.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.815+0000 7f2990d53640 1 -- 172.21.15.23:0/2995147278 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f298c107890 con 0x7f298c1021f0 2024-03-28T05:22:55.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.816+0000 7f29837fe640 1 -- 172.21.15.23:0/2995147278 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2984046020 con 0x7f298c1021f0 2024-03-28T05:22:55.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.816+0000 7f29837fe640 1 --2- 172.21.15.23:0/2995147278 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f29780a2800 0x7f29780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:55.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.816+0000 7f298a59c640 1 --2- 172.21.15.23:0/2995147278 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f29780a2800 0x7f29780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:55.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.817+0000 7f29837fe640 1 -- 172.21.15.23:0/2995147278 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f29840f0f60 con 0x7f298c1021f0 2024-03-28T05:22:55.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.817+0000 7f298a59c640 1 --2- 172.21.15.23:0/2995147278 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f29780a2800 0x7f29780a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f298c101690 tx=0x7f297c009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:55.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:55.821+0000 7f29837fe640 1 -- 172.21.15.23:0/2995147278 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f298404d020 con 0x7f298c1021f0 2024-03-28T05:22:56.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.007+0000 7f2990d53640 1 -- 172.21.15.23:0/2995147278 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f298c0622a0 con 0x7f298c1021f0 2024-03-28T05:22:56.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.008+0000 7f29837fe640 1 -- 172.21.15.23:0/2995147278 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24554 (secure 0 0 0) 0x7f29840bdca0 con 0x7f298c1021f0 2024-03-28T05:22:56.007 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:22:56.007 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":91,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","created":"2024-03-28T05:13:13.453653+0000","modified":"2024-03-28T05:22:10.494031+0000","last_up_change":"2024-03-28T05:20:20.221527+0000","last_in_change":"2024-03-28T05:20:00.365478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T05:16:32.986974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T05:21:06.103894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T05:21:06.827190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T05:21:10.507418+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"18150942-59a4-4c3b-83d4-c1fc3ccf1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6803","nonce":1219956881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6805","nonce":1219956881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6809","nonce":1219956881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6807","nonce":1219956881}]},"public_addr":"172.21.15.23:6803/1219956881","cluster_addr":"172.21.15.23:6805/1219956881","heartbeat_back_addr":"172.21.15.23:6809/1219956881","heartbeat_front_addr":"172.21.15.23:6807/1219956881","state":["exists","up"]},{"osd":1,"uuid":"ba6d1a95-5135-4670-95f9-7878c0e7bd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6801","nonce":555491234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6803","nonce":555491234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6807","nonce":555491234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6805","nonce":555491234}]},"public_addr":"172.21.15.98:6801/555491234","cluster_addr":"172.21.15.98:6803/555491234","heartbeat_back_addr":"172.21.15.98:6807/555491234","heartbeat_front_addr":"172.21.15.98:6805/555491234","state":["exists","up"]},{"osd":2,"uuid":"873bdd60-2a47-403b-b1f4-ee9226a95067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6801","nonce":1535856914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6803","nonce":1535856914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6807","nonce":1535856914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6805","nonce":1535856914}]},"public_addr":"172.21.15.113:6801/1535856914","cluster_addr":"172.21.15.113:6803/1535856914","heartbeat_back_addr":"172.21.15.113:6807/1535856914","heartbeat_front_addr":"172.21.15.113:6805/1535856914","state":["exists","up"]},{"osd":3,"uuid":"83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6811","nonce":2949415020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6813","nonce":2949415020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6817","nonce":2949415020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6815","nonce":2949415020}]},"public_addr":"172.21.15.23:6811/2949415020","cluster_addr":"172.21.15.23:6813/2949415020","heartbeat_back_addr":"172.21.15.23:6817/2949415020","heartbeat_front_addr":"172.21.15.23:6815/2949415020","state":["exists","up"]},{"osd":4,"uuid":"76a90bac-b221-40a3-8238-40e8049cec36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6809","nonce":2615902819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6811","nonce":2615902819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6815","nonce":2615902819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6813","nonce":2615902819}]},"public_addr":"172.21.15.98:6809/2615902819","cluster_addr":"172.21.15.98:6811/2615902819","heartbeat_back_addr":"172.21.15.98:6815/2615902819","heartbeat_front_addr":"172.21.15.98:6813/2615902819","state":["exists","up"]},{"osd":5,"uuid":"b53fdb55-a8df-40a8-bd1b-8bcef634e26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6809","nonce":625271528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6811","nonce":625271528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6815","nonce":625271528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6813","nonce":625271528}]},"public_addr":"172.21.15.113:6809/625271528","cluster_addr":"172.21.15.113:6811/625271528","heartbeat_back_addr":"172.21.15.113:6815/625271528","heartbeat_front_addr":"172.21.15.113:6813/625271528","state":["exists","up"]},{"osd":6,"uuid":"f00283d2-8d67-4e97-a575-9a9d3b55d5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6819","nonce":4188464176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6821","nonce":4188464176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6825","nonce":4188464176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6823","nonce":4188464176}]},"public_addr":"172.21.15.23:6819/4188464176","cluster_addr":"172.21.15.23:6821/4188464176","heartbeat_back_addr":"172.21.15.23:6825/4188464176","heartbeat_front_addr":"172.21.15.23:6823/4188464176","state":["exists","up"]},{"osd":7,"uuid":"edc008a7-b07b-40a8-88b7-1614ba38a775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6817","nonce":1985546028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6819","nonce":1985546028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6823","nonce":1985546028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6821","nonce":1985546028}]},"public_addr":"172.21.15.98:6817/1985546028","cluster_addr":"172.21.15.98:6819/1985546028","heartbeat_back_addr":"172.21.15.98:6823/1985546028","heartbeat_front_addr":"172.21.15.98:6821/1985546028","state":["exists","up"]},{"osd":8,"uuid":"56e7b2d1-81f0-47b2-8638-f29832bed8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6817","nonce":3534876081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6819","nonce":3534876081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6823","nonce":3534876081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6821","nonce":3534876081}]},"public_addr":"172.21.15.113:6817/3534876081","cluster_addr":"172.21.15.113:6819/3534876081","heartbeat_back_addr":"172.21.15.113:6823/3534876081","heartbeat_front_addr":"172.21.15.113:6821/3534876081","state":["exists","up"]},{"osd":9,"uuid":"a2e90caa-6e9c-42bf-9f18-81f7f47880cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6827","nonce":672264207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6829","nonce":672264207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6833","nonce":672264207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6831","nonce":672264207}]},"public_addr":"172.21.15.23:6827/672264207","cluster_addr":"172.21.15.23:6829/672264207","heartbeat_back_addr":"172.21.15.23:6833/672264207","heartbeat_front_addr":"172.21.15.23:6831/672264207","state":["exists","up"]},{"osd":10,"uuid":"0eda917f-020c-437f-95ae-b26203f6c68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6825","nonce":2699920595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6827","nonce":2699920595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6831","nonce":2699920595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6829","nonce":2699920595}]},"public_addr":"172.21.15.98:6825/2699920595","cluster_addr":"172.21.15.98:6827/2699920595","heartbeat_back_addr":"172.21.15.98:6831/2699920595","heartbeat_front_addr":"172.21.15.98:6829/2699920595","state":["exists","up"]},{"osd":11,"uuid":"aae78501-78a8-469a-be20-3006f5ec5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6825","nonce":2113644020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6827","nonce":2113644020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6831","nonce":2113644020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6829","nonce":2113644020}]},"public_addr":"172.21.15.113:6825/2113644020","cluster_addr":"172.21.15.113:6827/2113644020","heartbeat_back_addr":"172.21.15.113:6831/2113644020","heartbeat_front_addr":"172.21.15.113:6829/2113644020","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:15:43.527475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:07.006538+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:30.594176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:55.810999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:20.595811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:45.474138+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:10.500409+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:35.787638+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:01.001188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:26.761115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:52.952752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:20:18.559832+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":7}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:6800/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1876451381":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2253559862":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2653562762":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/3337463830":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6800/1735221490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/2784771635":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2389902490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6801/1735221490":"2024-03-29T05:13:42.579607+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T05:22:56.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.010+0000 7f2990d53640 1 -- 172.21.15.23:0/2995147278 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f29780a2800 msgr2=0x7f29780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:56.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.010+0000 7f2990d53640 1 --2- 172.21.15.23:0/2995147278 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f29780a2800 0x7f29780a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f298c101690 tx=0x7f297c009500 comp rx=0 tx=0).stop 2024-03-28T05:22:56.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.010+0000 7f2990d53640 1 -- 172.21.15.23:0/2995147278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f298c1021f0 msgr2=0x7f298c109b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:56.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.010+0000 7f2990d53640 1 --2- 172.21.15.23:0/2995147278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f298c1021f0 0x7f298c109b40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f298402c8e0 tx=0x7f298402c6a0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.010+0000 7f2990d53640 1 -- 172.21.15.23:0/2995147278 shutdown_connections 2024-03-28T05:22:56.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.010+0000 7f2990d53640 1 --2- 172.21.15.23:0/2995147278 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f29780a2800 0x7f29780a4cc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.010+0000 7f2990d53640 1 --2- 172.21.15.23:0/2995147278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f298c1021f0 0x7f298c109b40 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.010+0000 7f2990d53640 1 --2- 172.21.15.23:0/2995147278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f298c101830 0x7f298c109600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.010+0000 7f2990d53640 1 --2- 172.21.15.23:0/2995147278 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f298c100640 0x7f298c10afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.010+0000 7f2990d53640 1 -- 172.21.15.23:0/2995147278 >> 172.21.15.23:0/2995147278 conn(0x7f298c0fbbd0 msgr2=0x7f298c104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:56.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.010+0000 7f2990d53640 1 -- 172.21.15.23:0/2995147278 shutdown_connections 2024-03-28T05:22:56.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.010+0000 7f2990d53640 1 -- 172.21.15.23:0/2995147278 wait complete. 2024-03-28T05:22:56.020 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:22:56.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.125+0000 7f79b609a640 1 -- 172.21.15.23:0/3773954357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79b0100640 msgr2=0x7f79b0100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:56.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.125+0000 7f79b609a640 1 --2- 172.21.15.23:0/3773954357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79b0100640 0x7f79b0100a60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f79a4009900 tx=0x7f79a402bc30 comp rx=0 tx=0).stop 2024-03-28T05:22:56.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.125+0000 7f79b609a640 1 -- 172.21.15.23:0/3773954357 shutdown_connections 2024-03-28T05:22:56.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.125+0000 7f79b609a640 1 --2- 172.21.15.23:0/3773954357 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79b01021f0 0x7f79b0108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.125+0000 7f79b609a640 1 --2- 172.21.15.23:0/3773954357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79b0101830 0x7f79b0101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.125+0000 7f79b609a640 1 --2- 172.21.15.23:0/3773954357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79b0100640 0x7f79b0100a60 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.125+0000 7f79b609a640 1 -- 172.21.15.23:0/3773954357 >> 172.21.15.23:0/3773954357 conn(0x7f79b00fbbd0 msgr2=0x7f79b00fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.125+0000 7f79b609a640 1 -- 172.21.15.23:0/3773954357 shutdown_connections 2024-03-28T05:22:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.125+0000 7f79b609a640 1 -- 172.21.15.23:0/3773954357 wait complete. 2024-03-28T05:22:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.125+0000 7f79b609a640 1 Processor -- start 2024-03-28T05:22:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.126+0000 7f79b609a640 1 -- start start 2024-03-28T05:22:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.126+0000 7f79b609a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79b0100640 0x7f79b006a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.126+0000 7f79b609a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79b0101830 0x7f79b019c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.126+0000 7f79b609a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79b01021f0 0x7f79b019ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.126+0000 7f79b609a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b010c240 con 0x7f79b0100640 2024-03-28T05:22:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.126+0000 7f79b609a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b010c0c0 con 0x7f79b0101830 2024-03-28T05:22:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.126+0000 7f79b609a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b010c3c0 con 0x7f79b01021f0 2024-03-28T05:22:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.126+0000 7f79af7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79b0100640 0x7f79b006a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.126+0000 7f79af7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79b0100640 0x7f79b006a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34316/0 (socket says 172.21.15.23:34316) 2024-03-28T05:22:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.126+0000 7f79af7fe640 1 -- 172.21.15.23:0/723063153 learned_addr learned my addr 172.21.15.23:0/723063153 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.126+0000 7f79affff640 1 --2- 172.21.15.23:0/723063153 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79b01021f0 0x7f79b019ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.126+0000 7f79aeffd640 1 --2- 172.21.15.23:0/723063153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79b0101830 0x7f79b019c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.126+0000 7f79af7fe640 1 -- 172.21.15.23:0/723063153 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79b01021f0 msgr2=0x7f79b019ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.126+0000 7f79af7fe640 1 --2- 172.21.15.23:0/723063153 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79b01021f0 0x7f79b019ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.126+0000 7f79af7fe640 1 -- 172.21.15.23:0/723063153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79b0101830 msgr2=0x7f79b019c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.126+0000 7f79af7fe640 1 --2- 172.21.15.23:0/723063153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79b0101830 0x7f79b019c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.126+0000 7f79af7fe640 1 -- 172.21.15.23:0/723063153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b019f620 con 0x7f79b0100640 2024-03-28T05:22:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.126+0000 7f79aeffd640 1 --2- 172.21.15.23:0/723063153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79b0101830 0x7f79b019c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:22:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.126+0000 7f79af7fe640 1 --2- 172.21.15.23:0/723063153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79b0100640 0x7f79b006a860 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f79a4000c00 tx=0x7f79a4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.127+0000 7f79acff9640 1 -- 172.21.15.23:0/723063153 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a4041070 con 0x7f79b0100640 2024-03-28T05:22:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.127+0000 7f79b609a640 1 -- 172.21.15.23:0/723063153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b019f8b0 con 0x7f79b0100640 2024-03-28T05:22:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.127+0000 7f79acff9640 1 -- 172.21.15.23:0/723063153 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79a40044d0 con 0x7f79b0100640 2024-03-28T05:22:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.127+0000 7f79acff9640 1 -- 172.21.15.23:0/723063153 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a4034440 con 0x7f79b0100640 2024-03-28T05:22:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.127+0000 7f79b609a640 1 -- 172.21.15.23:0/723063153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b019fdf0 con 0x7f79b0100640 2024-03-28T05:22:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.128+0000 7f79b609a640 1 -- 172.21.15.23:0/723063153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b0107a80 con 0x7f79b0100640 2024-03-28T05:22:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.129+0000 7f79acff9640 1 -- 172.21.15.23:0/723063153 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f79a4044070 con 0x7f79b0100640 2024-03-28T05:22:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.129+0000 7f79acff9640 1 --2- 172.21.15.23:0/723063153 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f79980a2800 0x7f79980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.129+0000 7f79aeffd640 1 --2- 172.21.15.23:0/723063153 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f79980a2800 0x7f79980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:56.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.129+0000 7f79acff9640 1 -- 172.21.15.23:0/723063153 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f79a40f0b90 con 0x7f79b0100640 2024-03-28T05:22:56.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.130+0000 7f79aeffd640 1 --2- 172.21.15.23:0/723063153 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f79980a2800 0x7f79980a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f79b006a5d0 tx=0x7f79a0005d30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:56.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.133+0000 7f79acff9640 1 -- 172.21.15.23:0/723063153 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f79a40bd8d0 con 0x7f79b0100640 2024-03-28T05:22:56.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:56 smithi023 ceph-mon[31954]: pgmap v344: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-28T05:22:56.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:56 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2992721831' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:22:56.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:56 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/331071033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:22:56.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:56 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2995147278' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:22:56.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.357+0000 7f79b609a640 1 -- 172.21.15.23:0/723063153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79b002d070 con 0x7f79b0100640 2024-03-28T05:22:56.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.358+0000 7f79acff9640 1 -- 172.21.15.23:0/723063153 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11324 (secure 0 0 0) 0x7f79a40bd1d0 con 0x7f79b0100640 2024-03-28T05:22:56.356 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:22:56.356 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/733913400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":733913400},{"type":"v1","addr":"172.21.15.23:6835","nonce":733913400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1972901387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1972901387},{"type":"v1","addr":"172.21.15.23:6837","nonce":1972901387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/2390673788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2390673788},{"type":"v1","addr":"172.21.15.23:6839","nonce":2390673788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/289263156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":289263156},{"type":"v1","addr":"172.21.15.23:6841","nonce":289263156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3313932732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3313932732},{"type":"v1","addr":"172.21.15.98:6835","nonce":3313932732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2695340024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2695340024},{"type":"v1","addr":"172.21.15.113:6835","nonce":2695340024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2750367469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2750367469},{"type":"v1","addr":"172.21.15.98:6837","nonce":2750367469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/3222726274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":3222726274},{"type":"v1","addr":"172.21.15.113:6837","nonce":3222726274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T05:21:07.616553+0000","modified":"2024-03-28T05:21:32.732132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/144113007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":144113007},{"type":"v1","addr":"172.21.15.98:6839","nonce":144113007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.113:6839/212060489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":212060489},{"type":"v1","addr":"172.21.15.113:6839","nonce":212060489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24487,"qdb_cluster":[24487]},"id":1}]} 2024-03-28T05:22:56.357 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 20 2024-03-28T05:22:56.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.360+0000 7f79b609a640 1 -- 172.21.15.23:0/723063153 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f79980a2800 msgr2=0x7f79980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:56.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.360+0000 7f79b609a640 1 --2- 172.21.15.23:0/723063153 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f79980a2800 0x7f79980a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f79b006a5d0 tx=0x7f79a0005d30 comp rx=0 tx=0).stop 2024-03-28T05:22:56.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.360+0000 7f79b609a640 1 -- 172.21.15.23:0/723063153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79b0100640 msgr2=0x7f79b006a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:56.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.360+0000 7f79b609a640 1 --2- 172.21.15.23:0/723063153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79b0100640 0x7f79b006a860 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f79a4000c00 tx=0x7f79a4005d50 comp rx=0 tx=0).stop 2024-03-28T05:22:56.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.360+0000 7f79b609a640 1 -- 172.21.15.23:0/723063153 shutdown_connections 2024-03-28T05:22:56.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.360+0000 7f79b609a640 1 --2- 172.21.15.23:0/723063153 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f79980a2800 0x7f79980a4cc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.360+0000 7f79b609a640 1 --2- 172.21.15.23:0/723063153 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f79b01021f0 0x7f79b019ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.360+0000 7f79b609a640 1 --2- 172.21.15.23:0/723063153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79b0101830 0x7f79b019c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.360+0000 7f79b609a640 1 --2- 172.21.15.23:0/723063153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79b0100640 0x7f79b006a860 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.360+0000 7f79b609a640 1 -- 172.21.15.23:0/723063153 >> 172.21.15.23:0/723063153 conn(0x7f79b00fbbd0 msgr2=0x7f79b0104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:56.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.360+0000 7f79b609a640 1 -- 172.21.15.23:0/723063153 shutdown_connections 2024-03-28T05:22:56.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.360+0000 7f79b609a640 1 -- 172.21.15.23:0/723063153 wait complete. 2024-03-28T05:22:56.370 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T05:21:07.616553+0000', 'modified': '2024-03-28T05:21:32.732132+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24487}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24473': {'gid': 24473, 'name': 'b', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6839/144113007', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6838', 'nonce': 144113007}, {'type': 'v1', 'addr': '172.21.15.98:6839', 'nonce': 144113007}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24487': {'gid': 24487, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.113:6839/212060489', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6838', 'nonce': 212060489}, {'type': 'v1', 'addr': '172.21.15.113:6839', 'nonce': 212060489}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24487, 'qdb_cluster': [24487]} 2024-03-28T05:22:56.370 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T05:22:56.370 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-28T05:22:56.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.475+0000 7f9534219640 1 -- 172.21.15.23:0/3748389679 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f952c101790 msgr2=0x7f952c101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:56.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.475+0000 7f9534219640 1 --2- 172.21.15.23:0/3748389679 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f952c101790 0x7f952c101c10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f951c009900 tx=0x7f951c02bc30 comp rx=0 tx=0).stop 2024-03-28T05:22:56.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.476+0000 7f9534219640 1 -- 172.21.15.23:0/3748389679 shutdown_connections 2024-03-28T05:22:56.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.476+0000 7f9534219640 1 --2- 172.21.15.23:0/3748389679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f952c102150 0x7f952c1089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.476+0000 7f9534219640 1 --2- 172.21.15.23:0/3748389679 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f952c101790 0x7f952c101c10 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.476+0000 7f9534219640 1 --2- 172.21.15.23:0/3748389679 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f952c1005a0 0x7f952c1009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.476+0000 7f9534219640 1 -- 172.21.15.23:0/3748389679 >> 172.21.15.23:0/3748389679 conn(0x7f952c0fbb30 msgr2=0x7f952c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:56.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.476+0000 7f9534219640 1 -- 172.21.15.23:0/3748389679 shutdown_connections 2024-03-28T05:22:56.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.476+0000 7f9534219640 1 -- 172.21.15.23:0/3748389679 wait complete. 2024-03-28T05:22:56.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.476+0000 7f9534219640 1 Processor -- start 2024-03-28T05:22:56.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.476+0000 7f9534219640 1 -- start start 2024-03-28T05:22:56.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.476+0000 7f9534219640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f952c1005a0 0x7f952c1979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:56.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.476+0000 7f9534219640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f952c101790 0x7f952c197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:56.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.476+0000 7f9534219640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f952c102150 0x7f952c19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:56.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.476+0000 7f9534219640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c10d340 con 0x7f952c1005a0 2024-03-28T05:22:56.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.476+0000 7f9534219640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c10d1c0 con 0x7f952c101790 2024-03-28T05:22:56.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.476+0000 7f9534219640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c10d4c0 con 0x7f952c102150 2024-03-28T05:22:56.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.476+0000 7f9531fb5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f952c1005a0 0x7f952c1979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:56.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.476+0000 7f9531fb5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f952c1005a0 0x7f952c1979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34344/0 (socket says 172.21.15.23:34344) 2024-03-28T05:22:56.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.476+0000 7f9531fb5640 1 -- 172.21.15.23:0/2501127314 learned_addr learned my addr 172.21.15.23:0/2501127314 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:56.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.477+0000 7f95327b6640 1 --2- 172.21.15.23:0/2501127314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f952c102150 0x7f952c19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:56.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.477+0000 7f95317b4640 1 --2- 172.21.15.23:0/2501127314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f952c101790 0x7f952c197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:56.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.477+0000 7f9531fb5640 1 -- 172.21.15.23:0/2501127314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f952c102150 msgr2=0x7f952c19efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:56.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.477+0000 7f9531fb5640 1 --2- 172.21.15.23:0/2501127314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f952c102150 0x7f952c19efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.477+0000 7f9531fb5640 1 -- 172.21.15.23:0/2501127314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f952c101790 msgr2=0x7f952c197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:56.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.477+0000 7f9531fb5640 1 --2- 172.21.15.23:0/2501127314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f952c101790 0x7f952c197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.477+0000 7f9531fb5640 1 -- 172.21.15.23:0/2501127314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f952c19f610 con 0x7f952c1005a0 2024-03-28T05:22:56.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.477+0000 7f95327b6640 1 --2- 172.21.15.23:0/2501127314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f952c102150 0x7f952c19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:22:56.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.477+0000 7f95317b4640 1 --2- 172.21.15.23:0/2501127314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f952c101790 0x7f952c197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:22:56.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.477+0000 7f9531fb5640 1 --2- 172.21.15.23:0/2501127314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f952c1005a0 0x7f952c1979f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f952800e9c0 tx=0x7f952800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:56.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.477+0000 7f9522ffd640 1 -- 172.21.15.23:0/2501127314 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952800cd80 con 0x7f952c1005a0 2024-03-28T05:22:56.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.477+0000 7f9534219640 1 -- 172.21.15.23:0/2501127314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f952c19f900 con 0x7f952c1005a0 2024-03-28T05:22:56.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.477+0000 7f9522ffd640 1 -- 172.21.15.23:0/2501127314 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95280035f0 con 0x7f952c1005a0 2024-03-28T05:22:56.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.477+0000 7f9522ffd640 1 -- 172.21.15.23:0/2501127314 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9528010690 con 0x7f952c1005a0 2024-03-28T05:22:56.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.477+0000 7f9534219640 1 -- 172.21.15.23:0/2501127314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f952c19fdd0 con 0x7f952c1005a0 2024-03-28T05:22:56.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.479+0000 7f9522ffd640 1 -- 172.21.15.23:0/2501127314 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f95280037a0 con 0x7f952c1005a0 2024-03-28T05:22:56.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.479+0000 7f9536e07740 1 Processor -- start 2024-03-28T05:22:56.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.480+0000 7f9536e07740 1 -- start start 2024-03-28T05:22:56.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.480+0000 7f9522ffd640 1 --2- 172.21.15.23:0/2501127314 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f95180a2800 0x7f95180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:56.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.480+0000 7f9536e07740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a77740 0x55c2e2a77b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:56.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.480+0000 7f9536e07740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c2e2a78130 0x55c2e2a7cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:56.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.480+0000 7f9536e07740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2e2a7d2a0 0x55c2e2a7f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:56.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.480+0000 7f9536e07740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e29dcc20 con 0x55c2e2a77740 2024-03-28T05:22:56.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.480+0000 7f9536e07740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e2985a40 con 0x55c2e2a78130 2024-03-28T05:22:56.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.480+0000 7f9536e07740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e29d8cc0 con 0x55c2e2a7d2a0 2024-03-28T05:22:56.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.480+0000 7f95327b6640 1 --2- 172.21.15.23:0/2501127314 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f95180a2800 0x7f95180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:56.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.480+0000 7f9531fb5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a77740 0x55c2e2a77b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:56.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.480+0000 7f9531fb5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a77740 0x55c2e2a77b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34356/0 (socket says 172.21.15.23:34356) 2024-03-28T05:22:56.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.480+0000 7f9531fb5640 1 -- 172.21.15.23:0/45704945 learned_addr learned my addr 172.21.15.23:0/45704945 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:56.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.480+0000 7f95327b6640 1 --2- 172.21.15.23:0/45704945 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2e2a7d2a0 0x55c2e2a7f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:56.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.480+0000 7f9522ffd640 1 -- 172.21.15.23:0/2501127314 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f95280cf2b0 con 0x7f952c1005a0 2024-03-28T05:22:56.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.480+0000 7f95317b4640 1 --2- 172.21.15.23:0/45704945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c2e2a78130 0x55c2e2a7cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:56.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.480+0000 7f9531fb5640 1 -- 172.21.15.23:0/45704945 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2e2a7d2a0 msgr2=0x55c2e2a7f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:22:56.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.480+0000 7f9531fb5640 1 --2- 172.21.15.23:0/45704945 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2e2a7d2a0 0x55c2e2a7f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.480+0000 7f9531fb5640 1 -- 172.21.15.23:0/45704945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c2e2a78130 msgr2=0x55c2e2a7cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:22:56.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.480+0000 7f9531fb5640 1 --2- 172.21.15.23:0/45704945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c2e2a78130 0x55c2e2a7cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.480+0000 7f9531fb5640 1 -- 172.21.15.23:0/45704945 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2e29a7f90 con 0x55c2e2a77740 2024-03-28T05:22:56.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.480+0000 7f95327b6640 1 --2- 172.21.15.23:0/45704945 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2e2a7d2a0 0x55c2e2a7f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:22:56.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.481+0000 7f95327b6640 1 --2- 172.21.15.23:0/2501127314 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f95180a2800 0x7f95180a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9524005fd0 tx=0x7f9524014b70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:56.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.481+0000 7f9531fb5640 1 --2- 172.21.15.23:0/45704945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a77740 0x55c2e2a77b60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f95280a3b60 tx=0x7f9528097530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6f02cffc877d16f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:56.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.481+0000 7f9517fff640 1 -- 172.21.15.23:0/45704945 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952809b250 con 0x55c2e2a77740 2024-03-28T05:22:56.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.481+0000 7f9517fff640 1 -- 172.21.15.23:0/45704945 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95280a4f50 con 0x55c2e2a77740 2024-03-28T05:22:56.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.481+0000 7f9517fff640 1 -- 172.21.15.23:0/45704945 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95280a8c90 con 0x55c2e2a77740 2024-03-28T05:22:56.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.481+0000 7f9536e07740 1 -- 172.21.15.23:0/45704945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a77740 msgr2=0x55c2e2a77b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:56.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.481+0000 7f9536e07740 1 --2- 172.21.15.23:0/45704945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a77740 0x55c2e2a77b60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f95280a3b60 tx=0x7f9528097530 comp rx=0 tx=0).stop 2024-03-28T05:22:56.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.481+0000 7f9536e07740 1 -- 172.21.15.23:0/45704945 shutdown_connections 2024-03-28T05:22:56.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.481+0000 7f9536e07740 1 --2- 172.21.15.23:0/45704945 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2e2a7d2a0 0x55c2e2a7f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.481+0000 7f9536e07740 1 --2- 172.21.15.23:0/45704945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c2e2a78130 0x55c2e2a7cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.481+0000 7f9536e07740 1 --2- 172.21.15.23:0/45704945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a77740 0x55c2e2a77b60 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.481+0000 7f9536e07740 1 -- 172.21.15.23:0/45704945 >> 172.21.15.23:0/45704945 conn(0x55c2e2a3fa60 msgr2=0x55c2e2a730b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:56.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.482+0000 7f9536e07740 1 -- 172.21.15.23:0/45704945 shutdown_connections 2024-03-28T05:22:56.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.482+0000 7f9536e07740 1 -- 172.21.15.23:0/45704945 wait complete. 2024-03-28T05:22:56.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.482+0000 7f9536e07740 1 -- start start 2024-03-28T05:22:56.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.482+0000 7f9536e07740 1 Processor -- start 2024-03-28T05:22:56.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.482+0000 7f9536e07740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c2e2a77740 0x55c2e2a81b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:56.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.482+0000 7f9536e07740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a78130 0x55c2e2a820d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:56.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.482+0000 7f9536e07740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2e2a7d2a0 0x55c2e2a89110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:56.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.482+0000 7f9536e07740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e2a80190 con 0x55c2e2a78130 2024-03-28T05:22:56.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.482+0000 7f9536e07740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e2998e90 con 0x55c2e2a77740 2024-03-28T05:22:56.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.482+0000 7f9536e07740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e2a80310 con 0x55c2e2a7d2a0 2024-03-28T05:22:56.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.482+0000 7f9536e07740 10 client.0 _ll_register_callbacks cb 0x55c2e26f0ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:22:56.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.482+0000 7f95317b4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a78130 0x55c2e2a820d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:56.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.483+0000 7f95317b4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a78130 0x55c2e2a820d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34370/0 (socket says 172.21.15.23:34370) 2024-03-28T05:22:56.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.483+0000 7f95317b4640 1 -- 172.21.15.23:0/1867799863 learned_addr learned my addr 172.21.15.23:0/1867799863 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:56.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.483+0000 7f95327b6640 1 --2- 172.21.15.23:0/1867799863 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2e2a7d2a0 0x55c2e2a89110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:56.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.483+0000 7f9531fb5640 1 --2- 172.21.15.23:0/1867799863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c2e2a77740 0x55c2e2a81b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:56.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.483+0000 7f95317b4640 1 -- 172.21.15.23:0/1867799863 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2e2a7d2a0 msgr2=0x55c2e2a89110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:22:56.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.483+0000 7f95317b4640 1 --2- 172.21.15.23:0/1867799863 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2e2a7d2a0 0x55c2e2a89110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.483+0000 7f95317b4640 1 -- 172.21.15.23:0/1867799863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c2e2a77740 msgr2=0x55c2e2a81b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:22:56.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.483+0000 7f95317b4640 1 --2- 172.21.15.23:0/1867799863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c2e2a77740 0x55c2e2a81b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.483+0000 7f95317b4640 1 -- 172.21.15.23:0/1867799863 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2e29a59d0 con 0x55c2e2a78130 2024-03-28T05:22:56.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.483+0000 7f95317b4640 1 --2- 172.21.15.23:0/1867799863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a78130 0x55c2e2a820d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f951c036040 tx=0x7f951c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6dd95947f9ba750 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:56.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.483+0000 7f95157fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:22:56.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.483+0000 7f95157fa640 1 -- 172.21.15.23:0/1867799863 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c005d50 con 0x55c2e2a78130 2024-03-28T05:22:56.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.483+0000 7f95157fa640 1 -- 172.21.15.23:0/1867799863 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f951c005450 con 0x55c2e2a78130 2024-03-28T05:22:56.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.483+0000 7f9536e07740 1 -- 172.21.15.23:0/1867799863 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f95280a4f50 con 0x55c2e2a78130 2024-03-28T05:22:56.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.483+0000 7f95157fa640 1 -- 172.21.15.23:0/1867799863 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f951c04f050 con 0x55c2e2a78130 2024-03-28T05:22:56.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.483+0000 7f9536e07740 10 client.15228 fetch_fsmap learned FSMap version 20 2024-03-28T05:22:56.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.483+0000 7f9536e07740 1 -- 172.21.15.23:0/1867799863 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55c2e26f12d0 con 0x55c2e2a78130 2024-03-28T05:22:56.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.483+0000 7f95157fa640 1 -- 172.21.15.23:0/1867799863 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95280a4f50 con 0x55c2e2a78130 2024-03-28T05:22:56.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.484+0000 7f95157fa640 1 -- 172.21.15.23:0/1867799863 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f951c0037b0 con 0x55c2e2a78130 2024-03-28T05:22:56.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.484+0000 7f9536e07740 10 client.15228 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:22:56.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.484+0000 7f9536e07740 10 client.15228 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T05:22:56.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.484+0000 7f9536e07740 20 client.15228 populate_metadata read hostname 'smithi023' 2024-03-28T05:22:56.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.484+0000 7f9536e07740 1 --2- 172.21.15.23:0/1867799863 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55c2e2a8e3a0 0x55c2e2a8d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:56.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.484+0000 7f9536e07740 4 client.15228 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:22:56.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.484+0000 7f9536e07740 1 -- 172.21.15.23:0/1867799863 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c2e29ea160 con 0x55c2e2a8e3a0 2024-03-28T05:22:56.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.484+0000 7f9531fb5640 1 --2- 172.21.15.23:0/1867799863 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55c2e2a8e3a0 0x55c2e2a8d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:56.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.484+0000 7f95157fa640 10 client.15228 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:22:56.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.484+0000 7f9531fb5640 1 --2- 172.21.15.23:0/1867799863 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55c2e2a8e3a0 0x55c2e2a8d450 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dc2cd2ce94e083f server_cookie=eb8a66e768adf929 in_seq=0 out_seq=0 2024-03-28T05:22:56.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.485+0000 7f95157fa640 1 -- 172.21.15.23:0/1867799863 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c2e29ea160 con 0x55c2e2a8e3a0 2024-03-28T05:22:56.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.485+0000 7f95157fa640 10 client.15228 handle_command_reply: tid=0 2024-03-28T05:22:56.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.485+0000 7f9536e07740 1 client.15228 shutdown 2024-03-28T05:22:56.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.485+0000 7f9536e07740 1 -- 172.21.15.23:0/1867799863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a78130 msgr2=0x55c2e2a820d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:56.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.485+0000 7f9536e07740 1 --2- 172.21.15.23:0/1867799863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a78130 0x55c2e2a820d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f951c036040 tx=0x7f951c0042f0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.485+0000 7f9536e07740 1 -- 172.21.15.23:0/1867799863 shutdown_connections 2024-03-28T05:22:56.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.485+0000 7f9536e07740 1 --2- 172.21.15.23:0/1867799863 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2e2a7d2a0 0x55c2e2a89110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.485+0000 7f9536e07740 1 --2- 172.21.15.23:0/1867799863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a78130 0x55c2e2a820d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.485+0000 7f9536e07740 1 --2- 172.21.15.23:0/1867799863 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55c2e2a8e3a0 0x55c2e2a8d450 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.485+0000 7f9536e07740 1 --2- 172.21.15.23:0/1867799863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c2e2a77740 0x55c2e2a81b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.485+0000 7f95157fa640 0 client.15228 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:22:56.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.485+0000 7f9536e07740 1 -- 172.21.15.23:0/1867799863 >> 172.21.15.23:0/1867799863 conn(0x55c2e2a3fa60 msgr2=0x55c2e2a722f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:56.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.485+0000 7f9536e07740 1 -- 172.21.15.23:0/1867799863 shutdown_connections 2024-03-28T05:22:56.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.485+0000 7f9536e07740 1 -- 172.21.15.23:0/1867799863 wait complete. 2024-03-28T05:22:56.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.485+0000 7f9536e07740 20 client.15228 trim_cache size 0 max 16384 2024-03-28T05:22:56.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:56 smithi113 ceph-mon[37164]: pgmap v344: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-28T05:22:56.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:56 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2992721831' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:22:56.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:56 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/331071033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:22:56.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:56 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2995147278' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:22:56.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.502+0000 7f9536e07740 1 Processor -- start 2024-03-28T05:22:56.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.502+0000 7f9536e07740 1 -- start start 2024-03-28T05:22:56.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.502+0000 7f9536e07740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2e2a3fa60 0x55c2e2a81b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:56.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.502+0000 7f9536e07740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c2e2a77740 0x55c2e2a820d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:56.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.502+0000 7f9536e07740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a78130 0x55c2e2a890d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:56.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.502+0000 7f9536e07740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e2a8dbc0 con 0x55c2e2a78130 2024-03-28T05:22:56.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.502+0000 7f9536e07740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e2a8e220 con 0x55c2e2a77740 2024-03-28T05:22:56.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.502+0000 7f9536e07740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e2a8e880 con 0x55c2e2a3fa60 2024-03-28T05:22:56.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.502+0000 7f95327b6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a78130 0x55c2e2a890d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:56.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.502+0000 7f95327b6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a78130 0x55c2e2a890d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34386/0 (socket says 172.21.15.23:34386) 2024-03-28T05:22:56.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.502+0000 7f95327b6640 1 -- 172.21.15.23:0/1549765442 learned_addr learned my addr 172.21.15.23:0/1549765442 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:56.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.502+0000 7f9531fb5640 1 --2- 172.21.15.23:0/1549765442 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2e2a3fa60 0x55c2e2a81b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:56.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.502+0000 7f95317b4640 1 --2- 172.21.15.23:0/1549765442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c2e2a77740 0x55c2e2a820d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:56.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.502+0000 7f95327b6640 1 -- 172.21.15.23:0/1549765442 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2e2a3fa60 msgr2=0x55c2e2a81b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:22:56.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.502+0000 7f95327b6640 1 --2- 172.21.15.23:0/1549765442 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2e2a3fa60 0x55c2e2a81b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.502+0000 7f95327b6640 1 -- 172.21.15.23:0/1549765442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c2e2a77740 msgr2=0x55c2e2a820d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:22:56.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.502+0000 7f95327b6640 1 --2- 172.21.15.23:0/1549765442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c2e2a77740 0x55c2e2a820d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.502+0000 7f95327b6640 1 -- 172.21.15.23:0/1549765442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2e27bb6b0 con 0x55c2e2a78130 2024-03-28T05:22:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.503+0000 7f95327b6640 1 --2- 172.21.15.23:0/1549765442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a78130 0x55c2e2a890d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f9524012dc0 tx=0x7f952401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba30dde8d3ea1a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.503+0000 7f95177fe640 1 -- 172.21.15.23:0/1549765442 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952402d480 con 0x55c2e2a78130 2024-03-28T05:22:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.503+0000 7f95177fe640 1 -- 172.21.15.23:0/1549765442 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f952402db00 con 0x55c2e2a78130 2024-03-28T05:22:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.503+0000 7f95177fe640 1 -- 172.21.15.23:0/1549765442 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9524022a30 con 0x55c2e2a78130 2024-03-28T05:22:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.503+0000 7f9536e07740 1 -- 172.21.15.23:0/1549765442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a78130 msgr2=0x55c2e2a890d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.503+0000 7f9536e07740 1 --2- 172.21.15.23:0/1549765442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a78130 0x55c2e2a890d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f9524012dc0 tx=0x7f952401d000 comp rx=0 tx=0).stop 2024-03-28T05:22:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.503+0000 7f9536e07740 1 -- 172.21.15.23:0/1549765442 shutdown_connections 2024-03-28T05:22:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.503+0000 7f9536e07740 1 --2- 172.21.15.23:0/1549765442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a78130 0x55c2e2a890d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.503+0000 7f9536e07740 1 --2- 172.21.15.23:0/1549765442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c2e2a77740 0x55c2e2a820d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.503+0000 7f9536e07740 1 --2- 172.21.15.23:0/1549765442 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2e2a3fa60 0x55c2e2a81b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.503+0000 7f9536e07740 1 -- 172.21.15.23:0/1549765442 >> 172.21.15.23:0/1549765442 conn(0x55c2e2a78f10 msgr2=0x55c2e2a79630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.503+0000 7f9536e07740 1 -- 172.21.15.23:0/1549765442 shutdown_connections 2024-03-28T05:22:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.503+0000 7f9536e07740 1 -- 172.21.15.23:0/1549765442 wait complete. 2024-03-28T05:22:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.504+0000 7f9536e07740 1 -- start start 2024-03-28T05:22:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.504+0000 7f9536e07740 1 Processor -- start 2024-03-28T05:22:56.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.504+0000 7f9536e07740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2e2a3fa60 0x55c2e2a89030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:56.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.504+0000 7f9536e07740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a77740 0x55c2e2a85b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:56.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.504+0000 7f9536e07740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c2e2a78130 0x55c2e2a86080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:56.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.504+0000 7f9536e07740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e2a8f540 con 0x55c2e2a77740 2024-03-28T05:22:56.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.504+0000 7f9536e07740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e2a8eee0 con 0x55c2e2a78130 2024-03-28T05:22:56.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.504+0000 7f9536e07740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e2a8fba0 con 0x55c2e2a3fa60 2024-03-28T05:22:56.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.504+0000 7f9536e07740 10 client.0 _ll_register_callbacks cb 0x7f951c02cd20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:22:56.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.504+0000 7f95317b4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a77740 0x55c2e2a85b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:56.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.504+0000 7f9531fb5640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2e2a3fa60 0x55c2e2a89030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:56.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.504+0000 7f95317b4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a77740 0x55c2e2a85b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34392/0 (socket says 172.21.15.23:34392) 2024-03-28T05:22:56.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.504+0000 7f95317b4640 1 -- 172.21.15.23:0/2949639145 learned_addr learned my addr 172.21.15.23:0/2949639145 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:56.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.504+0000 7f95327b6640 1 --2- 172.21.15.23:0/2949639145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c2e2a78130 0x55c2e2a86080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:56.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.504+0000 7f95317b4640 1 -- 172.21.15.23:0/2949639145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2e2a3fa60 msgr2=0x55c2e2a89030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:22:56.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.504+0000 7f95317b4640 1 --2- 172.21.15.23:0/2949639145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2e2a3fa60 0x55c2e2a89030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.504+0000 7f95317b4640 1 -- 172.21.15.23:0/2949639145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c2e2a78130 msgr2=0x55c2e2a86080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:22:56.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.504+0000 7f95317b4640 1 --2- 172.21.15.23:0/2949639145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c2e2a78130 0x55c2e2a86080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.504+0000 7f95317b4640 1 -- 172.21.15.23:0/2949639145 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2e2ae9400 con 0x55c2e2a77740 2024-03-28T05:22:56.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.504+0000 7f95327b6640 1 --2- 172.21.15.23:0/2949639145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c2e2a78130 0x55c2e2a86080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:22:56.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.504+0000 7f9531fb5640 1 --2- 172.21.15.23:0/2949639145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2e2a3fa60 0x55c2e2a89030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:22:56.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.505+0000 7f95317b4640 1 --2- 172.21.15.23:0/2949639145 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a77740 0x55c2e2a85b40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f951c003c00 tx=0x7f951c02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f49ad835f172651 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:56.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.505+0000 7f95157fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:22:56.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.505+0000 7f95157fa640 1 -- 172.21.15.23:0/2949639145 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c0044e0 con 0x55c2e2a77740 2024-03-28T05:22:56.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.505+0000 7f95157fa640 1 -- 172.21.15.23:0/2949639145 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f951c005250 con 0x55c2e2a77740 2024-03-28T05:22:56.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.505+0000 7f9536e07740 1 -- 172.21.15.23:0/2949639145 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f952402db00 con 0x55c2e2a77740 2024-03-28T05:22:56.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.505+0000 7f95157fa640 1 -- 172.21.15.23:0/2949639145 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f951c005d50 con 0x55c2e2a77740 2024-03-28T05:22:56.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.505+0000 7f9536e07740 10 client.15234 fetch_fsmap learned FSMap version 20 2024-03-28T05:22:56.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.505+0000 7f9536e07740 1 -- 172.21.15.23:0/2949639145 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55c2e2aea650 con 0x55c2e2a77740 2024-03-28T05:22:56.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.505+0000 7f95157fa640 1 -- 172.21.15.23:0/2949639145 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f952402db00 con 0x55c2e2a77740 2024-03-28T05:22:56.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.505+0000 7f95157fa640 1 -- 172.21.15.23:0/2949639145 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f951c004c60 con 0x55c2e2a77740 2024-03-28T05:22:56.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.505+0000 7f9536e07740 10 client.15234 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:22:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.505+0000 7f9536e07740 10 client.15234 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T05:22:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.505+0000 7f9536e07740 20 client.15234 populate_metadata read hostname 'smithi023' 2024-03-28T05:22:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.505+0000 7f9536e07740 1 --2- 172.21.15.23:0/2949639145 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55c2e2a7d2a0 0x55c2e2aea8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.505+0000 7f9536e07740 4 client.15234 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T05:22:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.505+0000 7f9536e07740 1 -- 172.21.15.23:0/2949639145 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c2e29e44b0 con 0x55c2e2a7d2a0 2024-03-28T05:22:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.506+0000 7f9531fb5640 1 --2- 172.21.15.23:0/2949639145 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55c2e2a7d2a0 0x55c2e2aea8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.506+0000 7f95157fa640 10 client.15234 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:22:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.506+0000 7f9531fb5640 1 --2- 172.21.15.23:0/2949639145 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55c2e2a7d2a0 0x55c2e2aea8e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9366f7974c787b09 server_cookie=4add4605decca320 in_seq=0 out_seq=0 2024-03-28T05:22:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.506+0000 7f95157fa640 1 -- 172.21.15.23:0/2949639145 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x55c2e29e44b0 con 0x55c2e2a7d2a0 2024-03-28T05:22:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.506+0000 7f95157fa640 10 client.15234 handle_command_reply: tid=0 2024-03-28T05:22:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.506+0000 7f9536e07740 1 client.15234 shutdown 2024-03-28T05:22:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.506+0000 7f9536e07740 1 -- 172.21.15.23:0/2949639145 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a77740 msgr2=0x55c2e2a85b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.506+0000 7f9536e07740 1 --2- 172.21.15.23:0/2949639145 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a77740 0x55c2e2a85b40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f951c003c00 tx=0x7f951c02c5d0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.506+0000 7f9536e07740 1 -- 172.21.15.23:0/2949639145 shutdown_connections 2024-03-28T05:22:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.506+0000 7f9536e07740 1 --2- 172.21.15.23:0/2949639145 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55c2e2a7d2a0 0x55c2e2aea8e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.506+0000 7f9536e07740 1 --2- 172.21.15.23:0/2949639145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c2e2a78130 0x55c2e2a86080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.506+0000 7f9536e07740 1 --2- 172.21.15.23:0/2949639145 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e2a77740 0x55c2e2a85b40 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.506+0000 7f9536e07740 1 --2- 172.21.15.23:0/2949639145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2e2a3fa60 0x55c2e2a89030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.506+0000 7f95157fa640 0 client.15234 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:22:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.506+0000 7f9536e07740 1 -- 172.21.15.23:0/2949639145 >> 172.21.15.23:0/2949639145 conn(0x55c2e2a78f10 msgr2=0x55c2e2a79630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.506+0000 7f9536e07740 1 -- 172.21.15.23:0/2949639145 shutdown_connections 2024-03-28T05:22:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.506+0000 7f9536e07740 1 -- 172.21.15.23:0/2949639145 wait complete. 2024-03-28T05:22:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.506+0000 7f9536e07740 20 client.15234 trim_cache size 0 max 16384 2024-03-28T05:22:56.511 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T05:22:56.511 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "bf37ebc4-ecc1-11ee-95d0-87774f69a715", 2024-03-28T05:22:56.511 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 0, 2024-03-28T05:22:56.511 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24487, 2024-03-28T05:22:56.512 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-28T05:22:56.512 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-28T05:22:56.512 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-28T05:22:56.512 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 108.25825707, 2024-03-28T05:22:56.512 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 20, 2024-03-28T05:22:56.512 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 91, 2024-03-28T05:22:56.512 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 89, 2024-03-28T05:22:56.512 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 121.011902288 2024-03-28T05:22:56.512 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T05:22:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.507+0000 7f9534219640 1 -- 172.21.15.23:0/2501127314 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f95180a2800 msgr2=0x7f95180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.507+0000 7f9534219640 1 --2- 172.21.15.23:0/2501127314 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f95180a2800 0x7f95180a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9524005fd0 tx=0x7f9524014b70 comp rx=0 tx=0).stop 2024-03-28T05:22:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.507+0000 7f9534219640 1 -- 172.21.15.23:0/2501127314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f952c1005a0 msgr2=0x7f952c1979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.507+0000 7f9534219640 1 --2- 172.21.15.23:0/2501127314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f952c1005a0 0x7f952c1979f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f952800e9c0 tx=0x7f952800ee80 comp rx=0 tx=0).stop 2024-03-28T05:22:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.507+0000 7f9534219640 1 -- 172.21.15.23:0/2501127314 shutdown_connections 2024-03-28T05:22:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.507+0000 7f9534219640 1 --2- 172.21.15.23:0/2501127314 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f95180a2800 0x7f95180a4cc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.507+0000 7f9534219640 1 --2- 172.21.15.23:0/2501127314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f952c102150 0x7f952c19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.507+0000 7f9534219640 1 --2- 172.21.15.23:0/2501127314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f952c101790 0x7f952c197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.507+0000 7f9534219640 1 --2- 172.21.15.23:0/2501127314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f952c1005a0 0x7f952c1979f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.507+0000 7f9534219640 1 -- 172.21.15.23:0/2501127314 >> 172.21.15.23:0/2501127314 conn(0x7f952c0fbb30 msgr2=0x7f952c104ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.507+0000 7f9534219640 1 -- 172.21.15.23:0/2501127314 shutdown_connections 2024-03-28T05:22:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:56.507+0000 7f9534219640 1 -- 172.21.15.23:0/2501127314 wait complete. 2024-03-28T05:22:56.516 INFO:tasks.cephadm:Setup complete, yielding 2024-03-28T05:22:56.517 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-28T05:22:56.531 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-28T05:22:56.531 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- bash -c 'ceph orch status' 2024-03-28T05:22:56.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:56 smithi098 ceph-mon[36087]: pgmap v344: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-28T05:22:56.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:56 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2992721831' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:22:56.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:56 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/331071033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:22:56.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:56 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2995147278' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:22:56.773 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:22:57.494 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:57 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/723063153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:22:57.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:57 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/723063153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:22:57.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:57 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/723063153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:22:58.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:22:58 smithi113 ceph-mon[37164]: pgmap v345: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:58.513 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:22:58 smithi023 ceph-mon[31954]: pgmap v345: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:58.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.524+0000 7f2f1fdb6700 1 -- 172.21.15.23:0/3823478053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f181033f0 msgr2=0x7f2f18103850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:58.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.524+0000 7f2f1fdb6700 1 --2- 172.21.15.23:0/3823478053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f181033f0 0x7f2f18103850 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f2f080099c0 tx=0x7f2f08009cd0 comp rx=0 tx=0).stop 2024-03-28T05:22:58.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.524+0000 7f2f1fdb6700 1 -- 172.21.15.23:0/3823478053 shutdown_connections 2024-03-28T05:22:58.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.524+0000 7f2f1fdb6700 1 --2- 172.21.15.23:0/3823478053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f18103d90 0x7f2f1810a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:58.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.524+0000 7f2f1fdb6700 1 --2- 172.21.15.23:0/3823478053 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f181021f0 0x7f2f18102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:58.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.524+0000 7f2f1fdb6700 1 --2- 172.21.15.23:0/3823478053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f181033f0 0x7f2f18103850 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:58.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.524+0000 7f2f1fdb6700 1 -- 172.21.15.23:0/3823478053 >> 172.21.15.23:0/3823478053 conn(0x7f2f180fd790 msgr2=0x7f2f180ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:58.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.524+0000 7f2f1fdb6700 1 -- 172.21.15.23:0/3823478053 shutdown_connections 2024-03-28T05:22:58.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.524+0000 7f2f1fdb6700 1 -- 172.21.15.23:0/3823478053 wait complete. 2024-03-28T05:22:58.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.524+0000 7f2f1fdb6700 1 Processor -- start 2024-03-28T05:22:58.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.524+0000 7f2f1fdb6700 1 -- start start 2024-03-28T05:22:58.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.525+0000 7f2f1fdb6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f181021f0 0x7f2f181983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:58.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.525+0000 7f2f1fdb6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f181033f0 0x7f2f18198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:58.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.525+0000 7f2f1fdb6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f18103d90 0x7f2f1819f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:58.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.525+0000 7f2f1fdb6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1810ddb0 con 0x7f2f181033f0 2024-03-28T05:22:58.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.525+0000 7f2f1fdb6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1810dc30 con 0x7f2f18103d90 2024-03-28T05:22:58.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.525+0000 7f2f1fdb6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1810df30 con 0x7f2f181021f0 2024-03-28T05:22:58.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.525+0000 7f2f1d351700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f181033f0 0x7f2f18198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:58.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.525+0000 7f2f1db52700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f181021f0 0x7f2f181983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:58.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.525+0000 7f2f1d351700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f181033f0 0x7f2f18198910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34410/0 (socket says 172.21.15.23:34410) 2024-03-28T05:22:58.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.525+0000 7f2f1d351700 1 -- 172.21.15.23:0/1736505751 learned_addr learned my addr 172.21.15.23:0/1736505751 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:22:58.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.525+0000 7f2f1db52700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f181021f0 0x7f2f181983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:44274/0 (socket says 172.21.15.23:44274) 2024-03-28T05:22:58.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.525+0000 7f2f1e353700 1 --2- 172.21.15.23:0/1736505751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f18103d90 0x7f2f1819f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:58.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.525+0000 7f2f1d351700 1 -- 172.21.15.23:0/1736505751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f181021f0 msgr2=0x7f2f181983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:58.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.525+0000 7f2f1d351700 1 --2- 172.21.15.23:0/1736505751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f181021f0 0x7f2f181983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:58.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.525+0000 7f2f1d351700 1 -- 172.21.15.23:0/1736505751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f18103d90 msgr2=0x7f2f1819f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:58.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.525+0000 7f2f1d351700 1 --2- 172.21.15.23:0/1736505751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f18103d90 0x7f2f1819f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:58.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.525+0000 7f2f1d351700 1 -- 172.21.15.23:0/1736505751 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f181a0030 con 0x7f2f181033f0 2024-03-28T05:22:58.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.525+0000 7f2f1db52700 1 --2- 172.21.15.23:0/1736505751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f181021f0 0x7f2f181983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:22:58.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.525+0000 7f2f1d351700 1 --2- 172.21.15.23:0/1736505751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f181033f0 0x7f2f18198910 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2f0800b580 tx=0x7f2f0800bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:58.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.525+0000 7f2f0effd700 1 -- 172.21.15.23:0/1736505751 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f08009e60 con 0x7f2f181033f0 2024-03-28T05:22:58.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.526+0000 7f2f0effd700 1 -- 172.21.15.23:0/1736505751 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f0800f460 con 0x7f2f181033f0 2024-03-28T05:22:58.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.526+0000 7f2f1fdb6700 1 -- 172.21.15.23:0/1736505751 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f181a02c0 con 0x7f2f181033f0 2024-03-28T05:22:58.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.526+0000 7f2f0effd700 1 -- 172.21.15.23:0/1736505751 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f08017630 con 0x7f2f181033f0 2024-03-28T05:22:58.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.526+0000 7f2f1fdb6700 1 -- 172.21.15.23:0/1736505751 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f181a0710 con 0x7f2f181033f0 2024-03-28T05:22:58.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.526+0000 7f2f1fdb6700 1 -- 172.21.15.23:0/1736505751 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f18068e50 con 0x7f2f181033f0 2024-03-28T05:22:58.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.527+0000 7f2f0effd700 1 -- 172.21.15.23:0/1736505751 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2f080052d0 con 0x7f2f181033f0 2024-03-28T05:22:58.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.528+0000 7f2f0effd700 1 --2- 172.21.15.23:0/1736505751 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2f040a2890 0x7f2f040a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:22:58.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.528+0000 7f2f1db52700 1 --2- 172.21.15.23:0/1736505751 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2f040a2890 0x7f2f040a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:22:58.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.529+0000 7f2f0effd700 1 -- 172.21.15.23:0/1736505751 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f2f080cfbe0 con 0x7f2f181033f0 2024-03-28T05:22:58.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.529+0000 7f2f1db52700 1 --2- 172.21.15.23:0/1736505751 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2f040a2890 0x7f2f040a4d50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2f18103250 tx=0x7f2f14009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:22:58.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.533+0000 7f2f0effd700 1 -- 172.21.15.23:0/1736505751 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2f08013070 con 0x7f2f181033f0 2024-03-28T05:22:58.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:22:58 smithi098 ceph-mon[36087]: pgmap v345: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:22:58.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.756+0000 7f2f1fdb6700 1 -- 172.21.15.23:0/1736505751 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f180625d0 con 0x7f2f040a2890 2024-03-28T05:22:58.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.757+0000 7f2f0effd700 1 -- 172.21.15.23:0/1736505751 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f2f180625d0 con 0x7f2f040a2890 2024-03-28T05:22:58.756 INFO:teuthology.orchestra.run.smithi023.stdout:Backend: cephadm 2024-03-28T05:22:58.756 INFO:teuthology.orchestra.run.smithi023.stdout:Available: Yes 2024-03-28T05:22:58.756 INFO:teuthology.orchestra.run.smithi023.stdout:Paused: No 2024-03-28T05:22:58.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.760+0000 7f2f1fdb6700 1 -- 172.21.15.23:0/1736505751 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2f040a2890 msgr2=0x7f2f040a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:58.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.760+0000 7f2f1fdb6700 1 --2- 172.21.15.23:0/1736505751 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2f040a2890 0x7f2f040a4d50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2f18103250 tx=0x7f2f14009260 comp rx=0 tx=0).stop 2024-03-28T05:22:58.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.760+0000 7f2f1fdb6700 1 -- 172.21.15.23:0/1736505751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f181033f0 msgr2=0x7f2f18198910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:22:58.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.760+0000 7f2f1fdb6700 1 --2- 172.21.15.23:0/1736505751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f181033f0 0x7f2f18198910 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2f0800b580 tx=0x7f2f0800bee0 comp rx=0 tx=0).stop 2024-03-28T05:22:58.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.760+0000 7f2f1fdb6700 1 -- 172.21.15.23:0/1736505751 shutdown_connections 2024-03-28T05:22:58.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.760+0000 7f2f1fdb6700 1 --2- 172.21.15.23:0/1736505751 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f2f040a2890 0x7f2f040a4d50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:58.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.760+0000 7f2f1fdb6700 1 --2- 172.21.15.23:0/1736505751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f18103d90 0x7f2f1819f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:58.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.760+0000 7f2f1fdb6700 1 --2- 172.21.15.23:0/1736505751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f181021f0 0x7f2f181983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:58.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.760+0000 7f2f1fdb6700 1 --2- 172.21.15.23:0/1736505751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f181033f0 0x7f2f18198910 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:22:58.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.760+0000 7f2f1fdb6700 1 -- 172.21.15.23:0/1736505751 >> 172.21.15.23:0/1736505751 conn(0x7f2f180fd790 msgr2=0x7f2f18106690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:22:58.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.760+0000 7f2f1fdb6700 1 -- 172.21.15.23:0/1736505751 shutdown_connections 2024-03-28T05:22:58.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:22:58.760+0000 7f2f1fdb6700 1 -- 172.21.15.23:0/1736505751 wait complete. 2024-03-28T05:22:59.561 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-28T05:22:59.807 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:23:00.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:00 smithi113 ceph-mon[37164]: pgmap v346: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:23:00.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:00 smithi113 ceph-mon[37164]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:00.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:00 smithi023 ceph-mon[31954]: pgmap v346: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:23:00.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:00 smithi023 ceph-mon[31954]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:00.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:00 smithi098 ceph-mon[36087]: pgmap v346: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:23:00.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:00 smithi098 ceph-mon[36087]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:01.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.577+0000 7f94fc95d700 1 -- 172.21.15.23:0/2038756352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94f81021d0 msgr2=0x7f94f81025f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:01.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.577+0000 7f94fc95d700 1 --2- 172.21.15.23:0/2038756352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94f81021d0 0x7f94f81025f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f94ec0099c0 tx=0x7f94ec009cd0 comp rx=0 tx=0).stop 2024-03-28T05:23:01.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.577+0000 7f94fc95d700 1 -- 172.21.15.23:0/2038756352 shutdown_connections 2024-03-28T05:23:01.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.577+0000 7f94fc95d700 1 --2- 172.21.15.23:0/2038756352 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94f8103d70 0x7f94f810a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:01.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.577+0000 7f94fc95d700 1 --2- 172.21.15.23:0/2038756352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94f81021d0 0x7f94f81025f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:01.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.577+0000 7f94fc95d700 1 --2- 172.21.15.23:0/2038756352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94f81033d0 0x7f94f8103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:01.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.577+0000 7f94fc95d700 1 -- 172.21.15.23:0/2038756352 >> 172.21.15.23:0/2038756352 conn(0x7f94f80fd730 msgr2=0x7f94f80ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:01.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.577+0000 7f94fc95d700 1 -- 172.21.15.23:0/2038756352 shutdown_connections 2024-03-28T05:23:01.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.577+0000 7f94fc95d700 1 -- 172.21.15.23:0/2038756352 wait complete. 2024-03-28T05:23:01.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.578+0000 7f94fc95d700 1 Processor -- start 2024-03-28T05:23:01.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.578+0000 7f94fc95d700 1 -- start start 2024-03-28T05:23:01.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.578+0000 7f94fc95d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94f81021d0 0x7f94f8198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:01.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.578+0000 7f94fc95d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94f81033d0 0x7f94f8198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:01.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.578+0000 7f94fc95d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94f8103d70 0x7f94f819f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:01.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.578+0000 7f94fc95d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f810ddc0 con 0x7f94f8103d70 2024-03-28T05:23:01.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.578+0000 7f94fc95d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f810dc40 con 0x7f94f81021d0 2024-03-28T05:23:01.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.578+0000 7f94fc95d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f810df40 con 0x7f94f81033d0 2024-03-28T05:23:01.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.578+0000 7f94f6d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94f8103d70 0x7f94f819f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:01.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.578+0000 7f94f5d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94f81033d0 0x7f94f8198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:01.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.578+0000 7f94f5d9b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94f81033d0 0x7f94f8198940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:44284/0 (socket says 172.21.15.23:44284) 2024-03-28T05:23:01.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.578+0000 7f94f5d9b700 1 -- 172.21.15.23:0/1495719458 learned_addr learned my addr 172.21.15.23:0/1495719458 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:01.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.578+0000 7f94f659c700 1 --2- 172.21.15.23:0/1495719458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94f81021d0 0x7f94f8198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:01.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.579+0000 7f94f5d9b700 1 -- 172.21.15.23:0/1495719458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94f81021d0 msgr2=0x7f94f8198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:01.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.579+0000 7f94f5d9b700 1 --2- 172.21.15.23:0/1495719458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94f81021d0 0x7f94f8198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:01.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.579+0000 7f94f5d9b700 1 -- 172.21.15.23:0/1495719458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94f8103d70 msgr2=0x7f94f819f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:01.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.579+0000 7f94f5d9b700 1 --2- 172.21.15.23:0/1495719458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94f8103d70 0x7f94f819f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:01.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.579+0000 7f94f5d9b700 1 -- 172.21.15.23:0/1495719458 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f81a0060 con 0x7f94f81033d0 2024-03-28T05:23:01.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.579+0000 7f94f5d9b700 1 --2- 172.21.15.23:0/1495719458 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94f81033d0 0x7f94f8198940 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f94e800d980 tx=0x7f94e800dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:01.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.579+0000 7f94e77fe700 1 -- 172.21.15.23:0/1495719458 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e8016070 con 0x7f94f81033d0 2024-03-28T05:23:01.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.579+0000 7f94e77fe700 1 -- 172.21.15.23:0/1495719458 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94e8009db0 con 0x7f94f81033d0 2024-03-28T05:23:01.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.579+0000 7f94e77fe700 1 -- 172.21.15.23:0/1495719458 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e800f3c0 con 0x7f94f81033d0 2024-03-28T05:23:01.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.579+0000 7f94fc95d700 1 -- 172.21.15.23:0/1495719458 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f81a0350 con 0x7f94f81033d0 2024-03-28T05:23:01.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.579+0000 7f94fc95d700 1 -- 172.21.15.23:0/1495719458 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f81a0890 con 0x7f94f81033d0 2024-03-28T05:23:01.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.580+0000 7f94fc95d700 1 -- 172.21.15.23:0/1495719458 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f8068e50 con 0x7f94f81033d0 2024-03-28T05:23:01.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.581+0000 7f94e77fe700 1 -- 172.21.15.23:0/1495719458 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f94e8019070 con 0x7f94f81033d0 2024-03-28T05:23:01.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.581+0000 7f94e77fe700 1 --2- 172.21.15.23:0/1495719458 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f94e00a2890 0x7f94e00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:01.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.582+0000 7f94f659c700 1 --2- 172.21.15.23:0/1495719458 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f94e00a2890 0x7f94e00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:01.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.582+0000 7f94e77fe700 1 -- 172.21.15.23:0/1495719458 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f94e80ce890 con 0x7f94f81033d0 2024-03-28T05:23:01.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.582+0000 7f94f659c700 1 --2- 172.21.15.23:0/1495719458 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f94e00a2890 0x7f94e00a4d50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f94ec00b580 tx=0x7f94ec005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:01.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.586+0000 7f94e77fe700 1 -- 172.21.15.23:0/1495719458 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f94e809b5d0 con 0x7f94f81033d0 2024-03-28T05:23:01.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.802+0000 7f94fc95d700 1 -- 172.21.15.23:0/1495719458 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f94f80625d0 con 0x7f94e00a2890 2024-03-28T05:23:01.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.816+0000 7f94e77fe700 1 -- 172.21.15.23:0/1495719458 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7f94f80625d0 con 0x7f94e00a2890 2024-03-28T05:23:01.815 INFO:teuthology.orchestra.run.smithi023.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-28T05:23:01.815 INFO:teuthology.orchestra.run.smithi023.stdout:mds.a smithi023 running (2m) 19s ago 2m 16.1M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 85fb08824ad8 2024-03-28T05:23:01.815 INFO:teuthology.orchestra.run.smithi023.stdout:mds.b smithi098 running (118s) 51s ago 117s 18.5M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 61e83ec3302d 2024-03-28T05:23:01.815 INFO:teuthology.orchestra.run.smithi023.stdout:mds.c smithi113 running (2m) 51s ago 2m 16.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 9a95cbf359a8 2024-03-28T05:23:01.815 INFO:teuthology.orchestra.run.smithi023.stdout:mds.d smithi023 running (2m) 19s ago 2m 13.1M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 c2781f8fc1e5 2024-03-28T05:23:01.815 INFO:teuthology.orchestra.run.smithi023.stdout:mds.e smithi098 running (2m) 51s ago 2m 15.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 5ec0c7c7e8b8 2024-03-28T05:23:01.815 INFO:teuthology.orchestra.run.smithi023.stdout:mds.f smithi113 running (2m) 51s ago 2m 13.9M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 3e4fd97c12f5 2024-03-28T05:23:01.815 INFO:teuthology.orchestra.run.smithi023.stdout:mds.g smithi023 running (2m) 19s ago 2m 13.7M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 155fb204768b 2024-03-28T05:23:01.816 INFO:teuthology.orchestra.run.smithi023.stdout:mds.h smithi098 running (2m) 51s ago 2m 15.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 2747c54fcb7d 2024-03-28T05:23:01.816 INFO:teuthology.orchestra.run.smithi023.stdout:mds.i smithi113 running (2m) 51s ago 2m 25.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 a3c39012d51c 2024-03-28T05:23:01.816 INFO:teuthology.orchestra.run.smithi023.stdout:mds.j smithi023 running (2m) 19s ago 2m 15.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 13a59a33c52c 2024-03-28T05:23:01.816 INFO:teuthology.orchestra.run.smithi023.stdout:mds.k smithi098 running (2m) 51s ago 2m 14.0M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 925761f3a988 2024-03-28T05:23:01.816 INFO:teuthology.orchestra.run.smithi023.stdout:mds.l smithi113 running (2m) 51s ago 2m 15.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 20fe5eb045d5 2024-03-28T05:23:01.816 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.x smithi023 *:9283,8765 running (9m) 19s ago 9m 513M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 bb34103dde93 2024-03-28T05:23:01.816 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.y smithi098 *:8443,8765 running (7m) 51s ago 7m 451M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 61998b1d7dd0 2024-03-28T05:23:01.816 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.z smithi113 *:8443,8765 running (7m) 51s ago 7m 453M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 a0459dda73a9 2024-03-28T05:23:01.816 INFO:teuthology.orchestra.run.smithi023.stdout:mon.a smithi023 running (9m) 19s ago 9m 49.5M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 6ba187531ba0 2024-03-28T05:23:01.816 INFO:teuthology.orchestra.run.smithi023.stdout:mon.b smithi098 running (8m) 51s ago 8m 38.5M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 0b0ed0d41fb2 2024-03-28T05:23:01.816 INFO:teuthology.orchestra.run.smithi023.stdout:mon.c smithi113 running (8m) 51s ago 8m 42.1M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 b14ccc7fcf6f 2024-03-28T05:23:01.816 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0 smithi023 running (7m) 19s ago 7m 45.5M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 6c59b1300d65 2024-03-28T05:23:01.816 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1 smithi098 running (7m) 51s ago 6m 54.1M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 0d65114419cf 2024-03-28T05:23:01.816 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2 smithi113 running (6m) 51s ago 6m 47.2M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 1a819290bcdd 2024-03-28T05:23:01.816 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3 smithi023 running (6m) 19s ago 6m 53.5M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 2d3ce698d6bd 2024-03-28T05:23:01.817 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4 smithi098 running (5m) 51s ago 5m 53.0M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 e18bc9ad1099 2024-03-28T05:23:01.817 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5 smithi113 running (5m) 51s ago 5m 45.5M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 cfb21da01736 2024-03-28T05:23:01.817 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6 smithi023 running (4m) 19s ago 4m 42.2M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 ff62f518f7e4 2024-03-28T05:23:01.817 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7 smithi098 running (4m) 51s ago 4m 46.6M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 de4fe9f5cc53 2024-03-28T05:23:01.817 INFO:teuthology.orchestra.run.smithi023.stdout:osd.8 smithi113 running (4m) 51s ago 4m 53.1M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 7897eec27f8f 2024-03-28T05:23:01.817 INFO:teuthology.orchestra.run.smithi023.stdout:osd.9 smithi023 running (3m) 19s ago 3m 45.5M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 aa6939a2d2b8 2024-03-28T05:23:01.817 INFO:teuthology.orchestra.run.smithi023.stdout:osd.10 smithi098 running (3m) 51s ago 3m 42.7M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 365de499674e 2024-03-28T05:23:01.817 INFO:teuthology.orchestra.run.smithi023.stdout:osd.11 smithi113 running (2m) 51s ago 2m 48.7M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 9abf1efc270c 2024-03-28T05:23:01.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.818+0000 7f94fc95d700 1 -- 172.21.15.23:0/1495719458 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f94e00a2890 msgr2=0x7f94e00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:01.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.818+0000 7f94fc95d700 1 --2- 172.21.15.23:0/1495719458 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f94e00a2890 0x7f94e00a4d50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f94ec00b580 tx=0x7f94ec005fb0 comp rx=0 tx=0).stop 2024-03-28T05:23:01.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.818+0000 7f94fc95d700 1 -- 172.21.15.23:0/1495719458 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94f81033d0 msgr2=0x7f94f8198940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:01.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.818+0000 7f94fc95d700 1 --2- 172.21.15.23:0/1495719458 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94f81033d0 0x7f94f8198940 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f94e800d980 tx=0x7f94e800dc90 comp rx=0 tx=0).stop 2024-03-28T05:23:01.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.818+0000 7f94fc95d700 1 -- 172.21.15.23:0/1495719458 shutdown_connections 2024-03-28T05:23:01.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.818+0000 7f94fc95d700 1 --2- 172.21.15.23:0/1495719458 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f94e00a2890 0x7f94e00a4d50 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:01.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.818+0000 7f94fc95d700 1 --2- 172.21.15.23:0/1495719458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94f8103d70 0x7f94f819f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:01.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.818+0000 7f94fc95d700 1 --2- 172.21.15.23:0/1495719458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94f81021d0 0x7f94f8198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:01.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.818+0000 7f94fc95d700 1 --2- 172.21.15.23:0/1495719458 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94f81033d0 0x7f94f8198940 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:01.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.818+0000 7f94fc95d700 1 -- 172.21.15.23:0/1495719458 >> 172.21.15.23:0/1495719458 conn(0x7f94f80fd730 msgr2=0x7f94f81066d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:01.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.819+0000 7f94fc95d700 1 -- 172.21.15.23:0/1495719458 shutdown_connections 2024-03-28T05:23:01.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:01.819+0000 7f94fc95d700 1 -- 172.21.15.23:0/1495719458 wait complete. 2024-03-28T05:23:02.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:02 smithi113 ceph-mon[37164]: pgmap v347: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:23:02.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:02 smithi023 ceph-mon[31954]: pgmap v347: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:23:02.605 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-28T05:23:02.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:02 smithi098 ceph-mon[36087]: pgmap v347: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:23:02.852 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:23:03.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:03 smithi113 ceph-mon[37164]: from='client.25084 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:03.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:03 smithi023 ceph-mon[31954]: from='client.25084 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:03.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:03 smithi098 ceph-mon[36087]: from='client.25084 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:04.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.435+0000 7fa804641700 1 -- 172.21.15.23:0/1357671040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7fc102c50 msgr2=0x7fa7fc109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:04.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.435+0000 7fa804641700 1 --2- 172.21.15.23:0/1357671040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7fc102c50 0x7fa7fc109500 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec0099c0 tx=0x7fa7ec009cd0 comp rx=0 tx=0).stop 2024-03-28T05:23:04.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.435+0000 7fa804641700 1 -- 172.21.15.23:0/1357671040 shutdown_connections 2024-03-28T05:23:04.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.435+0000 7fa804641700 1 --2- 172.21.15.23:0/1357671040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7fc102c50 0x7fa7fc109500 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:04.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.435+0000 7fa804641700 1 --2- 172.21.15.23:0/1357671040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7fc1010b0 0x7fa7fc1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:04.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.435+0000 7fa804641700 1 --2- 172.21.15.23:0/1357671040 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7fc1022b0 0x7fa7fc102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:04.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.435+0000 7fa804641700 1 -- 172.21.15.23:0/1357671040 >> 172.21.15.23:0/1357671040 conn(0x7fa7fc0fc650 msgr2=0x7fa7fc0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:04.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.435+0000 7fa804641700 1 -- 172.21.15.23:0/1357671040 shutdown_connections 2024-03-28T05:23:04.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.435+0000 7fa804641700 1 -- 172.21.15.23:0/1357671040 wait complete. 2024-03-28T05:23:04.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.435+0000 7fa804641700 1 Processor -- start 2024-03-28T05:23:04.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.435+0000 7fa804641700 1 -- start start 2024-03-28T05:23:04.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.436+0000 7fa804641700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7fc1010b0 0x7fa7fc198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:04.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.436+0000 7fa804641700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7fc1022b0 0x7fa7fc1988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:04.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.436+0000 7fa804641700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7fc102c50 0x7fa7fc19f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:04.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.436+0000 7fa804641700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc10de30 con 0x7fa7fc1022b0 2024-03-28T05:23:04.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.436+0000 7fa804641700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc06b5c0 con 0x7fa7fc102c50 2024-03-28T05:23:04.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.436+0000 7fa804641700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc10dfb0 con 0x7fa7fc1010b0 2024-03-28T05:23:04.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.436+0000 7fa801bdc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7fc1022b0 0x7fa7fc1988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:04.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.436+0000 7fa801bdc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7fc1022b0 0x7fa7fc1988b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46106/0 (socket says 172.21.15.23:46106) 2024-03-28T05:23:04.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.436+0000 7fa801bdc700 1 -- 172.21.15.23:0/172730510 learned_addr learned my addr 172.21.15.23:0/172730510 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:04.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.436+0000 7fa8023dd700 1 --2- 172.21.15.23:0/172730510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7fc1010b0 0x7fa7fc198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:04.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.436+0000 7fa802bde700 1 --2- 172.21.15.23:0/172730510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7fc102c50 0x7fa7fc19f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:04.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.436+0000 7fa8023dd700 1 -- 172.21.15.23:0/172730510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7fc102c50 msgr2=0x7fa7fc19f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:04.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.436+0000 7fa8023dd700 1 --2- 172.21.15.23:0/172730510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7fc102c50 0x7fa7fc19f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:04.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.436+0000 7fa8023dd700 1 -- 172.21.15.23:0/172730510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7fc1022b0 msgr2=0x7fa7fc1988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:04.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.436+0000 7fa8023dd700 1 --2- 172.21.15.23:0/172730510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7fc1022b0 0x7fa7fc1988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:04.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.436+0000 7fa8023dd700 1 -- 172.21.15.23:0/172730510 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7fc19ffd0 con 0x7fa7fc1010b0 2024-03-28T05:23:04.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.436+0000 7fa801bdc700 1 --2- 172.21.15.23:0/172730510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7fc1022b0 0x7fa7fc1988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:23:04.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.436+0000 7fa8023dd700 1 --2- 172.21.15.23:0/172730510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7fc1010b0 0x7fa7fc198370 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa7f800d770 tx=0x7fa7f800da80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:04.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.436+0000 7fa7f37fe700 1 -- 172.21.15.23:0/172730510 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f8016070 con 0x7fa7fc1010b0 2024-03-28T05:23:04.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.436+0000 7fa7f37fe700 1 -- 172.21.15.23:0/172730510 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7f800fb70 con 0x7fa7fc1010b0 2024-03-28T05:23:04.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.436+0000 7fa804641700 1 -- 172.21.15.23:0/172730510 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7fc1a02c0 con 0x7fa7fc1010b0 2024-03-28T05:23:04.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.436+0000 7fa7f37fe700 1 -- 172.21.15.23:0/172730510 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f8003c20 con 0x7fa7fc1010b0 2024-03-28T05:23:04.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.437+0000 7fa804641700 1 -- 172.21.15.23:0/172730510 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7fc1a0800 con 0x7fa7fc1010b0 2024-03-28T05:23:04.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.437+0000 7fa804641700 1 -- 172.21.15.23:0/172730510 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7fc068e50 con 0x7fa7fc1010b0 2024-03-28T05:23:04.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.438+0000 7fa7f37fe700 1 -- 172.21.15.23:0/172730510 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa7f8004220 con 0x7fa7fc1010b0 2024-03-28T05:23:04.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.439+0000 7fa7f37fe700 1 --2- 172.21.15.23:0/172730510 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa7e80a2890 0x7fa7e80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:04.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.439+0000 7fa801bdc700 1 --2- 172.21.15.23:0/172730510 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa7e80a2890 0x7fa7e80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:04.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.439+0000 7fa7f37fe700 1 -- 172.21.15.23:0/172730510 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fa7f80ceba0 con 0x7fa7fc1010b0 2024-03-28T05:23:04.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.440+0000 7fa801bdc700 1 --2- 172.21.15.23:0/172730510 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa7e80a2890 0x7fa7e80a4d50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4009c10 tx=0x7fa7f400b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:04.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.442+0000 7fa7f37fe700 1 -- 172.21.15.23:0/172730510 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa7f80cee20 con 0x7fa7fc1010b0 2024-03-28T05:23:04.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.659+0000 7fa804641700 1 -- 172.21.15.23:0/172730510 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7fc0625d0 con 0x7fa7e80a2890 2024-03-28T05:23:04.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.665+0000 7fa7f37fe700 1 -- 172.21.15.23:0/172730510 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7fa7fc0625d0 con 0x7fa7e80a2890 2024-03-28T05:23:04.663 INFO:teuthology.orchestra.run.smithi023.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-28T05:23:04.663 INFO:teuthology.orchestra.run.smithi023.stdout:mds.all 12/12 54s ago 2m smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi098=b;smithi098=e;smithi098=h;smithi098=k;smithi113=c;smithi113=f;smithi113=i;smithi113=l;count:12 2024-03-28T05:23:04.664 INFO:teuthology.orchestra.run.smithi023.stdout:mgr 3/3 54s ago 7m smithi023=x;smithi098=y;smithi113=z;count:3 2024-03-28T05:23:04.664 INFO:teuthology.orchestra.run.smithi023.stdout:mon 3/3 54s ago 8m smithi023:172.21.15.23=a;smithi098:172.21.15.98=b;smithi113:172.21.15.113=c;count:3 2024-03-28T05:23:04.664 INFO:teuthology.orchestra.run.smithi023.stdout:osd 12 54s ago - 2024-03-28T05:23:04.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.667+0000 7fa804641700 1 -- 172.21.15.23:0/172730510 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa7e80a2890 msgr2=0x7fa7e80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:04.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.667+0000 7fa804641700 1 --2- 172.21.15.23:0/172730510 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa7e80a2890 0x7fa7e80a4d50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4009c10 tx=0x7fa7f400b040 comp rx=0 tx=0).stop 2024-03-28T05:23:04.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.667+0000 7fa804641700 1 -- 172.21.15.23:0/172730510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7fc1010b0 msgr2=0x7fa7fc198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:04.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.667+0000 7fa804641700 1 --2- 172.21.15.23:0/172730510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7fc1010b0 0x7fa7fc198370 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa7f800d770 tx=0x7fa7f800da80 comp rx=0 tx=0).stop 2024-03-28T05:23:04.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.667+0000 7fa804641700 1 -- 172.21.15.23:0/172730510 shutdown_connections 2024-03-28T05:23:04.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.667+0000 7fa804641700 1 --2- 172.21.15.23:0/172730510 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fa7e80a2890 0x7fa7e80a4d50 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:04.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.667+0000 7fa804641700 1 --2- 172.21.15.23:0/172730510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7fc102c50 0x7fa7fc19f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:04.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.667+0000 7fa804641700 1 --2- 172.21.15.23:0/172730510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7fc1010b0 0x7fa7fc198370 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:04.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.667+0000 7fa804641700 1 --2- 172.21.15.23:0/172730510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7fc1022b0 0x7fa7fc1988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:04.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.667+0000 7fa804641700 1 -- 172.21.15.23:0/172730510 >> 172.21.15.23:0/172730510 conn(0x7fa7fc0fc650 msgr2=0x7fa7fc105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:04.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.667+0000 7fa804641700 1 -- 172.21.15.23:0/172730510 shutdown_connections 2024-03-28T05:23:04.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:04.667+0000 7fa804641700 1 -- 172.21.15.23:0/172730510 wait complete. 2024-03-28T05:23:04.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:04 smithi023 ceph-mon[31954]: pgmap v348: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:23:04.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:04 smithi113 ceph-mon[37164]: pgmap v348: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:23:05.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:04 smithi098 ceph-mon[36087]: pgmap v348: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:23:05.356 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-28T05:23:05.602 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:23:05.980 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:05 smithi023 ceph-mon[31954]: from='client.25090 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:05.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:05 smithi113 ceph-mon[37164]: from='client.25090 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:06.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:05 smithi098 ceph-mon[36087]: from='client.25090 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:06.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:06 smithi113 ceph-mon[37164]: pgmap v349: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:23:07.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:06 smithi023 ceph-mon[31954]: pgmap v349: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:23:07.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:06 smithi098 ceph-mon[36087]: pgmap v349: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:23:07.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.404+0000 7f1c1cc42700 1 -- 172.21.15.23:0/2184979970 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c18107eb0 msgr2=0x7f1c1810a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:07.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.404+0000 7f1c1cc42700 1 --2- 172.21.15.23:0/2184979970 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c18107eb0 0x7f1c1810a320 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1c1000b0a0 tx=0x7f1c1000b3b0 comp rx=0 tx=0).stop 2024-03-28T05:23:07.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.405+0000 7f1c1cc42700 1 -- 172.21.15.23:0/2184979970 shutdown_connections 2024-03-28T05:23:07.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.405+0000 7f1c1cc42700 1 --2- 172.21.15.23:0/2184979970 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c18107eb0 0x7f1c1810a320 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:07.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.405+0000 7f1c1cc42700 1 --2- 172.21.15.23:0/2184979970 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c18102c50 0x7f1c18105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:07.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.405+0000 7f1c1cc42700 1 --2- 172.21.15.23:0/2184979970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c18105580 0x7f1c18107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:07.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.405+0000 7f1c1cc42700 1 -- 172.21.15.23:0/2184979970 >> 172.21.15.23:0/2184979970 conn(0x7f1c180fc680 msgr2=0x7f1c180feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:07.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.405+0000 7f1c1cc42700 1 -- 172.21.15.23:0/2184979970 shutdown_connections 2024-03-28T05:23:07.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.405+0000 7f1c1cc42700 1 -- 172.21.15.23:0/2184979970 wait complete. 2024-03-28T05:23:07.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.405+0000 7f1c1cc42700 1 Processor -- start 2024-03-28T05:23:07.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.405+0000 7f1c1cc42700 1 -- start start 2024-03-28T05:23:07.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.406+0000 7f1c1cc42700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c18102c50 0x7f1c18198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:07.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.406+0000 7f1c1cc42700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c18105580 0x7f1c181988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:07.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.406+0000 7f1c1cc42700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c18107eb0 0x7f1c1819f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:07.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.406+0000 7f1c1cc42700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1810ddd0 con 0x7f1c18102c50 2024-03-28T05:23:07.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.406+0000 7f1c1cc42700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1810dc50 con 0x7f1c18107eb0 2024-03-28T05:23:07.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.406+0000 7f1c1cc42700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1810df50 con 0x7f1c18105580 2024-03-28T05:23:07.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.406+0000 7f1c1659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c18102c50 0x7f1c18198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:07.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.406+0000 7f1c1659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c18102c50 0x7f1c18198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46124/0 (socket says 172.21.15.23:46124) 2024-03-28T05:23:07.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.406+0000 7f1c1659c700 1 -- 172.21.15.23:0/1525429342 learned_addr learned my addr 172.21.15.23:0/1525429342 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:07.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.406+0000 7f1c15d9b700 1 --2- 172.21.15.23:0/1525429342 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c18105580 0x7f1c181988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:07.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.406+0000 7f1c16d9d700 1 --2- 172.21.15.23:0/1525429342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c18107eb0 0x7f1c1819f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:07.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.406+0000 7f1c1659c700 1 -- 172.21.15.23:0/1525429342 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c18105580 msgr2=0x7f1c181988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:07.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.406+0000 7f1c1659c700 1 --2- 172.21.15.23:0/1525429342 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c18105580 0x7f1c181988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:07.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.406+0000 7f1c1659c700 1 -- 172.21.15.23:0/1525429342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c18107eb0 msgr2=0x7f1c1819f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:07.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.406+0000 7f1c1659c700 1 --2- 172.21.15.23:0/1525429342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c18107eb0 0x7f1c1819f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:07.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.406+0000 7f1c1659c700 1 -- 172.21.15.23:0/1525429342 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c1819ffe0 con 0x7f1c18102c50 2024-03-28T05:23:07.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.406+0000 7f1c15d9b700 1 --2- 172.21.15.23:0/1525429342 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c18105580 0x7f1c181988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:23:07.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.406+0000 7f1c1659c700 1 --2- 172.21.15.23:0/1525429342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c18102c50 0x7f1c18198380 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1c0800e9f0 tx=0x7f1c0800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:07.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.407+0000 7f1c0f7fe700 1 -- 172.21.15.23:0/1525429342 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0800cbd0 con 0x7f1c18102c50 2024-03-28T05:23:07.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.407+0000 7f1c0f7fe700 1 -- 172.21.15.23:0/1525429342 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c0800ee90 con 0x7f1c18102c50 2024-03-28T05:23:07.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.407+0000 7f1c1cc42700 1 -- 172.21.15.23:0/1525429342 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c181a02d0 con 0x7f1c18102c50 2024-03-28T05:23:07.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.407+0000 7f1c0f7fe700 1 -- 172.21.15.23:0/1525429342 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c080056c0 con 0x7f1c18102c50 2024-03-28T05:23:07.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.407+0000 7f1c1cc42700 1 -- 172.21.15.23:0/1525429342 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c181a0810 con 0x7f1c18102c50 2024-03-28T05:23:07.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.408+0000 7f1c1cc42700 1 -- 172.21.15.23:0/1525429342 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c18068e50 con 0x7f1c18102c50 2024-03-28T05:23:07.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.409+0000 7f1c0f7fe700 1 -- 172.21.15.23:0/1525429342 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1c08004750 con 0x7f1c18102c50 2024-03-28T05:23:07.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.409+0000 7f1c0f7fe700 1 --2- 172.21.15.23:0/1525429342 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1c040a2910 0x7f1c040a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:07.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.409+0000 7f1c15d9b700 1 --2- 172.21.15.23:0/1525429342 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1c040a2910 0x7f1c040a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:07.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.409+0000 7f1c0f7fe700 1 -- 172.21.15.23:0/1525429342 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f1c080cef80 con 0x7f1c18102c50 2024-03-28T05:23:07.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.410+0000 7f1c15d9b700 1 --2- 172.21.15.23:0/1525429342 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1c040a2910 0x7f1c040a4dd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1c00007fd0 tx=0x7f1c000073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:07.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.414+0000 7f1c0f7fe700 1 -- 172.21.15.23:0/1525429342 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1c0809bcc0 con 0x7f1c18102c50 2024-03-28T05:23:07.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.622+0000 7f1c1cc42700 1 -- 172.21.15.23:0/1525429342 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c180625d0 con 0x7f1c040a2910 2024-03-28T05:23:07.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.623+0000 7f1c0f7fe700 1 -- 172.21.15.23:0/1525429342 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f1c180625d0 con 0x7f1c040a2910 2024-03-28T05:23:07.622 INFO:teuthology.orchestra.run.smithi023.stdout:HOST ADDR LABELS STATUS 2024-03-28T05:23:07.622 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023 172.21.15.23 2024-03-28T05:23:07.622 INFO:teuthology.orchestra.run.smithi023.stdout:smithi098 172.21.15.98 2024-03-28T05:23:07.622 INFO:teuthology.orchestra.run.smithi023.stdout:smithi113 172.21.15.113 2024-03-28T05:23:07.622 INFO:teuthology.orchestra.run.smithi023.stdout:3 hosts in cluster 2024-03-28T05:23:07.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.626+0000 7f1c1cc42700 1 -- 172.21.15.23:0/1525429342 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1c040a2910 msgr2=0x7f1c040a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:07.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.626+0000 7f1c1cc42700 1 --2- 172.21.15.23:0/1525429342 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1c040a2910 0x7f1c040a4dd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1c00007fd0 tx=0x7f1c000073d0 comp rx=0 tx=0).stop 2024-03-28T05:23:07.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.626+0000 7f1c1cc42700 1 -- 172.21.15.23:0/1525429342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c18102c50 msgr2=0x7f1c18198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:07.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.626+0000 7f1c1cc42700 1 --2- 172.21.15.23:0/1525429342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c18102c50 0x7f1c18198380 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1c0800e9f0 tx=0x7f1c0800ed00 comp rx=0 tx=0).stop 2024-03-28T05:23:07.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.626+0000 7f1c1cc42700 1 -- 172.21.15.23:0/1525429342 shutdown_connections 2024-03-28T05:23:07.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.626+0000 7f1c1cc42700 1 --2- 172.21.15.23:0/1525429342 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1c040a2910 0x7f1c040a4dd0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:07.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.626+0000 7f1c1cc42700 1 --2- 172.21.15.23:0/1525429342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1c18107eb0 0x7f1c1819f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:07.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.626+0000 7f1c1cc42700 1 --2- 172.21.15.23:0/1525429342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c18102c50 0x7f1c18198380 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:07.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.626+0000 7f1c1cc42700 1 --2- 172.21.15.23:0/1525429342 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c18105580 0x7f1c181988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:07.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.626+0000 7f1c1cc42700 1 -- 172.21.15.23:0/1525429342 >> 172.21.15.23:0/1525429342 conn(0x7f1c180fc680 msgr2=0x7f1c18103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:07.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.626+0000 7f1c1cc42700 1 -- 172.21.15.23:0/1525429342 shutdown_connections 2024-03-28T05:23:07.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:07.626+0000 7f1c1cc42700 1 -- 172.21.15.23:0/1525429342 wait complete. 2024-03-28T05:23:08.391 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-28T05:23:08.636 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:23:08.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:08 smithi113 ceph-mon[37164]: pgmap v350: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:23:08.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:08 smithi113 ceph-mon[37164]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:08.999 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:08 smithi023 ceph-mon[31954]: pgmap v350: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:23:08.999 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:08 smithi023 ceph-mon[31954]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:09.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:08 smithi098 ceph-mon[36087]: pgmap v350: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:23:09.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:08 smithi098 ceph-mon[36087]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:09.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-28T05:23:09.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-28T05:23:09.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:09 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-03-28T05:23:10.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-28T05:23:10.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-28T05:23:10.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:09 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-03-28T05:23:10.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2024-03-28T05:23:10.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-28T05:23:10.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:09 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]: dispatch 2024-03-28T05:23:10.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.390+0000 7f23e5da3700 1 -- 172.21.15.23:0/1789409808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e01033e0 msgr2=0x7f23e0103840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:10.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.390+0000 7f23e5da3700 1 --2- 172.21.15.23:0/1789409808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e01033e0 0x7f23e0103840 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f23c80099c0 tx=0x7f23c8009cd0 comp rx=0 tx=0).stop 2024-03-28T05:23:10.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.390+0000 7f23e5da3700 1 -- 172.21.15.23:0/1789409808 shutdown_connections 2024-03-28T05:23:10.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.390+0000 7f23e5da3700 1 --2- 172.21.15.23:0/1789409808 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23e0103d80 0x7f23e010a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:10.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.390+0000 7f23e5da3700 1 --2- 172.21.15.23:0/1789409808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e01021e0 0x7f23e0102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:10.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.390+0000 7f23e5da3700 1 --2- 172.21.15.23:0/1789409808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e01033e0 0x7f23e0103840 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:10.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.390+0000 7f23e5da3700 1 -- 172.21.15.23:0/1789409808 >> 172.21.15.23:0/1789409808 conn(0x7f23e00fd7a0 msgr2=0x7f23e00ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:10.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.390+0000 7f23e5da3700 1 -- 172.21.15.23:0/1789409808 shutdown_connections 2024-03-28T05:23:10.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.390+0000 7f23e5da3700 1 -- 172.21.15.23:0/1789409808 wait complete. 2024-03-28T05:23:10.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.391+0000 7f23e5da3700 1 Processor -- start 2024-03-28T05:23:10.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.391+0000 7f23e5da3700 1 -- start start 2024-03-28T05:23:10.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.391+0000 7f23e5da3700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23e01021e0 0x7f23e0198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:10.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.391+0000 7f23e5da3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e01033e0 0x7f23e0198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:10.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.391+0000 7f23e5da3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e0103d80 0x7f23e019f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:10.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.391+0000 7f23e5da3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e010dda0 con 0x7f23e0103d80 2024-03-28T05:23:10.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.391+0000 7f23e5da3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e010dc20 con 0x7f23e01033e0 2024-03-28T05:23:10.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.391+0000 7f23e5da3700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e010df20 con 0x7f23e01021e0 2024-03-28T05:23:10.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.391+0000 7f23dffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e0103d80 0x7f23e019f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:10.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.391+0000 7f23df7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23e01021e0 0x7f23e0198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:10.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.391+0000 7f23dffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e0103d80 0x7f23e019f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46130/0 (socket says 172.21.15.23:46130) 2024-03-28T05:23:10.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.391+0000 7f23dffff700 1 -- 172.21.15.23:0/3397113 learned_addr learned my addr 172.21.15.23:0/3397113 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:10.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.391+0000 7f23df7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23e01021e0 0x7f23e0198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:46966/0 (socket says 172.21.15.23:46966) 2024-03-28T05:23:10.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.391+0000 7f23deffd700 1 --2- 172.21.15.23:0/3397113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e01033e0 0x7f23e0198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:10.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.391+0000 7f23df7fe700 1 -- 172.21.15.23:0/3397113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e01033e0 msgr2=0x7f23e0198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:10.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.391+0000 7f23df7fe700 1 --2- 172.21.15.23:0/3397113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e01033e0 0x7f23e0198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:10.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.391+0000 7f23df7fe700 1 -- 172.21.15.23:0/3397113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e0103d80 msgr2=0x7f23e019f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:10.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.391+0000 7f23df7fe700 1 --2- 172.21.15.23:0/3397113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e0103d80 0x7f23e019f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:10.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.391+0000 7f23df7fe700 1 -- 172.21.15.23:0/3397113 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e019ff60 con 0x7f23e01021e0 2024-03-28T05:23:10.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.392+0000 7f23dffff700 1 --2- 172.21.15.23:0/3397113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e0103d80 0x7f23e019f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:23:10.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.392+0000 7f23deffd700 1 --2- 172.21.15.23:0/3397113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e01033e0 0x7f23e0198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:23:10.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.392+0000 7f23df7fe700 1 --2- 172.21.15.23:0/3397113 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23e01021e0 0x7f23e0198300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f23d8007d70 tx=0x7f23d800a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:10.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.392+0000 7f23dcff9700 1 -- 172.21.15.23:0/3397113 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d8016020 con 0x7f23e01021e0 2024-03-28T05:23:10.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.392+0000 7f23dcff9700 1 -- 172.21.15.23:0/3397113 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23d800ad00 con 0x7f23e01021e0 2024-03-28T05:23:10.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.392+0000 7f23dcff9700 1 -- 172.21.15.23:0/3397113 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d80115c0 con 0x7f23e01021e0 2024-03-28T05:23:10.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.392+0000 7f23e5da3700 1 -- 172.21.15.23:0/3397113 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e01a0250 con 0x7f23e01021e0 2024-03-28T05:23:10.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.392+0000 7f23e5da3700 1 -- 172.21.15.23:0/3397113 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e01a0790 con 0x7f23e01021e0 2024-03-28T05:23:10.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.393+0000 7f23e5da3700 1 -- 172.21.15.23:0/3397113 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e0068e50 con 0x7f23e01021e0 2024-03-28T05:23:10.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.394+0000 7f23dcff9700 1 -- 172.21.15.23:0/3397113 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f23d8019020 con 0x7f23e01021e0 2024-03-28T05:23:10.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.395+0000 7f23dcff9700 1 --2- 172.21.15.23:0/3397113 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f23cc0a2890 0x7f23cc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:10.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.395+0000 7f23deffd700 1 --2- 172.21.15.23:0/3397113 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f23cc0a2890 0x7f23cc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:10.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.395+0000 7f23dcff9700 1 -- 172.21.15.23:0/3397113 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f23d80cee60 con 0x7f23e01021e0 2024-03-28T05:23:10.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.395+0000 7f23deffd700 1 --2- 172.21.15.23:0/3397113 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f23cc0a2890 0x7f23cc0a4d50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f23c8000c00 tx=0x7f23c8003860 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:10.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.403+0000 7f23dcff9700 1 -- 172.21.15.23:0/3397113 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f23d8020030 con 0x7f23e01021e0 2024-03-28T05:23:10.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.615+0000 7f23e5da3700 1 -- 172.21.15.23:0/3397113 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f23e00625d0 con 0x7f23cc0a2890 2024-03-28T05:23:10.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.617+0000 7f23dcff9700 1 -- 172.21.15.23:0/3397113 <== mgr.14150 v2:172.21.15.23:6800/222178108 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7f23e00625d0 con 0x7f23cc0a2890 2024-03-28T05:23:10.615 INFO:teuthology.orchestra.run.smithi023.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-28T05:23:10.615 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736103B14P0KGN 3726G No 3m ago Has a FileSystem, LVM detected 2024-03-28T05:23:10.616 INFO:teuthology.orchestra.run.smithi023.stdout:smithi098 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204010H400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-28T05:23:10.616 INFO:teuthology.orchestra.run.smithi023.stdout:smithi113 /dev/nvme0n1 ssd INTEL_SSDPEDMX020T4_CVPD5171000M2P0SGN 1863G No 2m ago Has a FileSystem, LVM detected 2024-03-28T05:23:10.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.619+0000 7f23e5da3700 1 -- 172.21.15.23:0/3397113 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f23cc0a2890 msgr2=0x7f23cc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:10.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.619+0000 7f23e5da3700 1 --2- 172.21.15.23:0/3397113 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f23cc0a2890 0x7f23cc0a4d50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f23c8000c00 tx=0x7f23c8003860 comp rx=0 tx=0).stop 2024-03-28T05:23:10.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.619+0000 7f23e5da3700 1 -- 172.21.15.23:0/3397113 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23e01021e0 msgr2=0x7f23e0198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:10.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.619+0000 7f23e5da3700 1 --2- 172.21.15.23:0/3397113 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23e01021e0 0x7f23e0198300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f23d8007d70 tx=0x7f23d800a340 comp rx=0 tx=0).stop 2024-03-28T05:23:10.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.619+0000 7f23e5da3700 1 -- 172.21.15.23:0/3397113 shutdown_connections 2024-03-28T05:23:10.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.619+0000 7f23e5da3700 1 --2- 172.21.15.23:0/3397113 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f23cc0a2890 0x7f23cc0a4d50 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:10.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.619+0000 7f23e5da3700 1 --2- 172.21.15.23:0/3397113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e0103d80 0x7f23e019f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:10.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.619+0000 7f23e5da3700 1 --2- 172.21.15.23:0/3397113 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f23e01021e0 0x7f23e0198300 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:10.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.619+0000 7f23e5da3700 1 --2- 172.21.15.23:0/3397113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23e01033e0 0x7f23e0198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:10.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.619+0000 7f23e5da3700 1 -- 172.21.15.23:0/3397113 >> 172.21.15.23:0/3397113 conn(0x7f23e00fd7a0 msgr2=0x7f23e0106610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:10.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.619+0000 7f23e5da3700 1 -- 172.21.15.23:0/3397113 shutdown_connections 2024-03-28T05:23:10.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:10.619+0000 7f23e5da3700 1 -- 172.21.15.23:0/3397113 wait complete. 2024-03-28T05:23:10.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:10 smithi113 ceph-mon[37164]: pgmap v351: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:23:10.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-28T05:23:10.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-28T05:23:10.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:10 smithi113 ceph-mon[37164]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]': finished 2024-03-28T05:23:10.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:10 smithi113 ceph-mon[37164]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T05:23:10.997 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:10 smithi023 ceph-mon[31954]: pgmap v351: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:23:10.998 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-28T05:23:10.998 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-28T05:23:10.998 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:10 smithi023 ceph-mon[31954]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]': finished 2024-03-28T05:23:10.998 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:10 smithi023 ceph-mon[31954]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T05:23:11.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:10 smithi098 ceph-mon[36087]: pgmap v351: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:23:11.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2024-03-28T05:23:11.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-28T05:23:11.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:10 smithi098 ceph-mon[36087]: from='mgr.14150 172.21.15.23:0/2425306588' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 4]}]': finished 2024-03-28T05:23:11.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:10 smithi098 ceph-mon[36087]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T05:23:11.423 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-28T05:23:11.433 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-28T05:23:11.433 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-28T05:23:11.686 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:23:11.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:11 smithi113 ceph-mon[37164]: from='client.25102 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:11.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:11 smithi113 ceph-mon[37164]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T05:23:12.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:11 smithi023 ceph-mon[31954]: from='client.25102 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:12.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:11 smithi023 ceph-mon[31954]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T05:23:12.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:11 smithi098 ceph-mon[36087]: from='client.25102 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:12.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:11 smithi098 ceph-mon[36087]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T05:23:12.780 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:12 smithi023 ceph-mon[31954]: pgmap v354: 129 pgs: 1 activating, 128 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:23:12.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:12 smithi113 ceph-mon[37164]: pgmap v354: 129 pgs: 1 activating, 128 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:23:13.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:12 smithi098 ceph-mon[36087]: pgmap v354: 129 pgs: 1 activating, 128 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:23:13.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.419+0000 7f2019fec700 1 -- 172.21.15.23:0/198738191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20141033e0 msgr2=0x7f2014103840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:13.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.419+0000 7f2019fec700 1 --2- 172.21.15.23:0/198738191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20141033e0 0x7f2014103840 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f20080099c0 tx=0x7f2008009cd0 comp rx=0 tx=0).stop 2024-03-28T05:23:13.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.419+0000 7f2019fec700 1 -- 172.21.15.23:0/198738191 shutdown_connections 2024-03-28T05:23:13.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.419+0000 7f2019fec700 1 --2- 172.21.15.23:0/198738191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2014103d80 0x7f201410a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:13.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.419+0000 7f2019fec700 1 --2- 172.21.15.23:0/198738191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f20141021e0 0x7f2014102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:13.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.419+0000 7f2019fec700 1 --2- 172.21.15.23:0/198738191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20141033e0 0x7f2014103840 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:13.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.419+0000 7f2019fec700 1 -- 172.21.15.23:0/198738191 >> 172.21.15.23:0/198738191 conn(0x7f20140fd7a0 msgr2=0x7f20140ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:13.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.419+0000 7f2019fec700 1 -- 172.21.15.23:0/198738191 shutdown_connections 2024-03-28T05:23:13.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.419+0000 7f2019fec700 1 -- 172.21.15.23:0/198738191 wait complete. 2024-03-28T05:23:13.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.420+0000 7f2019fec700 1 Processor -- start 2024-03-28T05:23:13.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.420+0000 7f2019fec700 1 -- start start 2024-03-28T05:23:13.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.420+0000 7f2019fec700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20141021e0 0x7f2014075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:13.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.420+0000 7f2019fec700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20141033e0 0x7f2014074580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:13.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.420+0000 7f2019fec700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2014103d80 0x7f2014074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:13.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.420+0000 7f2019fec700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201410dda0 con 0x7f20141021e0 2024-03-28T05:23:13.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.420+0000 7f2019fec700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201410dc20 con 0x7f20141033e0 2024-03-28T05:23:13.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.420+0000 7f2019fec700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201410df20 con 0x7f2014103d80 2024-03-28T05:23:13.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.421+0000 7f20137fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20141021e0 0x7f2014075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:13.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.421+0000 7f20137fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20141021e0 0x7f2014075f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43266/0 (socket says 172.21.15.23:43266) 2024-03-28T05:23:13.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.421+0000 7f20137fe700 1 -- 172.21.15.23:0/2238891631 learned_addr learned my addr 172.21.15.23:0/2238891631 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:13.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.421+0000 7f2013fff700 1 --2- 172.21.15.23:0/2238891631 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2014103d80 0x7f2014074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:13.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.421+0000 7f2012ffd700 1 --2- 172.21.15.23:0/2238891631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20141033e0 0x7f2014074580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:13.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.421+0000 7f20137fe700 1 -- 172.21.15.23:0/2238891631 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2014103d80 msgr2=0x7f2014074ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:13.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.421+0000 7f20137fe700 1 --2- 172.21.15.23:0/2238891631 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2014103d80 0x7f2014074ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:13.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.421+0000 7f20137fe700 1 -- 172.21.15.23:0/2238891631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20141033e0 msgr2=0x7f2014074580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:13.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.421+0000 7f20137fe700 1 --2- 172.21.15.23:0/2238891631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20141033e0 0x7f2014074580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:13.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.421+0000 7f20137fe700 1 -- 172.21.15.23:0/2238891631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2014075370 con 0x7f20141021e0 2024-03-28T05:23:13.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.421+0000 7f2013fff700 1 --2- 172.21.15.23:0/2238891631 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2014103d80 0x7f2014074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:23:13.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.421+0000 7f20137fe700 1 --2- 172.21.15.23:0/2238891631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20141021e0 0x7f2014075f30 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f200400e970 tx=0x7f200400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:13.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.421+0000 7f2010ff9700 1 -- 172.21.15.23:0/2238891631 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200400cbd0 con 0x7f20141021e0 2024-03-28T05:23:13.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.421+0000 7f2010ff9700 1 -- 172.21.15.23:0/2238891631 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2004004500 con 0x7f20141021e0 2024-03-28T05:23:13.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.421+0000 7f2019fec700 1 -- 172.21.15.23:0/2238891631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20141a2510 con 0x7f20141021e0 2024-03-28T05:23:13.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.421+0000 7f2010ff9700 1 -- 172.21.15.23:0/2238891631 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2004003e30 con 0x7f20141021e0 2024-03-28T05:23:13.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.421+0000 7f2019fec700 1 -- 172.21.15.23:0/2238891631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20141a2a20 con 0x7f20141021e0 2024-03-28T05:23:13.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.422+0000 7f2019fec700 1 -- 172.21.15.23:0/2238891631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2014068e50 con 0x7f20141021e0 2024-03-28T05:23:13.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.423+0000 7f2010ff9700 1 -- 172.21.15.23:0/2238891631 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2004004000 con 0x7f20141021e0 2024-03-28T05:23:13.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.424+0000 7f2010ff9700 1 --2- 172.21.15.23:0/2238891631 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1ffc0a2890 0x7f1ffc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:13.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.424+0000 7f2012ffd700 1 --2- 172.21.15.23:0/2238891631 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1ffc0a2890 0x7f1ffc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:13.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.424+0000 7f2010ff9700 1 -- 172.21.15.23:0/2238891631 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f20040cf230 con 0x7f20141021e0 2024-03-28T05:23:13.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.424+0000 7f2012ffd700 1 --2- 172.21.15.23:0/2238891631 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1ffc0a2890 0x7f1ffc0a4d50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2008000c00 tx=0x7f2008005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:13.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.428+0000 7f2010ff9700 1 -- 172.21.15.23:0/2238891631 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f200401f340 con 0x7f20141021e0 2024-03-28T05:23:13.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.680+0000 7f2019fec700 1 -- 172.21.15.23:0/2238891631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2014062da0 con 0x7f20141021e0 2024-03-28T05:23:13.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.680+0000 7f2010ff9700 1 -- 172.21.15.23:0/2238891631 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) v1 ==== 76+0+2948 (secure 0 0 0) 0x7f200409bf20 con 0x7f20141021e0 2024-03-28T05:23:13.679 INFO:teuthology.orchestra.run.smithi023.stdout:e20 2024-03-28T05:23:13.679 INFO:teuthology.orchestra.run.smithi023.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T05:23:13.679 INFO:teuthology.orchestra.run.smithi023.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T05:23:13.679 INFO:teuthology.orchestra.run.smithi023.stdout:legacy client fscid: 1 2024-03-28T05:23:13.679 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:23:13.679 INFO:teuthology.orchestra.run.smithi023.stdout:Filesystem 'cephfs' (1) 2024-03-28T05:23:13.679 INFO:teuthology.orchestra.run.smithi023.stdout:fs_name cephfs 2024-03-28T05:23:13.679 INFO:teuthology.orchestra.run.smithi023.stdout:epoch 20 2024-03-28T05:23:13.679 INFO:teuthology.orchestra.run.smithi023.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-28T05:23:13.679 INFO:teuthology.orchestra.run.smithi023.stdout:created 2024-03-28T05:21:07.616553+0000 2024-03-28T05:23:13.680 INFO:teuthology.orchestra.run.smithi023.stdout:modified 2024-03-28T05:21:32.732132+0000 2024-03-28T05:23:13.680 INFO:teuthology.orchestra.run.smithi023.stdout:tableserver 0 2024-03-28T05:23:13.680 INFO:teuthology.orchestra.run.smithi023.stdout:root 0 2024-03-28T05:23:13.680 INFO:teuthology.orchestra.run.smithi023.stdout:session_timeout 300 2024-03-28T05:23:13.680 INFO:teuthology.orchestra.run.smithi023.stdout:session_autoclose 300 2024-03-28T05:23:13.680 INFO:teuthology.orchestra.run.smithi023.stdout:max_file_size 1099511627776 2024-03-28T05:23:13.680 INFO:teuthology.orchestra.run.smithi023.stdout:max_xattr_size 65536 2024-03-28T05:23:13.680 INFO:teuthology.orchestra.run.smithi023.stdout:required_client_features {} 2024-03-28T05:23:13.680 INFO:teuthology.orchestra.run.smithi023.stdout:last_failure 0 2024-03-28T05:23:13.680 INFO:teuthology.orchestra.run.smithi023.stdout:last_failure_osd_epoch 0 2024-03-28T05:23:13.680 INFO:teuthology.orchestra.run.smithi023.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T05:23:13.680 INFO:teuthology.orchestra.run.smithi023.stdout:max_mds 1 2024-03-28T05:23:13.680 INFO:teuthology.orchestra.run.smithi023.stdout:in 0 2024-03-28T05:23:13.680 INFO:teuthology.orchestra.run.smithi023.stdout:up {0=24487} 2024-03-28T05:23:13.680 INFO:teuthology.orchestra.run.smithi023.stdout:failed 2024-03-28T05:23:13.681 INFO:teuthology.orchestra.run.smithi023.stdout:damaged 2024-03-28T05:23:13.681 INFO:teuthology.orchestra.run.smithi023.stdout:stopped 2024-03-28T05:23:13.681 INFO:teuthology.orchestra.run.smithi023.stdout:data_pools [3,4] 2024-03-28T05:23:13.681 INFO:teuthology.orchestra.run.smithi023.stdout:metadata_pool 2 2024-03-28T05:23:13.681 INFO:teuthology.orchestra.run.smithi023.stdout:inline_data disabled 2024-03-28T05:23:13.681 INFO:teuthology.orchestra.run.smithi023.stdout:balancer 2024-03-28T05:23:13.681 INFO:teuthology.orchestra.run.smithi023.stdout:bal_rank_mask -1 2024-03-28T05:23:13.681 INFO:teuthology.orchestra.run.smithi023.stdout:standby_count_wanted 0 2024-03-28T05:23:13.681 INFO:teuthology.orchestra.run.smithi023.stdout:qdb_cluster leader: 24487 members: 24487 2024-03-28T05:23:13.681 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.i{0:24487} state up:active seq 5 addr [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T05:23:13.681 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.b{0:24473} state up:standby-replay seq 1 addr [v2:172.21.15.98:6838/144113007,v1:172.21.15.98:6839/144113007] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T05:23:13.681 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:23:13.681 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:23:13.681 INFO:teuthology.orchestra.run.smithi023.stdout:Standby daemons: 2024-03-28T05:23:13.681 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:23:13.681 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.j{-1:14553} state up:standby seq 1 addr [v2:172.21.15.23:6834/733913400,v1:172.21.15.23:6835/733913400] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T05:23:13.681 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.d{-1:14565} state up:standby seq 1 addr [v2:172.21.15.23:6836/1972901387,v1:172.21.15.23:6837/1972901387] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T05:23:13.682 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.g{-1:14640} state up:standby seq 1 addr [v2:172.21.15.23:6838/2390673788,v1:172.21.15.23:6839/2390673788] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T05:23:13.682 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.a{-1:14652} state up:standby seq 1 addr [v2:172.21.15.23:6840/289263156,v1:172.21.15.23:6841/289263156] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T05:23:13.682 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.k{-1:24389} state up:standby seq 1 addr [v2:172.21.15.98:6832/1563379945,v1:172.21.15.98:6833/1563379945] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T05:23:13.682 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.l{-1:24406} state up:standby seq 1 addr [v2:172.21.15.113:6832/3436936808,v1:172.21.15.113:6833/3436936808] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T05:23:13.682 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.e{-1:24413} state up:standby seq 1 addr [v2:172.21.15.98:6834/3313932732,v1:172.21.15.98:6835/3313932732] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T05:23:13.682 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.f{-1:24445} state up:standby seq 1 addr [v2:172.21.15.113:6834/2695340024,v1:172.21.15.113:6835/2695340024] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T05:23:13.682 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.h{-1:24446} state up:standby seq 1 addr [v2:172.21.15.98:6836/2750367469,v1:172.21.15.98:6837/2750367469] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T05:23:13.682 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.c{-1:24457} state up:standby seq 1 addr [v2:172.21.15.113:6836/3222726274,v1:172.21.15.113:6837/3222726274] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T05:23:13.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.682+0000 7f2019fec700 1 -- 172.21.15.23:0/2238891631 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1ffc0a2890 msgr2=0x7f1ffc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:13.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.682+0000 7f2019fec700 1 --2- 172.21.15.23:0/2238891631 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1ffc0a2890 0x7f1ffc0a4d50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2008000c00 tx=0x7f2008005fd0 comp rx=0 tx=0).stop 2024-03-28T05:23:13.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.682+0000 7f2019fec700 1 -- 172.21.15.23:0/2238891631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20141021e0 msgr2=0x7f2014075f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:13.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.682+0000 7f2019fec700 1 --2- 172.21.15.23:0/2238891631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20141021e0 0x7f2014075f30 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f200400e970 tx=0x7f200400ec80 comp rx=0 tx=0).stop 2024-03-28T05:23:13.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.682+0000 7f2019fec700 1 -- 172.21.15.23:0/2238891631 shutdown_connections 2024-03-28T05:23:13.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.682+0000 7f2019fec700 1 --2- 172.21.15.23:0/2238891631 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f1ffc0a2890 0x7f1ffc0a4d50 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:13.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.682+0000 7f2019fec700 1 --2- 172.21.15.23:0/2238891631 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2014103d80 0x7f2014074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:13.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.682+0000 7f2019fec700 1 --2- 172.21.15.23:0/2238891631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20141021e0 0x7f2014075f30 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:13.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.682+0000 7f2019fec700 1 --2- 172.21.15.23:0/2238891631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20141033e0 0x7f2014074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:13.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.682+0000 7f2019fec700 1 -- 172.21.15.23:0/2238891631 >> 172.21.15.23:0/2238891631 conn(0x7f20140fd7a0 msgr2=0x7f20141066e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:13.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.682+0000 7f2019fec700 1 -- 172.21.15.23:0/2238891631 shutdown_connections 2024-03-28T05:23:13.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:13.682+0000 7f2019fec700 1 -- 172.21.15.23:0/2238891631 wait complete. 2024-03-28T05:23:13.683 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 20 2024-03-28T05:23:13.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:13 smithi113 ceph-mon[37164]: pgmap v355: 129 pgs: 1 activating, 128 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-03-28T05:23:13.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:13 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2238891631' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T05:23:14.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:13 smithi023 ceph-mon[31954]: pgmap v355: 129 pgs: 1 activating, 128 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-03-28T05:23:14.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:13 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2238891631' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T05:23:14.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:13 smithi098 ceph-mon[36087]: pgmap v355: 129 pgs: 1 activating, 128 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-03-28T05:23:14.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:13 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2238891631' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T05:23:14.410 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-28T05:23:14.662 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:23:16.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:15 smithi023 ceph-mon[31954]: pgmap v356: 129 pgs: 1 activating, 128 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 72 B/s, 0 objects/s recovering 2024-03-28T05:23:16.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:15 smithi098 ceph-mon[36087]: pgmap v356: 129 pgs: 1 activating, 128 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 72 B/s, 0 objects/s recovering 2024-03-28T05:23:16.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:15 smithi113 ceph-mon[37164]: pgmap v356: 129 pgs: 1 activating, 128 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 72 B/s, 0 objects/s recovering 2024-03-28T05:23:16.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.303+0000 7f4636470700 1 -- 172.21.15.23:0/2663866869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46301021c0 msgr2=0x7f46301025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:16.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.303+0000 7f4636470700 1 --2- 172.21.15.23:0/2663866869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46301021c0 0x7f46301025e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f46200099c0 tx=0x7f4620009cd0 comp rx=0 tx=0).stop 2024-03-28T05:23:16.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.303+0000 7f4636470700 1 -- 172.21.15.23:0/2663866869 shutdown_connections 2024-03-28T05:23:16.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.303+0000 7f4636470700 1 --2- 172.21.15.23:0/2663866869 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4630103d60 0x7f463010a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:16.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.303+0000 7f4636470700 1 --2- 172.21.15.23:0/2663866869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46301021c0 0x7f46301025e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:16.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.303+0000 7f4636470700 1 --2- 172.21.15.23:0/2663866869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46301033c0 0x7f4630103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:16.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.303+0000 7f4636470700 1 -- 172.21.15.23:0/2663866869 >> 172.21.15.23:0/2663866869 conn(0x7f46300fd7a0 msgr2=0x7f46300ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:16.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.303+0000 7f4636470700 1 -- 172.21.15.23:0/2663866869 shutdown_connections 2024-03-28T05:23:16.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.303+0000 7f4636470700 1 -- 172.21.15.23:0/2663866869 wait complete. 2024-03-28T05:23:16.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.304+0000 7f4636470700 1 Processor -- start 2024-03-28T05:23:16.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.304+0000 7f4636470700 1 -- start start 2024-03-28T05:23:16.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.304+0000 7f4636470700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46301021c0 0x7f46301984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:16.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.304+0000 7f4636470700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46301033c0 0x7f4630198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:16.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.304+0000 7f4636470700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4630103d60 0x7f463019f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:16.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.304+0000 7f4636470700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463010de00 con 0x7f46301033c0 2024-03-28T05:23:16.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.304+0000 7f4636470700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463010dc80 con 0x7f46301021c0 2024-03-28T05:23:16.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.304+0000 7f4636470700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463010df80 con 0x7f4630103d60 2024-03-28T05:23:16.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.304+0000 7f462f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46301033c0 0x7f4630198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:16.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.304+0000 7f4634a0d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4630103d60 0x7f463019f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:16.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.304+0000 7f462f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46301033c0 0x7f4630198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43276/0 (socket says 172.21.15.23:43276) 2024-03-28T05:23:16.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.304+0000 7f462f7fe700 1 -- 172.21.15.23:0/3368550174 learned_addr learned my addr 172.21.15.23:0/3368550174 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:16.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.304+0000 7f462ffff700 1 --2- 172.21.15.23:0/3368550174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46301021c0 0x7f46301984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:16.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.305+0000 7f4634a0d700 1 -- 172.21.15.23:0/3368550174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46301021c0 msgr2=0x7f46301984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:16.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.305+0000 7f4634a0d700 1 --2- 172.21.15.23:0/3368550174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46301021c0 0x7f46301984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:16.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.305+0000 7f4634a0d700 1 -- 172.21.15.23:0/3368550174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46301033c0 msgr2=0x7f4630198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:16.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.305+0000 7f4634a0d700 1 --2- 172.21.15.23:0/3368550174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46301033c0 0x7f4630198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:16.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.305+0000 7f4634a0d700 1 -- 172.21.15.23:0/3368550174 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46301a0090 con 0x7f4630103d60 2024-03-28T05:23:16.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.305+0000 7f4634a0d700 1 --2- 172.21.15.23:0/3368550174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4630103d60 0x7f463019f9f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4628009f60 tx=0x7f4628009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:16.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.305+0000 7f462d7fa700 1 -- 172.21.15.23:0/3368550174 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46280092e0 con 0x7f4630103d60 2024-03-28T05:23:16.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.305+0000 7f462d7fa700 1 -- 172.21.15.23:0/3368550174 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4628004bb0 con 0x7f4630103d60 2024-03-28T05:23:16.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.305+0000 7f4636470700 1 -- 172.21.15.23:0/3368550174 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46301a0320 con 0x7f4630103d60 2024-03-28T05:23:16.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.305+0000 7f462d7fa700 1 -- 172.21.15.23:0/3368550174 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4628004280 con 0x7f4630103d60 2024-03-28T05:23:16.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.305+0000 7f4636470700 1 -- 172.21.15.23:0/3368550174 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46301a08b0 con 0x7f4630103d60 2024-03-28T05:23:16.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.306+0000 7f4636470700 1 -- 172.21.15.23:0/3368550174 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4630068e50 con 0x7f4630103d60 2024-03-28T05:23:16.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.307+0000 7f462d7fa700 1 -- 172.21.15.23:0/3368550174 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f462801b030 con 0x7f4630103d60 2024-03-28T05:23:16.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.308+0000 7f462d7fa700 1 --2- 172.21.15.23:0/3368550174 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f46180a2910 0x7f46180a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:16.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.308+0000 7f462ffff700 1 --2- 172.21.15.23:0/3368550174 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f46180a2910 0x7f46180a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:16.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.308+0000 7f462d7fa700 1 -- 172.21.15.23:0/3368550174 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f46280ce960 con 0x7f4630103d60 2024-03-28T05:23:16.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.308+0000 7f462ffff700 1 --2- 172.21.15.23:0/3368550174 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f46180a2910 0x7f46180a4dd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f462000b580 tx=0x7f4620005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:16.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.311+0000 7f462d7fa700 1 -- 172.21.15.23:0/3368550174 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4628022020 con 0x7f4630103d60 2024-03-28T05:23:16.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.514+0000 7f4636470700 1 -- 172.21.15.23:0/3368550174 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f463002da60 con 0x7f4630103d60 2024-03-28T05:23:16.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.515+0000 7f462d7fa700 1 -- 172.21.15.23:0/3368550174 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) v1 ==== 56+0+5667 (secure 0 0 0) 0x7f4628010040 con 0x7f4630103d60 2024-03-28T05:23:16.515 INFO:teuthology.orchestra.run.smithi023.stdout:epoch 93 2024-03-28T05:23:16.515 INFO:teuthology.orchestra.run.smithi023.stdout:fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:23:16.515 INFO:teuthology.orchestra.run.smithi023.stdout:created 2024-03-28T05:13:13.453653+0000 2024-03-28T05:23:16.515 INFO:teuthology.orchestra.run.smithi023.stdout:modified 2024-03-28T05:23:10.695543+0000 2024-03-28T05:23:16.515 INFO:teuthology.orchestra.run.smithi023.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-28T05:23:16.515 INFO:teuthology.orchestra.run.smithi023.stdout:crush_version 27 2024-03-28T05:23:16.515 INFO:teuthology.orchestra.run.smithi023.stdout:full_ratio 0.95 2024-03-28T05:23:16.515 INFO:teuthology.orchestra.run.smithi023.stdout:backfillfull_ratio 0.9 2024-03-28T05:23:16.515 INFO:teuthology.orchestra.run.smithi023.stdout:nearfull_ratio 0.85 2024-03-28T05:23:16.515 INFO:teuthology.orchestra.run.smithi023.stdout:require_min_compat_client luminous 2024-03-28T05:23:16.515 INFO:teuthology.orchestra.run.smithi023.stdout:min_compat_client luminous 2024-03-28T05:23:16.515 INFO:teuthology.orchestra.run.smithi023.stdout:require_osd_release squid 2024-03-28T05:23:16.515 INFO:teuthology.orchestra.run.smithi023.stdout:stretch_mode_enabled false 2024-03-28T05:23:16.515 INFO:teuthology.orchestra.run.smithi023.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-28T05:23:16.515 INFO:teuthology.orchestra.run.smithi023.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-03-28T05:23:16.516 INFO:teuthology.orchestra.run.smithi023.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-28T05:23:16.516 INFO:teuthology.orchestra.run.smithi023.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-28T05:23:16.516 INFO:teuthology.orchestra.run.smithi023.stdout:max_osd 12 2024-03-28T05:23:16.516 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6802/1219956881,v1:172.21.15.23:6803/1219956881] [v2:172.21.15.23:6804/1219956881,v1:172.21.15.23:6805/1219956881] exists,up 18150942-59a4-4c3b-83d4-c1fc3ccf1c09 2024-03-28T05:23:16.516 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6800/555491234,v1:172.21.15.98:6801/555491234] [v2:172.21.15.98:6802/555491234,v1:172.21.15.98:6803/555491234] exists,up ba6d1a95-5135-4670-95f9-7878c0e7bd64 2024-03-28T05:23:16.516 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2 up in weight 1 up_from 18 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.113:6800/1535856914,v1:172.21.15.113:6801/1535856914] [v2:172.21.15.113:6802/1535856914,v1:172.21.15.113:6803/1535856914] exists,up 873bdd60-2a47-403b-b1f4-ee9226a95067 2024-03-28T05:23:16.516 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6810/2949415020,v1:172.21.15.23:6811/2949415020] [v2:172.21.15.23:6812/2949415020,v1:172.21.15.23:6813/2949415020] exists,up 83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6 2024-03-28T05:23:16.516 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4 up in weight 1 up_from 33 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6808/2615902819,v1:172.21.15.98:6809/2615902819] [v2:172.21.15.98:6810/2615902819,v1:172.21.15.98:6811/2615902819] exists,up 76a90bac-b221-40a3-8238-40e8049cec36 2024-03-28T05:23:16.516 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5 up in weight 1 up_from 38 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.113:6808/625271528,v1:172.21.15.113:6809/625271528] [v2:172.21.15.113:6810/625271528,v1:172.21.15.113:6811/625271528] exists,up b53fdb55-a8df-40a8-bd1b-8bcef634e26a 2024-03-28T05:23:16.516 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6 up in weight 1 up_from 46 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6818/4188464176,v1:172.21.15.23:6819/4188464176] [v2:172.21.15.23:6820/4188464176,v1:172.21.15.23:6821/4188464176] exists,up f00283d2-8d67-4e97-a575-9a9d3b55d5cc 2024-03-28T05:23:16.516 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7 up in weight 1 up_from 51 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6816/1985546028,v1:172.21.15.98:6817/1985546028] [v2:172.21.15.98:6818/1985546028,v1:172.21.15.98:6819/1985546028] exists,up edc008a7-b07b-40a8-88b7-1614ba38a775 2024-03-28T05:23:16.516 INFO:teuthology.orchestra.run.smithi023.stdout:osd.8 up in weight 1 up_from 59 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.113:6816/3534876081,v1:172.21.15.113:6817/3534876081] [v2:172.21.15.113:6818/3534876081,v1:172.21.15.113:6819/3534876081] exists,up 56e7b2d1-81f0-47b2-8638-f29832bed8de 2024-03-28T05:23:16.516 INFO:teuthology.orchestra.run.smithi023.stdout:osd.9 up in weight 1 up_from 67 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6826/672264207,v1:172.21.15.23:6827/672264207] [v2:172.21.15.23:6828/672264207,v1:172.21.15.23:6829/672264207] exists,up a2e90caa-6e9c-42bf-9f18-81f7f47880cd 2024-03-28T05:23:16.516 INFO:teuthology.orchestra.run.smithi023.stdout:osd.10 up in weight 1 up_from 72 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6824/2699920595,v1:172.21.15.98:6825/2699920595] [v2:172.21.15.98:6826/2699920595,v1:172.21.15.98:6827/2699920595] exists,up 0eda917f-020c-437f-95ae-b26203f6c68e 2024-03-28T05:23:16.516 INFO:teuthology.orchestra.run.smithi023.stdout:osd.11 up in weight 1 up_from 80 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.113:6824/2113644020,v1:172.21.15.113:6825/2113644020] [v2:172.21.15.113:6826/2113644020,v1:172.21.15.113:6827/2113644020] exists,up aae78501-78a8-469a-be20-3006f5ec5037 2024-03-28T05:23:16.516 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 2.4 [2,11] 2024-03-28T05:23:16.517 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 2.a [2,11] 2024-03-28T05:23:16.517 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 2.1f [8,4] 2024-03-28T05:23:16.517 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.12 [3,9] 2024-03-28T05:23:16.517 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.1d [4,8] 2024-03-28T05:23:16.517 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.1e [3,9] 2024-03-28T05:23:16.517 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.21 [4,7] 2024-03-28T05:23:16.517 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.24 [4,9] 2024-03-28T05:23:16.517 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.39 [5,6] 2024-03-28T05:23:16.517 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 4.5 [5,8] 2024-03-28T05:23:16.517 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 4.6 [9,11] 2024-03-28T05:23:16.517 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 4.12 [9,10] 2024-03-28T05:23:16.517 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 4.1c [9,8] 2024-03-28T05:23:16.517 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6801/1530849123 expires 2024-03-29T05:14:08.640340+0000 2024-03-28T05:23:16.517 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6800/1530849123 expires 2024-03-29T05:14:08.640340+0000 2024-03-28T05:23:16.517 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/1876451381 expires 2024-03-29T05:14:08.640340+0000 2024-03-28T05:23:16.517 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/2253559862 expires 2024-03-29T05:14:08.640340+0000 2024-03-28T05:23:16.518 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/2653562762 expires 2024-03-29T05:13:42.579607+0000 2024-03-28T05:23:16.518 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/3337463830 expires 2024-03-29T05:13:42.579607+0000 2024-03-28T05:23:16.518 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6800/1735221490 expires 2024-03-29T05:13:42.579607+0000 2024-03-28T05:23:16.518 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/2784771635 expires 2024-03-29T05:14:08.640340+0000 2024-03-28T05:23:16.518 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/2389902490 expires 2024-03-29T05:13:42.579607+0000 2024-03-28T05:23:16.518 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6801/1735221490 expires 2024-03-29T05:13:42.579607+0000 2024-03-28T05:23:16.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.517+0000 7f4636470700 1 -- 172.21.15.23:0/3368550174 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f46180a2910 msgr2=0x7f46180a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:16.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.517+0000 7f4636470700 1 --2- 172.21.15.23:0/3368550174 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f46180a2910 0x7f46180a4dd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f462000b580 tx=0x7f4620005fb0 comp rx=0 tx=0).stop 2024-03-28T05:23:16.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.517+0000 7f4636470700 1 -- 172.21.15.23:0/3368550174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4630103d60 msgr2=0x7f463019f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:16.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.517+0000 7f4636470700 1 --2- 172.21.15.23:0/3368550174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4630103d60 0x7f463019f9f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4628009f60 tx=0x7f4628009f90 comp rx=0 tx=0).stop 2024-03-28T05:23:16.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.517+0000 7f4636470700 1 -- 172.21.15.23:0/3368550174 shutdown_connections 2024-03-28T05:23:16.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.517+0000 7f4636470700 1 --2- 172.21.15.23:0/3368550174 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f46180a2910 0x7f46180a4dd0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:16.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.517+0000 7f4636470700 1 --2- 172.21.15.23:0/3368550174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4630103d60 0x7f463019f9f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:16.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.517+0000 7f4636470700 1 --2- 172.21.15.23:0/3368550174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46301021c0 0x7f46301984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:16.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.517+0000 7f4636470700 1 --2- 172.21.15.23:0/3368550174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46301033c0 0x7f4630198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:16.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.517+0000 7f4636470700 1 -- 172.21.15.23:0/3368550174 >> 172.21.15.23:0/3368550174 conn(0x7f46300fd7a0 msgr2=0x7f46301066e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:16.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.517+0000 7f4636470700 1 -- 172.21.15.23:0/3368550174 shutdown_connections 2024-03-28T05:23:16.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:16.517+0000 7f4636470700 1 -- 172.21.15.23:0/3368550174 wait complete. 2024-03-28T05:23:17.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:16 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3368550174' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T05:23:17.118 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:16 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3368550174' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T05:23:17.149 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-28T05:23:17.160 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T05:23:17.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:16 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3368550174' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T05:23:17.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.271+0000 7f61a4332640 1 -- 172.21.15.23:0/1071792980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f619c1005c0 msgr2=0x7f619c1009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:17.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.271+0000 7f61a4332640 1 --2- 172.21.15.23:0/1071792980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f619c1005c0 0x7f619c1009e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6194009900 tx=0x7f619402bc30 comp rx=0 tx=0).stop 2024-03-28T05:23:17.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.271+0000 7f61a4332640 1 -- 172.21.15.23:0/1071792980 shutdown_connections 2024-03-28T05:23:17.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.271+0000 7f61a4332640 1 --2- 172.21.15.23:0/1071792980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f619c102170 0x7f619c108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.271+0000 7f61a4332640 1 --2- 172.21.15.23:0/1071792980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f619c1017b0 0x7f619c101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.271+0000 7f61a4332640 1 --2- 172.21.15.23:0/1071792980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f619c1005c0 0x7f619c1009e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.271+0000 7f61a4332640 1 -- 172.21.15.23:0/1071792980 >> 172.21.15.23:0/1071792980 conn(0x7f619c0fbb30 msgr2=0x7f619c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:17.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.271+0000 7f61a4332640 1 -- 172.21.15.23:0/1071792980 shutdown_connections 2024-03-28T05:23:17.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.271+0000 7f61a4332640 1 -- 172.21.15.23:0/1071792980 wait complete. 2024-03-28T05:23:17.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.272+0000 7f61a4332640 1 Processor -- start 2024-03-28T05:23:17.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.272+0000 7f61a4332640 1 -- start start 2024-03-28T05:23:17.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.272+0000 7f61a4332640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f619c1005c0 0x7f619c199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:17.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.272+0000 7f61a4332640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f619c1017b0 0x7f619c19a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:17.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.272+0000 7f61a4332640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f619c102170 0x7f619c1a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:17.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.272+0000 7f61a4332640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c10c290 con 0x7f619c1005c0 2024-03-28T05:23:17.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.272+0000 7f61a4332640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c10c110 con 0x7f619c1017b0 2024-03-28T05:23:17.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.272+0000 7f61a4332640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c10c410 con 0x7f619c102170 2024-03-28T05:23:17.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.272+0000 7f61a20ce640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f619c1005c0 0x7f619c199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:17.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.272+0000 7f61a20ce640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f619c1005c0 0x7f619c199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43294/0 (socket says 172.21.15.23:43294) 2024-03-28T05:23:17.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.272+0000 7f61a20ce640 1 -- 172.21.15.23:0/908861441 learned_addr learned my addr 172.21.15.23:0/908861441 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:17.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.272+0000 7f61a28cf640 1 --2- 172.21.15.23:0/908861441 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f619c102170 0x7f619c1a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:17.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.272+0000 7f61a18cd640 1 --2- 172.21.15.23:0/908861441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f619c1017b0 0x7f619c19a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:17.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.272+0000 7f61a20ce640 1 -- 172.21.15.23:0/908861441 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f619c102170 msgr2=0x7f619c1a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:17.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.272+0000 7f61a20ce640 1 --2- 172.21.15.23:0/908861441 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f619c102170 0x7f619c1a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.272+0000 7f61a20ce640 1 -- 172.21.15.23:0/908861441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f619c1017b0 msgr2=0x7f619c19a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:17.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.272+0000 7f61a20ce640 1 --2- 172.21.15.23:0/908861441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f619c1017b0 0x7f619c19a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.272+0000 7f61a20ce640 1 -- 172.21.15.23:0/908861441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619c1a1850 con 0x7f619c1005c0 2024-03-28T05:23:17.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.272+0000 7f61a20ce640 1 --2- 172.21.15.23:0/908861441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f619c1005c0 0x7f619c199c40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6194000c00 tx=0x7f619402c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:17.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.273+0000 7f619b7fe640 1 -- 172.21.15.23:0/908861441 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194043020 con 0x7f619c1005c0 2024-03-28T05:23:17.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.273+0000 7f619b7fe640 1 -- 172.21.15.23:0/908861441 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61940054e0 con 0x7f619c1005c0 2024-03-28T05:23:17.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.273+0000 7f61a4332640 1 -- 172.21.15.23:0/908861441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f619c1a1ae0 con 0x7f619c1005c0 2024-03-28T05:23:17.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.273+0000 7f619b7fe640 1 -- 172.21.15.23:0/908861441 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194003ce0 con 0x7f619c1005c0 2024-03-28T05:23:17.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.273+0000 7f61a4332640 1 -- 172.21.15.23:0/908861441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f619c1a2000 con 0x7f619c1005c0 2024-03-28T05:23:17.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.274+0000 7f61a4332640 1 -- 172.21.15.23:0/908861441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f619c1a21d0 con 0x7f619c1005c0 2024-03-28T05:23:17.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.275+0000 7f619b7fe640 1 -- 172.21.15.23:0/908861441 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6194046020 con 0x7f619c1005c0 2024-03-28T05:23:17.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.276+0000 7f619b7fe640 1 --2- 172.21.15.23:0/908861441 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f61840a2800 0x7f61840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:17.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.276+0000 7f61a18cd640 1 --2- 172.21.15.23:0/908861441 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f61840a2800 0x7f61840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:17.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.276+0000 7f619b7fe640 1 -- 172.21.15.23:0/908861441 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f61940f13e0 con 0x7f619c1005c0 2024-03-28T05:23:17.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.277+0000 7f61a18cd640 1 --2- 172.21.15.23:0/908861441 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f61840a2800 0x7f61840a4cc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6190009b60 tx=0x7f6190005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:17.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.280+0000 7f619b7fe640 1 -- 172.21.15.23:0/908861441 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f619404d020 con 0x7f619c1005c0 2024-03-28T05:23:17.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.466+0000 7f61a4332640 1 -- 172.21.15.23:0/908861441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f619c1a24c0 con 0x7f619c1005c0 2024-03-28T05:23:17.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.468+0000 7f619b7fe640 1 -- 172.21.15.23:0/908861441 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24695 (secure 0 0 0) 0x7f61940be0d0 con 0x7f619c1005c0 2024-03-28T05:23:17.468 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:23:17.468 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":93,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","created":"2024-03-28T05:13:13.453653+0000","modified":"2024-03-28T05:23:10.695543+0000","last_up_change":"2024-03-28T05:20:20.221527+0000","last_in_change":"2024-03-28T05:20:00.365478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T05:16:32.986974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T05:21:06.103894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T05:21:06.827190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T05:21:10.507418+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"18150942-59a4-4c3b-83d4-c1fc3ccf1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6803","nonce":1219956881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6805","nonce":1219956881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6809","nonce":1219956881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6807","nonce":1219956881}]},"public_addr":"172.21.15.23:6803/1219956881","cluster_addr":"172.21.15.23:6805/1219956881","heartbeat_back_addr":"172.21.15.23:6809/1219956881","heartbeat_front_addr":"172.21.15.23:6807/1219956881","state":["exists","up"]},{"osd":1,"uuid":"ba6d1a95-5135-4670-95f9-7878c0e7bd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6801","nonce":555491234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6803","nonce":555491234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6807","nonce":555491234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6805","nonce":555491234}]},"public_addr":"172.21.15.98:6801/555491234","cluster_addr":"172.21.15.98:6803/555491234","heartbeat_back_addr":"172.21.15.98:6807/555491234","heartbeat_front_addr":"172.21.15.98:6805/555491234","state":["exists","up"]},{"osd":2,"uuid":"873bdd60-2a47-403b-b1f4-ee9226a95067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6801","nonce":1535856914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6803","nonce":1535856914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6807","nonce":1535856914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6805","nonce":1535856914}]},"public_addr":"172.21.15.113:6801/1535856914","cluster_addr":"172.21.15.113:6803/1535856914","heartbeat_back_addr":"172.21.15.113:6807/1535856914","heartbeat_front_addr":"172.21.15.113:6805/1535856914","state":["exists","up"]},{"osd":3,"uuid":"83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6811","nonce":2949415020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6813","nonce":2949415020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6817","nonce":2949415020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6815","nonce":2949415020}]},"public_addr":"172.21.15.23:6811/2949415020","cluster_addr":"172.21.15.23:6813/2949415020","heartbeat_back_addr":"172.21.15.23:6817/2949415020","heartbeat_front_addr":"172.21.15.23:6815/2949415020","state":["exists","up"]},{"osd":4,"uuid":"76a90bac-b221-40a3-8238-40e8049cec36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6809","nonce":2615902819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6811","nonce":2615902819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6815","nonce":2615902819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6813","nonce":2615902819}]},"public_addr":"172.21.15.98:6809/2615902819","cluster_addr":"172.21.15.98:6811/2615902819","heartbeat_back_addr":"172.21.15.98:6815/2615902819","heartbeat_front_addr":"172.21.15.98:6813/2615902819","state":["exists","up"]},{"osd":5,"uuid":"b53fdb55-a8df-40a8-bd1b-8bcef634e26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6809","nonce":625271528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6811","nonce":625271528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6815","nonce":625271528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6813","nonce":625271528}]},"public_addr":"172.21.15.113:6809/625271528","cluster_addr":"172.21.15.113:6811/625271528","heartbeat_back_addr":"172.21.15.113:6815/625271528","heartbeat_front_addr":"172.21.15.113:6813/625271528","state":["exists","up"]},{"osd":6,"uuid":"f00283d2-8d67-4e97-a575-9a9d3b55d5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6819","nonce":4188464176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6821","nonce":4188464176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6825","nonce":4188464176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6823","nonce":4188464176}]},"public_addr":"172.21.15.23:6819/4188464176","cluster_addr":"172.21.15.23:6821/4188464176","heartbeat_back_addr":"172.21.15.23:6825/4188464176","heartbeat_front_addr":"172.21.15.23:6823/4188464176","state":["exists","up"]},{"osd":7,"uuid":"edc008a7-b07b-40a8-88b7-1614ba38a775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6817","nonce":1985546028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6819","nonce":1985546028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6823","nonce":1985546028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6821","nonce":1985546028}]},"public_addr":"172.21.15.98:6817/1985546028","cluster_addr":"172.21.15.98:6819/1985546028","heartbeat_back_addr":"172.21.15.98:6823/1985546028","heartbeat_front_addr":"172.21.15.98:6821/1985546028","state":["exists","up"]},{"osd":8,"uuid":"56e7b2d1-81f0-47b2-8638-f29832bed8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6817","nonce":3534876081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6819","nonce":3534876081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6823","nonce":3534876081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6821","nonce":3534876081}]},"public_addr":"172.21.15.113:6817/3534876081","cluster_addr":"172.21.15.113:6819/3534876081","heartbeat_back_addr":"172.21.15.113:6823/3534876081","heartbeat_front_addr":"172.21.15.113:6821/3534876081","state":["exists","up"]},{"osd":9,"uuid":"a2e90caa-6e9c-42bf-9f18-81f7f47880cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6827","nonce":672264207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6829","nonce":672264207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6833","nonce":672264207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6831","nonce":672264207}]},"public_addr":"172.21.15.23:6827/672264207","cluster_addr":"172.21.15.23:6829/672264207","heartbeat_back_addr":"172.21.15.23:6833/672264207","heartbeat_front_addr":"172.21.15.23:6831/672264207","state":["exists","up"]},{"osd":10,"uuid":"0eda917f-020c-437f-95ae-b26203f6c68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6825","nonce":2699920595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6827","nonce":2699920595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6831","nonce":2699920595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6829","nonce":2699920595}]},"public_addr":"172.21.15.98:6825/2699920595","cluster_addr":"172.21.15.98:6827/2699920595","heartbeat_back_addr":"172.21.15.98:6831/2699920595","heartbeat_front_addr":"172.21.15.98:6829/2699920595","state":["exists","up"]},{"osd":11,"uuid":"aae78501-78a8-469a-be20-3006f5ec5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6825","nonce":2113644020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6827","nonce":2113644020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6831","nonce":2113644020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6829","nonce":2113644020}]},"public_addr":"172.21.15.113:6825/2113644020","cluster_addr":"172.21.15.113:6827/2113644020","heartbeat_back_addr":"172.21.15.113:6831/2113644020","heartbeat_front_addr":"172.21.15.113:6829/2113644020","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:15:43.527475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:07.006538+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:30.594176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:55.810999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:20.595811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:45.474138+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:10.500409+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:35.787638+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:01.001188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:26.761115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:52.952752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:20:18.559832+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":7}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:6800/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1876451381":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2253559862":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2653562762":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/3337463830":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6800/1735221490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/2784771635":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2389902490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6801/1735221490":"2024-03-29T05:13:42.579607+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T05:23:17.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.470+0000 7f61a4332640 1 -- 172.21.15.23:0/908861441 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f61840a2800 msgr2=0x7f61840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:17.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.470+0000 7f61a4332640 1 --2- 172.21.15.23:0/908861441 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f61840a2800 0x7f61840a4cc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6190009b60 tx=0x7f6190005f90 comp rx=0 tx=0).stop 2024-03-28T05:23:17.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.470+0000 7f61a4332640 1 -- 172.21.15.23:0/908861441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f619c1005c0 msgr2=0x7f619c199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:17.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.470+0000 7f61a4332640 1 --2- 172.21.15.23:0/908861441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f619c1005c0 0x7f619c199c40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6194000c00 tx=0x7f619402c6c0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.470+0000 7f61a4332640 1 -- 172.21.15.23:0/908861441 shutdown_connections 2024-03-28T05:23:17.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.470+0000 7f61a4332640 1 --2- 172.21.15.23:0/908861441 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f61840a2800 0x7f61840a4cc0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.470+0000 7f61a4332640 1 --2- 172.21.15.23:0/908861441 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f619c102170 0x7f619c1a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.470+0000 7f61a4332640 1 --2- 172.21.15.23:0/908861441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f619c1017b0 0x7f619c19a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.470+0000 7f61a4332640 1 --2- 172.21.15.23:0/908861441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f619c1005c0 0x7f619c199c40 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.470+0000 7f61a4332640 1 -- 172.21.15.23:0/908861441 >> 172.21.15.23:0/908861441 conn(0x7f619c0fbb30 msgr2=0x7f619c0fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:17.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.470+0000 7f61a4332640 1 -- 172.21.15.23:0/908861441 shutdown_connections 2024-03-28T05:23:17.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.470+0000 7f61a4332640 1 -- 172.21.15.23:0/908861441 wait complete. 2024-03-28T05:23:17.481 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T05:16:32.986974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T05:21:06.103894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T05:21:06.827190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T05:21:10.507418+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T05:23:17.481 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T05:23:17.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.588+0000 7f3fa0ae8640 1 -- 172.21.15.23:0/1348069921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f9c1011e0 msgr2=0x7f3f9c108a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:17.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.588+0000 7f3fa0ae8640 1 --2- 172.21.15.23:0/1348069921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f9c1011e0 0x7f3f9c108a80 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3f94009ca0 tx=0x7f3f9402c0f0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.588+0000 7f3fa0ae8640 1 -- 172.21.15.23:0/1348069921 shutdown_connections 2024-03-28T05:23:17.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.588+0000 7f3fa0ae8640 1 --2- 172.21.15.23:0/1348069921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f9c1011e0 0x7f3f9c108a80 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.588+0000 7f3fa0ae8640 1 --2- 172.21.15.23:0/1348069921 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f9c100820 0x7f3f9c100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.588+0000 7f3fa0ae8640 1 --2- 172.21.15.23:0/1348069921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f9c101bd0 0x7f3f9c101ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.588+0000 7f3fa0ae8640 1 -- 172.21.15.23:0/1348069921 >> 172.21.15.23:0/1348069921 conn(0x7f3f9c0fb290 msgr2=0x7f3f9c0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:17.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.588+0000 7f3fa0ae8640 1 -- 172.21.15.23:0/1348069921 shutdown_connections 2024-03-28T05:23:17.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.588+0000 7f3fa0ae8640 1 -- 172.21.15.23:0/1348069921 wait complete. 2024-03-28T05:23:17.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.589+0000 7f3fa0ae8640 1 Processor -- start 2024-03-28T05:23:17.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.589+0000 7f3fa0ae8640 1 -- start start 2024-03-28T05:23:17.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.589+0000 7f3fa0ae8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f9c100820 0x7f3f9c196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:17.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.589+0000 7f3fa0ae8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f9c1011e0 0x7f3f9c196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:17.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.589+0000 7f3fa0ae8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f9c101bd0 0x7f3f9c19de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:17.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.589+0000 7f3fa0ae8640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c10c240 con 0x7f3f9c1011e0 2024-03-28T05:23:17.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.589+0000 7f3fa0ae8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c10c0c0 con 0x7f3f9c101bd0 2024-03-28T05:23:17.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.589+0000 7f3fa0ae8640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c10c3c0 con 0x7f3f9c100820 2024-03-28T05:23:17.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.589+0000 7f3f99d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f9c1011e0 0x7f3f9c196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:17.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.589+0000 7f3f99d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f9c1011e0 0x7f3f9c196da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43314/0 (socket says 172.21.15.23:43314) 2024-03-28T05:23:17.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.589+0000 7f3f99d9b640 1 -- 172.21.15.23:0/3146076560 learned_addr learned my addr 172.21.15.23:0/3146076560 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:17.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.589+0000 7f3f9a59c640 1 --2- 172.21.15.23:0/3146076560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f9c100820 0x7f3f9c196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:17.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.589+0000 7f3f9ad9d640 1 --2- 172.21.15.23:0/3146076560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f9c101bd0 0x7f3f9c19de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:17.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.589+0000 7f3f99d9b640 1 -- 172.21.15.23:0/3146076560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f9c100820 msgr2=0x7f3f9c196860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:17.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.589+0000 7f3f99d9b640 1 --2- 172.21.15.23:0/3146076560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f9c100820 0x7f3f9c196860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.589+0000 7f3f99d9b640 1 -- 172.21.15.23:0/3146076560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f9c101bd0 msgr2=0x7f3f9c19de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:17.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.589+0000 7f3f99d9b640 1 --2- 172.21.15.23:0/3146076560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f9c101bd0 0x7f3f9c19de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.589+0000 7f3f99d9b640 1 -- 172.21.15.23:0/3146076560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9c19e480 con 0x7f3f9c1011e0 2024-03-28T05:23:17.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.589+0000 7f3f9a59c640 1 --2- 172.21.15.23:0/3146076560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f9c100820 0x7f3f9c196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:23:17.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.589+0000 7f3f9ad9d640 1 --2- 172.21.15.23:0/3146076560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f9c101bd0 0x7f3f9c19de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:23:17.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.589+0000 7f3f99d9b640 1 --2- 172.21.15.23:0/3146076560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f9c1011e0 0x7f3f9c196da0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3f9000e9c0 tx=0x7f3f9000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:17.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.590+0000 7f3f8b7fe640 1 -- 172.21.15.23:0/3146076560 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9000cd80 con 0x7f3f9c1011e0 2024-03-28T05:23:17.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.590+0000 7f3f8b7fe640 1 -- 172.21.15.23:0/3146076560 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f900035f0 con 0x7f3f9c1011e0 2024-03-28T05:23:17.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.590+0000 7f3fa0ae8640 1 -- 172.21.15.23:0/3146076560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f9c19e770 con 0x7f3f9c1011e0 2024-03-28T05:23:17.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.590+0000 7f3f8b7fe640 1 -- 172.21.15.23:0/3146076560 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f90010690 con 0x7f3f9c1011e0 2024-03-28T05:23:17.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.590+0000 7f3fa0ae8640 1 -- 172.21.15.23:0/3146076560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f9c19ecc0 con 0x7f3f9c1011e0 2024-03-28T05:23:17.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.591+0000 7f3fa0ae8640 1 -- 172.21.15.23:0/3146076560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f9c107900 con 0x7f3f9c1011e0 2024-03-28T05:23:17.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.592+0000 7f3f8b7fe640 1 -- 172.21.15.23:0/3146076560 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3f900037a0 con 0x7f3f9c1011e0 2024-03-28T05:23:17.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.592+0000 7f3f8b7fe640 1 --2- 172.21.15.23:0/3146076560 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3f840a2800 0x7f3f840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:17.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.593+0000 7f3f9a59c640 1 --2- 172.21.15.23:0/3146076560 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3f840a2800 0x7f3f840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:17.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.593+0000 7f3f8b7fe640 1 -- 172.21.15.23:0/3146076560 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f3f900cf380 con 0x7f3f9c1011e0 2024-03-28T05:23:17.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.593+0000 7f3f9a59c640 1 --2- 172.21.15.23:0/3146076560 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3f840a2800 0x7f3f840a4cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c069a50 tx=0x7f3f8c00a560 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:17.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.596+0000 7f3f8b7fe640 1 -- 172.21.15.23:0/3146076560 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3f9009c1a0 con 0x7f3f9c1011e0 2024-03-28T05:23:17.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.778+0000 7f3fa0ae8640 1 -- 172.21.15.23:0/3146076560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3f9c02d320 con 0x7f3f9c1011e0 2024-03-28T05:23:17.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.778+0000 7f3f8b7fe640 1 -- 172.21.15.23:0/3146076560 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3f9009baa0 con 0x7f3f9c1011e0 2024-03-28T05:23:17.778 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-28T05:23:17.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.780+0000 7f3fa0ae8640 1 -- 172.21.15.23:0/3146076560 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3f840a2800 msgr2=0x7f3f840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:17.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.780+0000 7f3fa0ae8640 1 --2- 172.21.15.23:0/3146076560 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3f840a2800 0x7f3f840a4cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c069a50 tx=0x7f3f8c00a560 comp rx=0 tx=0).stop 2024-03-28T05:23:17.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.780+0000 7f3fa0ae8640 1 -- 172.21.15.23:0/3146076560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f9c1011e0 msgr2=0x7f3f9c196da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:17.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.780+0000 7f3fa0ae8640 1 --2- 172.21.15.23:0/3146076560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f9c1011e0 0x7f3f9c196da0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3f9000e9c0 tx=0x7f3f9000ee80 comp rx=0 tx=0).stop 2024-03-28T05:23:17.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.780+0000 7f3fa0ae8640 1 -- 172.21.15.23:0/3146076560 shutdown_connections 2024-03-28T05:23:17.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.780+0000 7f3fa0ae8640 1 --2- 172.21.15.23:0/3146076560 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f3f840a2800 0x7f3f840a4cc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.780+0000 7f3fa0ae8640 1 --2- 172.21.15.23:0/3146076560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f9c101bd0 0x7f3f9c19de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.780+0000 7f3fa0ae8640 1 --2- 172.21.15.23:0/3146076560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f9c1011e0 0x7f3f9c196da0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.780+0000 7f3fa0ae8640 1 --2- 172.21.15.23:0/3146076560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3f9c100820 0x7f3f9c196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.780+0000 7f3fa0ae8640 1 -- 172.21.15.23:0/3146076560 >> 172.21.15.23:0/3146076560 conn(0x7f3f9c0fb290 msgr2=0x7f3f9c104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:17.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.781+0000 7f3fa0ae8640 1 -- 172.21.15.23:0/3146076560 shutdown_connections 2024-03-28T05:23:17.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.781+0000 7f3fa0ae8640 1 -- 172.21.15.23:0/3146076560 wait complete. 2024-03-28T05:23:17.791 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T05:23:17.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.898+0000 7fc6b5d53640 1 -- 172.21.15.23:0/1232817054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6b0104a70 msgr2=0x7fc6b0106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:17.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.898+0000 7fc6b5d53640 1 --2- 172.21.15.23:0/1232817054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6b0104a70 0x7fc6b0106e60 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0009900 tx=0x7fc6a002bc30 comp rx=0 tx=0).stop 2024-03-28T05:23:17.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.898+0000 7fc6b5d53640 1 -- 172.21.15.23:0/1232817054 shutdown_connections 2024-03-28T05:23:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.898+0000 7fc6b5d53640 1 --2- 172.21.15.23:0/1232817054 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6b01073a0 0x7fc6b01097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.898+0000 7fc6b5d53640 1 --2- 172.21.15.23:0/1232817054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6b0104a70 0x7fc6b0106e60 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.898+0000 7fc6b5d53640 1 --2- 172.21.15.23:0/1232817054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6b0102140 0x7fc6b0104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.898+0000 7fc6b5d53640 1 -- 172.21.15.23:0/1232817054 >> 172.21.15.23:0/1232817054 conn(0x7fc6b00fbb50 msgr2=0x7fc6b00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.898+0000 7fc6b5d53640 1 -- 172.21.15.23:0/1232817054 shutdown_connections 2024-03-28T05:23:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.898+0000 7fc6b5d53640 1 -- 172.21.15.23:0/1232817054 wait complete. 2024-03-28T05:23:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.898+0000 7fc6b5d53640 1 Processor -- start 2024-03-28T05:23:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6b5d53640 1 -- start start 2024-03-28T05:23:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6b5d53640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6b0102140 0x7fc6b0197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6b5d53640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6b0104a70 0x7fc6b0197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6b5d53640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6b01073a0 0x7fc6b019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6b5d53640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b010d390 con 0x7fc6b0104a70 2024-03-28T05:23:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6b5d53640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b010d210 con 0x7fc6b0102140 2024-03-28T05:23:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6b5d53640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b010d510 con 0x7fc6b01073a0 2024-03-28T05:23:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6aeffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6b0104a70 0x7fc6b0197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:17.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6aeffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6b0104a70 0x7fc6b0197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43334/0 (socket says 172.21.15.23:43334) 2024-03-28T05:23:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6aeffd640 1 -- 172.21.15.23:0/3512097243 learned_addr learned my addr 172.21.15.23:0/3512097243 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6af7fe640 1 --2- 172.21.15.23:0/3512097243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6b0102140 0x7fc6b0197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6affff640 1 --2- 172.21.15.23:0/3512097243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6b01073a0 0x7fc6b019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6aeffd640 1 -- 172.21.15.23:0/3512097243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6b01073a0 msgr2=0x7fc6b019efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6aeffd640 1 --2- 172.21.15.23:0/3512097243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6b01073a0 0x7fc6b019efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6aeffd640 1 -- 172.21.15.23:0/3512097243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6b0102140 msgr2=0x7fc6b0197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6aeffd640 1 --2- 172.21.15.23:0/3512097243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6b0102140 0x7fc6b0197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6aeffd640 1 -- 172.21.15.23:0/3512097243 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b019f620 con 0x7fc6b0104a70 2024-03-28T05:23:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6af7fe640 1 --2- 172.21.15.23:0/3512097243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6b0102140 0x7fc6b0197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:23:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6aeffd640 1 --2- 172.21.15.23:0/3512097243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6b0104a70 0x7fc6b0197f40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0000c00 tx=0x7fc6a002c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6acff9640 1 -- 172.21.15.23:0/3512097243 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0043020 con 0x7fc6b0104a70 2024-03-28T05:23:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6b5d53640 1 -- 172.21.15.23:0/3512097243 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b019f8b0 con 0x7fc6b0104a70 2024-03-28T05:23:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6acff9640 1 -- 172.21.15.23:0/3512097243 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6a00054e0 con 0x7fc6b0104a70 2024-03-28T05:23:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6acff9640 1 -- 172.21.15.23:0/3512097243 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0003ce0 con 0x7fc6b0104a70 2024-03-28T05:23:17.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.899+0000 7fc6b5d53640 1 -- 172.21.15.23:0/3512097243 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b019fe20 con 0x7fc6b0104a70 2024-03-28T05:23:17.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.900+0000 7fc6b5d53640 1 -- 172.21.15.23:0/3512097243 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6b00fd640 con 0x7fc6b0104a70 2024-03-28T05:23:17.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.902+0000 7fc6acff9640 1 -- 172.21.15.23:0/3512097243 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc6a0046020 con 0x7fc6b0104a70 2024-03-28T05:23:17.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.902+0000 7fc6acff9640 1 --2- 172.21.15.23:0/3512097243 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc6980a2800 0x7fc6980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:17.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.902+0000 7fc6af7fe640 1 --2- 172.21.15.23:0/3512097243 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc6980a2800 0x7fc6980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:17.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.902+0000 7fc6acff9640 1 -- 172.21.15.23:0/3512097243 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fc6a00f1380 con 0x7fc6b0104a70 2024-03-28T05:23:17.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.903+0000 7fc6af7fe640 1 --2- 172.21.15.23:0/3512097243 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc6980a2800 0x7fc6980a4cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc6a40058c0 tx=0x7fc6a4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:17.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:17.906+0000 7fc6acff9640 1 -- 172.21.15.23:0/3512097243 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc6a004d020 con 0x7fc6b0104a70 2024-03-28T05:23:18.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.087+0000 7fc6b5d53640 1 -- 172.21.15.23:0/3512097243 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc6b002d230 con 0x7fc6b0104a70 2024-03-28T05:23:18.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.087+0000 7fc6acff9640 1 -- 172.21.15.23:0/3512097243 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc6a00be070 con 0x7fc6b0104a70 2024-03-28T05:23:18.087 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T05:23:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.089+0000 7fc6b5d53640 1 -- 172.21.15.23:0/3512097243 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc6980a2800 msgr2=0x7fc6980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.089+0000 7fc6b5d53640 1 --2- 172.21.15.23:0/3512097243 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc6980a2800 0x7fc6980a4cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc6a40058c0 tx=0x7fc6a4005850 comp rx=0 tx=0).stop 2024-03-28T05:23:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.089+0000 7fc6b5d53640 1 -- 172.21.15.23:0/3512097243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6b0104a70 msgr2=0x7fc6b0197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.089+0000 7fc6b5d53640 1 --2- 172.21.15.23:0/3512097243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6b0104a70 0x7fc6b0197f40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0000c00 tx=0x7fc6a002c6c0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.089+0000 7fc6b5d53640 1 -- 172.21.15.23:0/3512097243 shutdown_connections 2024-03-28T05:23:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.089+0000 7fc6b5d53640 1 --2- 172.21.15.23:0/3512097243 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fc6980a2800 0x7fc6980a4cc0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.089+0000 7fc6b5d53640 1 --2- 172.21.15.23:0/3512097243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6b01073a0 0x7fc6b019efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.089+0000 7fc6b5d53640 1 --2- 172.21.15.23:0/3512097243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6b0104a70 0x7fc6b0197f40 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.089+0000 7fc6b5d53640 1 --2- 172.21.15.23:0/3512097243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6b0102140 0x7fc6b0197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.089+0000 7fc6b5d53640 1 -- 172.21.15.23:0/3512097243 >> 172.21.15.23:0/3512097243 conn(0x7fc6b00fbb50 msgr2=0x7fc6b0102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.089+0000 7fc6b5d53640 1 -- 172.21.15.23:0/3512097243 shutdown_connections 2024-03-28T05:23:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.089+0000 7fc6b5d53640 1 -- 172.21.15.23:0/3512097243 wait complete. 2024-03-28T05:23:18.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:17 smithi023 ceph-mon[31954]: pgmap v357: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 72 B/s, 0 objects/s recovering 2024-03-28T05:23:18.098 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:17 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/908861441' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:23:18.099 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:17 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3146076560' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:23:18.099 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T05:23:18.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:17 smithi098 ceph-mon[36087]: pgmap v357: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 72 B/s, 0 objects/s recovering 2024-03-28T05:23:18.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:17 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/908861441' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:23:18.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:17 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3146076560' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:23:18.143 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:17 smithi113 ceph-mon[37164]: pgmap v357: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 72 B/s, 0 objects/s recovering 2024-03-28T05:23:18.143 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:17 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/908861441' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:23:18.143 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:17 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3146076560' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:23:18.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.206+0000 7f5e4f31b640 1 -- 172.21.15.23:0/181030969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e48104a70 msgr2=0x7f5e48106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:18.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.206+0000 7f5e4f31b640 1 --2- 172.21.15.23:0/181030969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e48104a70 0x7f5e48106e60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5e38009900 tx=0x7f5e3802bc30 comp rx=0 tx=0).stop 2024-03-28T05:23:18.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.206+0000 7f5e4f31b640 1 -- 172.21.15.23:0/181030969 shutdown_connections 2024-03-28T05:23:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.206+0000 7f5e4f31b640 1 --2- 172.21.15.23:0/181030969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e481073a0 0x7f5e481097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.206+0000 7f5e4f31b640 1 --2- 172.21.15.23:0/181030969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e48104a70 0x7f5e48106e60 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.206+0000 7f5e4f31b640 1 --2- 172.21.15.23:0/181030969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e48102140 0x7f5e48104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.206+0000 7f5e4f31b640 1 -- 172.21.15.23:0/181030969 >> 172.21.15.23:0/181030969 conn(0x7f5e480fbb50 msgr2=0x7f5e480fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.206+0000 7f5e4f31b640 1 -- 172.21.15.23:0/181030969 shutdown_connections 2024-03-28T05:23:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.206+0000 7f5e4f31b640 1 -- 172.21.15.23:0/181030969 wait complete. 2024-03-28T05:23:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.207+0000 7f5e4f31b640 1 Processor -- start 2024-03-28T05:23:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.207+0000 7f5e4f31b640 1 -- start start 2024-03-28T05:23:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.207+0000 7f5e4f31b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e48102140 0x7f5e48197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.207+0000 7f5e4f31b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e48104a70 0x7f5e48197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.207+0000 7f5e4f31b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e481073a0 0x7f5e4819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.207+0000 7f5e4f31b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4810d330 con 0x7f5e481073a0 2024-03-28T05:23:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.207+0000 7f5e4f31b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4810d1b0 con 0x7f5e48102140 2024-03-28T05:23:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.207+0000 7f5e4f31b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4810d4b0 con 0x7f5e48104a70 2024-03-28T05:23:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.207+0000 7f5e4d8b8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e481073a0 0x7f5e4819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.207+0000 7f5e4d8b8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e481073a0 0x7f5e4819efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43346/0 (socket says 172.21.15.23:43346) 2024-03-28T05:23:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.207+0000 7f5e4d8b8640 1 -- 172.21.15.23:0/3103699955 learned_addr learned my addr 172.21.15.23:0/3103699955 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.207+0000 7f5e4c8b6640 1 --2- 172.21.15.23:0/3103699955 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e48104a70 0x7f5e48197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.207+0000 7f5e4d0b7640 1 --2- 172.21.15.23:0/3103699955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e48102140 0x7f5e48197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.207+0000 7f5e4d8b8640 1 -- 172.21.15.23:0/3103699955 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e48104a70 msgr2=0x7f5e48197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.207+0000 7f5e4d8b8640 1 --2- 172.21.15.23:0/3103699955 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e48104a70 0x7f5e48197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.207+0000 7f5e4d8b8640 1 -- 172.21.15.23:0/3103699955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e48102140 msgr2=0x7f5e48197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.207+0000 7f5e4d8b8640 1 --2- 172.21.15.23:0/3103699955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e48102140 0x7f5e48197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.207+0000 7f5e4d8b8640 1 -- 172.21.15.23:0/3103699955 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4819f650 con 0x7f5e481073a0 2024-03-28T05:23:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.207+0000 7f5e4d0b7640 1 --2- 172.21.15.23:0/3103699955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e48102140 0x7f5e48197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:23:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.207+0000 7f5e4d8b8640 1 --2- 172.21.15.23:0/3103699955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e481073a0 0x7f5e4819efe0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5e4000eaa0 tx=0x7f5e4000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.208+0000 7f5e3e7fc640 1 -- 172.21.15.23:0/3103699955 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4000cdb0 con 0x7f5e481073a0 2024-03-28T05:23:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.208+0000 7f5e3e7fc640 1 -- 172.21.15.23:0/3103699955 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e40005020 con 0x7f5e481073a0 2024-03-28T05:23:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.208+0000 7f5e4f31b640 1 -- 172.21.15.23:0/3103699955 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e4819f940 con 0x7f5e481073a0 2024-03-28T05:23:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.208+0000 7f5e3e7fc640 1 -- 172.21.15.23:0/3103699955 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e400116b0 con 0x7f5e481073a0 2024-03-28T05:23:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.208+0000 7f5e4f31b640 1 -- 172.21.15.23:0/3103699955 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e4819fe10 con 0x7f5e481073a0 2024-03-28T05:23:18.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.209+0000 7f5e4f31b640 1 -- 172.21.15.23:0/3103699955 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e480fd640 con 0x7f5e481073a0 2024-03-28T05:23:18.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.210+0000 7f5e3e7fc640 1 -- 172.21.15.23:0/3103699955 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5e40005560 con 0x7f5e481073a0 2024-03-28T05:23:18.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.210+0000 7f5e3e7fc640 1 --2- 172.21.15.23:0/3103699955 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5e340a2800 0x7f5e340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:18.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.210+0000 7f5e4d0b7640 1 --2- 172.21.15.23:0/3103699955 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5e340a2800 0x7f5e340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:18.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.210+0000 7f5e3e7fc640 1 -- 172.21.15.23:0/3103699955 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f5e400cf070 con 0x7f5e481073a0 2024-03-28T05:23:18.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.211+0000 7f5e4d0b7640 1 --2- 172.21.15.23:0/3103699955 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5e340a2800 0x7f5e340a4cc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5e44009c20 tx=0x7f5e44005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:18.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.215+0000 7f5e3e7fc640 1 -- 172.21.15.23:0/3103699955 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5e40016030 con 0x7f5e481073a0 2024-03-28T05:23:18.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.399+0000 7f5e4f31b640 1 -- 172.21.15.23:0/3103699955 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5e480622a0 con 0x7f5e481073a0 2024-03-28T05:23:18.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.399+0000 7f5e3e7fc640 1 -- 172.21.15.23:0/3103699955 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5e4009bd60 con 0x7f5e481073a0 2024-03-28T05:23:18.399 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-28T05:23:18.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.401+0000 7f5e4f31b640 1 -- 172.21.15.23:0/3103699955 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5e340a2800 msgr2=0x7f5e340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:18.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.401+0000 7f5e4f31b640 1 --2- 172.21.15.23:0/3103699955 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5e340a2800 0x7f5e340a4cc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5e44009c20 tx=0x7f5e44005f90 comp rx=0 tx=0).stop 2024-03-28T05:23:18.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.401+0000 7f5e4f31b640 1 -- 172.21.15.23:0/3103699955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e481073a0 msgr2=0x7f5e4819efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:18.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.401+0000 7f5e4f31b640 1 --2- 172.21.15.23:0/3103699955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e481073a0 0x7f5e4819efe0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5e4000eaa0 tx=0x7f5e4000ef60 comp rx=0 tx=0).stop 2024-03-28T05:23:18.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.401+0000 7f5e4f31b640 1 -- 172.21.15.23:0/3103699955 shutdown_connections 2024-03-28T05:23:18.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.401+0000 7f5e4f31b640 1 --2- 172.21.15.23:0/3103699955 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5e340a2800 0x7f5e340a4cc0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.401+0000 7f5e4f31b640 1 --2- 172.21.15.23:0/3103699955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e481073a0 0x7f5e4819efe0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.401+0000 7f5e4f31b640 1 --2- 172.21.15.23:0/3103699955 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e48104a70 0x7f5e48197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.401+0000 7f5e4f31b640 1 --2- 172.21.15.23:0/3103699955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e48102140 0x7f5e48197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.401+0000 7f5e4f31b640 1 -- 172.21.15.23:0/3103699955 >> 172.21.15.23:0/3103699955 conn(0x7f5e480fbb50 msgr2=0x7f5e48102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:18.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.401+0000 7f5e4f31b640 1 -- 172.21.15.23:0/3103699955 shutdown_connections 2024-03-28T05:23:18.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.401+0000 7f5e4f31b640 1 -- 172.21.15.23:0/3103699955 wait complete. 2024-03-28T05:23:18.411 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T05:23:18.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.518+0000 7f93dc3b9640 1 -- 172.21.15.23:0/2850650565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93d41005a0 msgr2=0x7f93d41009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:18.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.518+0000 7f93dc3b9640 1 --2- 172.21.15.23:0/2850650565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93d41005a0 0x7f93d41009c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f93c8009900 tx=0x7f93c802bc30 comp rx=0 tx=0).stop 2024-03-28T05:23:18.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.518+0000 7f93dc3b9640 1 -- 172.21.15.23:0/2850650565 shutdown_connections 2024-03-28T05:23:18.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.518+0000 7f93dc3b9640 1 --2- 172.21.15.23:0/2850650565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93d4102150 0x7f93d41089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.518+0000 7f93dc3b9640 1 --2- 172.21.15.23:0/2850650565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93d4101790 0x7f93d4101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.518+0000 7f93dc3b9640 1 --2- 172.21.15.23:0/2850650565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93d41005a0 0x7f93d41009c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.518+0000 7f93dc3b9640 1 -- 172.21.15.23:0/2850650565 >> 172.21.15.23:0/2850650565 conn(0x7f93d40fbb30 msgr2=0x7f93d40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:18.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.518+0000 7f93dc3b9640 1 -- 172.21.15.23:0/2850650565 shutdown_connections 2024-03-28T05:23:18.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.518+0000 7f93dc3b9640 1 -- 172.21.15.23:0/2850650565 wait complete. 2024-03-28T05:23:18.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.518+0000 7f93dc3b9640 1 Processor -- start 2024-03-28T05:23:18.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.518+0000 7f93dc3b9640 1 -- start start 2024-03-28T05:23:18.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.518+0000 7f93dc3b9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93d41005a0 0x7f93d4069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:18.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.518+0000 7f93dc3b9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93d4101790 0x7f93d419c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:18.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.518+0000 7f93dc3b9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93d4102150 0x7f93d419ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:18.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.518+0000 7f93dc3b9640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d410c230 con 0x7f93d4101790 2024-03-28T05:23:18.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.518+0000 7f93dc3b9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d410c0b0 con 0x7f93d41005a0 2024-03-28T05:23:18.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.518+0000 7f93dc3b9640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d410c3b0 con 0x7f93d4102150 2024-03-28T05:23:18.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.519+0000 7f93d9954640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93d4101790 0x7f93d419c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:18.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.519+0000 7f93d9954640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93d4101790 0x7f93d419c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43376/0 (socket says 172.21.15.23:43376) 2024-03-28T05:23:18.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.519+0000 7f93d9954640 1 -- 172.21.15.23:0/115543208 learned_addr learned my addr 172.21.15.23:0/115543208 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:18.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.519+0000 7f93da956640 1 --2- 172.21.15.23:0/115543208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93d4102150 0x7f93d419ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:18.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.519+0000 7f93da155640 1 --2- 172.21.15.23:0/115543208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93d41005a0 0x7f93d4069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:18.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.519+0000 7f93d9954640 1 -- 172.21.15.23:0/115543208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93d4102150 msgr2=0x7f93d419ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:18.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.519+0000 7f93d9954640 1 --2- 172.21.15.23:0/115543208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93d4102150 0x7f93d419ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.519+0000 7f93d9954640 1 -- 172.21.15.23:0/115543208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93d41005a0 msgr2=0x7f93d4069390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:18.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.519+0000 7f93d9954640 1 --2- 172.21.15.23:0/115543208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93d41005a0 0x7f93d4069390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.519+0000 7f93d9954640 1 -- 172.21.15.23:0/115543208 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d419f620 con 0x7f93d4101790 2024-03-28T05:23:18.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.519+0000 7f93d9954640 1 --2- 172.21.15.23:0/115543208 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93d4101790 0x7f93d419c720 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f93d000ea60 tx=0x7f93d000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:18.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.519+0000 7f93c77fe640 1 -- 172.21.15.23:0/115543208 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d000cde0 con 0x7f93d4101790 2024-03-28T05:23:18.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.519+0000 7f93dc3b9640 1 -- 172.21.15.23:0/115543208 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d419f910 con 0x7f93d4101790 2024-03-28T05:23:18.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.519+0000 7f93c77fe640 1 -- 172.21.15.23:0/115543208 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93d00035f0 con 0x7f93d4101790 2024-03-28T05:23:18.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.519+0000 7f93c77fe640 1 -- 172.21.15.23:0/115543208 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d0010670 con 0x7f93d4101790 2024-03-28T05:23:18.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.519+0000 7f93dc3b9640 1 -- 172.21.15.23:0/115543208 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d419feb0 con 0x7f93d4101790 2024-03-28T05:23:18.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.520+0000 7f93dc3b9640 1 -- 172.21.15.23:0/115543208 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d4107960 con 0x7f93d4101790 2024-03-28T05:23:18.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.521+0000 7f93c77fe640 1 -- 172.21.15.23:0/115543208 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f93d00037a0 con 0x7f93d4101790 2024-03-28T05:23:18.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.522+0000 7f93c77fe640 1 --2- 172.21.15.23:0/115543208 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f93c00a2800 0x7f93c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:18.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.522+0000 7f93da155640 1 --2- 172.21.15.23:0/115543208 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f93c00a2800 0x7f93c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:18.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.522+0000 7f93c77fe640 1 -- 172.21.15.23:0/115543208 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f93d00cf0f0 con 0x7f93d4101790 2024-03-28T05:23:18.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.522+0000 7f93da155640 1 --2- 172.21.15.23:0/115543208 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f93c00a2800 0x7f93c00a4cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f93c8000c00 tx=0x7f93c8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:18.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.526+0000 7f93c77fe640 1 -- 172.21.15.23:0/115543208 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f93d009bde0 con 0x7f93d4101790 2024-03-28T05:23:18.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.707+0000 7f93dc3b9640 1 -- 172.21.15.23:0/115543208 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f93d40622a0 con 0x7f93d4101790 2024-03-28T05:23:18.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.708+0000 7f93c77fe640 1 -- 172.21.15.23:0/115543208 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f93d00860f0 con 0x7f93d4101790 2024-03-28T05:23:18.707 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T05:23:18.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.710+0000 7f93dc3b9640 1 -- 172.21.15.23:0/115543208 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f93c00a2800 msgr2=0x7f93c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.710+0000 7f93dc3b9640 1 --2- 172.21.15.23:0/115543208 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f93c00a2800 0x7f93c00a4cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f93c8000c00 tx=0x7f93c8005fd0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.710+0000 7f93dc3b9640 1 -- 172.21.15.23:0/115543208 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93d4101790 msgr2=0x7f93d419c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.710+0000 7f93dc3b9640 1 --2- 172.21.15.23:0/115543208 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93d4101790 0x7f93d419c720 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f93d000ea60 tx=0x7f93d000ef20 comp rx=0 tx=0).stop 2024-03-28T05:23:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.710+0000 7f93dc3b9640 1 -- 172.21.15.23:0/115543208 shutdown_connections 2024-03-28T05:23:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.710+0000 7f93dc3b9640 1 --2- 172.21.15.23:0/115543208 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f93c00a2800 0x7f93c00a4cc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.710+0000 7f93dc3b9640 1 --2- 172.21.15.23:0/115543208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f93d4102150 0x7f93d419ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.710+0000 7f93dc3b9640 1 --2- 172.21.15.23:0/115543208 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93d4101790 0x7f93d419c720 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.710+0000 7f93dc3b9640 1 --2- 172.21.15.23:0/115543208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93d41005a0 0x7f93d4069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.710+0000 7f93dc3b9640 1 -- 172.21.15.23:0/115543208 >> 172.21.15.23:0/115543208 conn(0x7f93d40fbb30 msgr2=0x7f93d4104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.710+0000 7f93dc3b9640 1 -- 172.21.15.23:0/115543208 shutdown_connections 2024-03-28T05:23:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.710+0000 7f93dc3b9640 1 -- 172.21.15.23:0/115543208 wait complete. 2024-03-28T05:23:18.721 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:23:18.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.827+0000 7f55830f8640 1 -- 172.21.15.23:0/2146406373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f557c1016f0 msgr2=0x7f557c101b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:18.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.827+0000 7f55830f8640 1 --2- 172.21.15.23:0/2146406373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f557c1016f0 0x7f557c101b10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f5570009900 tx=0x7f557002bc30 comp rx=0 tx=0).stop 2024-03-28T05:23:18.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.827+0000 7f55830f8640 1 -- 172.21.15.23:0/2146406373 shutdown_connections 2024-03-28T05:23:18.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.827+0000 7f55830f8640 1 --2- 172.21.15.23:0/2146406373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f557c1032a0 0x7f557c109b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.827+0000 7f55830f8640 1 --2- 172.21.15.23:0/2146406373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f557c1028e0 0x7f557c102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.827+0000 7f55830f8640 1 --2- 172.21.15.23:0/2146406373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f557c1016f0 0x7f557c101b10 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.827+0000 7f55830f8640 1 -- 172.21.15.23:0/2146406373 >> 172.21.15.23:0/2146406373 conn(0x7f557c0fcc40 msgr2=0x7f557c0ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:18.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.827+0000 7f55830f8640 1 -- 172.21.15.23:0/2146406373 shutdown_connections 2024-03-28T05:23:18.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.827+0000 7f55830f8640 1 -- 172.21.15.23:0/2146406373 wait complete. 2024-03-28T05:23:18.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.827+0000 7f55830f8640 1 Processor -- start 2024-03-28T05:23:18.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.827+0000 7f55830f8640 1 -- start start 2024-03-28T05:23:18.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.827+0000 7f55830f8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f557c1016f0 0x7f557c197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:18.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.827+0000 7f55830f8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f557c1028e0 0x7f557c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:18.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.827+0000 7f55830f8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f557c1032a0 0x7f557c19efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:18.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.827+0000 7f55830f8640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557c10d3b0 con 0x7f557c1028e0 2024-03-28T05:23:18.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.827+0000 7f55830f8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557c10d230 con 0x7f557c1032a0 2024-03-28T05:23:18.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.827+0000 7f55830f8640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557c10d530 con 0x7f557c1016f0 2024-03-28T05:23:18.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.828+0000 7f557bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f557c1028e0 0x7f557c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:18.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.828+0000 7f557bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f557c1028e0 0x7f557c197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43402/0 (socket says 172.21.15.23:43402) 2024-03-28T05:23:18.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.828+0000 7f557bfff640 1 -- 172.21.15.23:0/3646551435 learned_addr learned my addr 172.21.15.23:0/3646551435 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:18.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.828+0000 7f5580e94640 1 --2- 172.21.15.23:0/3646551435 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f557c1016f0 0x7f557c197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:18.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.828+0000 7f5581695640 1 --2- 172.21.15.23:0/3646551435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f557c1032a0 0x7f557c19efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:18.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.828+0000 7f557bfff640 1 -- 172.21.15.23:0/3646551435 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f557c1016f0 msgr2=0x7f557c197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:18.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.828+0000 7f557bfff640 1 --2- 172.21.15.23:0/3646551435 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f557c1016f0 0x7f557c197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.828+0000 7f557bfff640 1 -- 172.21.15.23:0/3646551435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f557c1032a0 msgr2=0x7f557c19efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:18.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.828+0000 7f557bfff640 1 --2- 172.21.15.23:0/3646551435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f557c1032a0 0x7f557c19efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:18.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.828+0000 7f557bfff640 1 -- 172.21.15.23:0/3646551435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f557c19f640 con 0x7f557c1028e0 2024-03-28T05:23:18.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.828+0000 7f5581695640 1 --2- 172.21.15.23:0/3646551435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f557c1032a0 0x7f557c19efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:23:18.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.828+0000 7f557bfff640 1 --2- 172.21.15.23:0/3646551435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f557c1028e0 0x7f557c197f60 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f556800e9f0 tx=0x7f556800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:18.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.828+0000 7f5579ffb640 1 -- 172.21.15.23:0/3646551435 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556800cd80 con 0x7f557c1028e0 2024-03-28T05:23:18.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.828+0000 7f5579ffb640 1 -- 172.21.15.23:0/3646551435 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5568005500 con 0x7f557c1028e0 2024-03-28T05:23:18.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.828+0000 7f5579ffb640 1 -- 172.21.15.23:0/3646551435 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568010690 con 0x7f557c1028e0 2024-03-28T05:23:18.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.828+0000 7f55830f8640 1 -- 172.21.15.23:0/3646551435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f557c19f930 con 0x7f557c1028e0 2024-03-28T05:23:18.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.828+0000 7f55830f8640 1 -- 172.21.15.23:0/3646551435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f557c19fe80 con 0x7f557c1028e0 2024-03-28T05:23:18.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.829+0000 7f55830f8640 1 -- 172.21.15.23:0/3646551435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f557c108a10 con 0x7f557c1028e0 2024-03-28T05:23:18.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.830+0000 7f5579ffb640 1 -- 172.21.15.23:0/3646551435 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f55680046c0 con 0x7f557c1028e0 2024-03-28T05:23:18.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.831+0000 7f5579ffb640 1 --2- 172.21.15.23:0/3646551435 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f556c0a2880 0x7f556c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:18.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.831+0000 7f5580e94640 1 --2- 172.21.15.23:0/3646551435 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f556c0a2880 0x7f556c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:18.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.831+0000 7f5579ffb640 1 -- 172.21.15.23:0/3646551435 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f55680cf220 con 0x7f557c1028e0 2024-03-28T05:23:18.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.832+0000 7f5580e94640 1 --2- 172.21.15.23:0/3646551435 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f556c0a2880 0x7f556c0a4d40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f557002c170 tx=0x7f557002df90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:18.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:18.836+0000 7f5579ffb640 1 -- 172.21.15.23:0/3646551435 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f556809bf10 con 0x7f557c1028e0 2024-03-28T05:23:19.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:18 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3512097243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:23:19.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:18 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3103699955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:23:19.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:18 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/115543208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:23:19.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.057+0000 7f55830f8640 1 -- 172.21.15.23:0/3646551435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f557c02d070 con 0x7f557c1028e0 2024-03-28T05:23:19.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.058+0000 7f5579ffb640 1 -- 172.21.15.23:0/3646551435 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11324 (secure 0 0 0) 0x7f556809b810 con 0x7f557c1028e0 2024-03-28T05:23:19.057 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:23:19.058 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/733913400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":733913400},{"type":"v1","addr":"172.21.15.23:6835","nonce":733913400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1972901387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1972901387},{"type":"v1","addr":"172.21.15.23:6837","nonce":1972901387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/2390673788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2390673788},{"type":"v1","addr":"172.21.15.23:6839","nonce":2390673788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/289263156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":289263156},{"type":"v1","addr":"172.21.15.23:6841","nonce":289263156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3313932732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3313932732},{"type":"v1","addr":"172.21.15.98:6835","nonce":3313932732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2695340024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2695340024},{"type":"v1","addr":"172.21.15.113:6835","nonce":2695340024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2750367469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2750367469},{"type":"v1","addr":"172.21.15.98:6837","nonce":2750367469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/3222726274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":3222726274},{"type":"v1","addr":"172.21.15.113:6837","nonce":3222726274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T05:21:07.616553+0000","modified":"2024-03-28T05:21:32.732132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/144113007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":144113007},{"type":"v1","addr":"172.21.15.98:6839","nonce":144113007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.113:6839/212060489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":212060489},{"type":"v1","addr":"172.21.15.113:6839","nonce":212060489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24487,"qdb_cluster":[24487]},"id":1}]} 2024-03-28T05:23:19.058 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 20 2024-03-28T05:23:19.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.060+0000 7f55830f8640 1 -- 172.21.15.23:0/3646551435 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f556c0a2880 msgr2=0x7f556c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:19.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.060+0000 7f55830f8640 1 --2- 172.21.15.23:0/3646551435 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f556c0a2880 0x7f556c0a4d40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f557002c170 tx=0x7f557002df90 comp rx=0 tx=0).stop 2024-03-28T05:23:19.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.060+0000 7f55830f8640 1 -- 172.21.15.23:0/3646551435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f557c1028e0 msgr2=0x7f557c197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:19.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.060+0000 7f55830f8640 1 --2- 172.21.15.23:0/3646551435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f557c1028e0 0x7f557c197f60 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f556800e9f0 tx=0x7f556800edf0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.060+0000 7f55830f8640 1 -- 172.21.15.23:0/3646551435 shutdown_connections 2024-03-28T05:23:19.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.060+0000 7f55830f8640 1 --2- 172.21.15.23:0/3646551435 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f556c0a2880 0x7f556c0a4d40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.060+0000 7f55830f8640 1 --2- 172.21.15.23:0/3646551435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f557c1032a0 0x7f557c19efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.060+0000 7f55830f8640 1 --2- 172.21.15.23:0/3646551435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f557c1028e0 0x7f557c197f60 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.060+0000 7f55830f8640 1 --2- 172.21.15.23:0/3646551435 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f557c1016f0 0x7f557c197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.060+0000 7f55830f8640 1 -- 172.21.15.23:0/3646551435 >> 172.21.15.23:0/3646551435 conn(0x7f557c0fcc40 msgr2=0x7f557c105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:19.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.060+0000 7f55830f8640 1 -- 172.21.15.23:0/3646551435 shutdown_connections 2024-03-28T05:23:19.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.060+0000 7f55830f8640 1 -- 172.21.15.23:0/3646551435 wait complete. 2024-03-28T05:23:19.071 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T05:23:19.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:18 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3512097243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:23:19.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:18 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3103699955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:23:19.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:18 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/115543208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:23:19.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.174+0000 7f8e593e8640 1 -- 172.21.15.23:0/1862269308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e54100640 msgr2=0x7f8e54100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:19.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.174+0000 7f8e593e8640 1 --2- 172.21.15.23:0/1862269308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e54100640 0x7f8e54100a60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009900 tx=0x7f8e4402bc30 comp rx=0 tx=0).stop 2024-03-28T05:23:19.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.174+0000 7f8e593e8640 1 -- 172.21.15.23:0/1862269308 shutdown_connections 2024-03-28T05:23:19.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.174+0000 7f8e593e8640 1 --2- 172.21.15.23:0/1862269308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e541021f0 0x7f8e54108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.174+0000 7f8e593e8640 1 --2- 172.21.15.23:0/1862269308 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e54101830 0x7f8e54101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.174+0000 7f8e593e8640 1 --2- 172.21.15.23:0/1862269308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e54100640 0x7f8e54100a60 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.174+0000 7f8e593e8640 1 -- 172.21.15.23:0/1862269308 >> 172.21.15.23:0/1862269308 conn(0x7f8e540fbbd0 msgr2=0x7f8e540fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:19.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.174+0000 7f8e593e8640 1 -- 172.21.15.23:0/1862269308 shutdown_connections 2024-03-28T05:23:19.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.174+0000 7f8e593e8640 1 -- 172.21.15.23:0/1862269308 wait complete. 2024-03-28T05:23:19.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.175+0000 7f8e593e8640 1 Processor -- start 2024-03-28T05:23:19.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.175+0000 7f8e593e8640 1 -- start start 2024-03-28T05:23:19.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.175+0000 7f8e593e8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e54100640 0x7f8e54104580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:19.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.175+0000 7f8e593e8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e54101830 0x7f8e54102c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:19.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.175+0000 7f8e593e8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e541021f0 0x7f8e54103150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:19.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.175+0000 7f8e593e8640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5410c220 con 0x7f8e541021f0 2024-03-28T05:23:19.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.175+0000 7f8e593e8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5410c0a0 con 0x7f8e54101830 2024-03-28T05:23:19.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.175+0000 7f8e593e8640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5410c3a0 con 0x7f8e54100640 2024-03-28T05:23:19.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.175+0000 7f8e537fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e541021f0 0x7f8e54103150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:19.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.175+0000 7f8e537fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e541021f0 0x7f8e54103150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43408/0 (socket says 172.21.15.23:43408) 2024-03-28T05:23:19.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.175+0000 7f8e537fe640 1 -- 172.21.15.23:0/1995459745 learned_addr learned my addr 172.21.15.23:0/1995459745 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:19.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.175+0000 7f8e52ffd640 1 --2- 172.21.15.23:0/1995459745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e54100640 0x7f8e54104580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:19.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.175+0000 7f8e4bfff640 1 --2- 172.21.15.23:0/1995459745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e54101830 0x7f8e54102c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:19.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.176+0000 7f8e537fe640 1 -- 172.21.15.23:0/1995459745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e54100640 msgr2=0x7f8e54104580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:19.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.176+0000 7f8e537fe640 1 --2- 172.21.15.23:0/1995459745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e54100640 0x7f8e54104580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.176+0000 7f8e537fe640 1 -- 172.21.15.23:0/1995459745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e54101830 msgr2=0x7f8e54102c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:19.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.176+0000 7f8e537fe640 1 --2- 172.21.15.23:0/1995459745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e54101830 0x7f8e54102c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.176+0000 7f8e537fe640 1 -- 172.21.15.23:0/1995459745 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e54103a10 con 0x7f8e541021f0 2024-03-28T05:23:19.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.176+0000 7f8e537fe640 1 --2- 172.21.15.23:0/1995459745 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e541021f0 0x7f8e54103150 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c009fd0 tx=0x7f8e4c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:19.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.176+0000 7f8e50ff9640 1 -- 172.21.15.23:0/1995459745 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c017070 con 0x7f8e541021f0 2024-03-28T05:23:19.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.176+0000 7f8e50ff9640 1 -- 172.21.15.23:0/1995459745 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e4c003e50 con 0x7f8e541021f0 2024-03-28T05:23:19.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.176+0000 7f8e593e8640 1 -- 172.21.15.23:0/1995459745 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e541a1bc0 con 0x7f8e541021f0 2024-03-28T05:23:19.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.176+0000 7f8e50ff9640 1 -- 172.21.15.23:0/1995459745 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c0106c0 con 0x7f8e541021f0 2024-03-28T05:23:19.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.176+0000 7f8e593e8640 1 -- 172.21.15.23:0/1995459745 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e541a2090 con 0x7f8e541021f0 2024-03-28T05:23:19.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.177+0000 7f8e593e8640 1 -- 172.21.15.23:0/1995459745 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e541a2260 con 0x7f8e541021f0 2024-03-28T05:23:19.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.178+0000 7f8e50ff9640 1 -- 172.21.15.23:0/1995459745 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8e4c003970 con 0x7f8e541021f0 2024-03-28T05:23:19.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.179+0000 7f8e50ff9640 1 --2- 172.21.15.23:0/1995459745 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8e400a2800 0x7f8e400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:19.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.179+0000 7f8e52ffd640 1 --2- 172.21.15.23:0/1995459745 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8e400a2800 0x7f8e400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:19.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.179+0000 7f8e50ff9640 1 -- 172.21.15.23:0/1995459745 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f8e4c0ceac0 con 0x7f8e541021f0 2024-03-28T05:23:19.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.179+0000 7f8e52ffd640 1 --2- 172.21.15.23:0/1995459745 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8e400a2800 0x7f8e400a4cc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8e44000c00 tx=0x7f8e44005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:19.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.184+0000 7f8e50ff9640 1 -- 172.21.15.23:0/1995459745 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8e4c09b160 con 0x7f8e541021f0 2024-03-28T05:23:19.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:18 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3512097243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:23:19.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:18 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3103699955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:23:19.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:18 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/115543208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:23:19.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.368+0000 7f8e593e8640 1 -- 172.21.15.23:0/1995459745 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e541a25f0 con 0x7f8e541021f0 2024-03-28T05:23:19.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.370+0000 7f8e50ff9640 1 -- 172.21.15.23:0/1995459745 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24695 (secure 0 0 0) 0x7f8e4c09f200 con 0x7f8e541021f0 2024-03-28T05:23:19.369 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:23:19.370 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":93,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","created":"2024-03-28T05:13:13.453653+0000","modified":"2024-03-28T05:23:10.695543+0000","last_up_change":"2024-03-28T05:20:20.221527+0000","last_in_change":"2024-03-28T05:20:00.365478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T05:16:32.986974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T05:21:06.103894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T05:21:06.827190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T05:21:10.507418+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"18150942-59a4-4c3b-83d4-c1fc3ccf1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6803","nonce":1219956881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6805","nonce":1219956881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6809","nonce":1219956881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6807","nonce":1219956881}]},"public_addr":"172.21.15.23:6803/1219956881","cluster_addr":"172.21.15.23:6805/1219956881","heartbeat_back_addr":"172.21.15.23:6809/1219956881","heartbeat_front_addr":"172.21.15.23:6807/1219956881","state":["exists","up"]},{"osd":1,"uuid":"ba6d1a95-5135-4670-95f9-7878c0e7bd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6801","nonce":555491234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6803","nonce":555491234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6807","nonce":555491234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6805","nonce":555491234}]},"public_addr":"172.21.15.98:6801/555491234","cluster_addr":"172.21.15.98:6803/555491234","heartbeat_back_addr":"172.21.15.98:6807/555491234","heartbeat_front_addr":"172.21.15.98:6805/555491234","state":["exists","up"]},{"osd":2,"uuid":"873bdd60-2a47-403b-b1f4-ee9226a95067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6801","nonce":1535856914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6803","nonce":1535856914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6807","nonce":1535856914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6805","nonce":1535856914}]},"public_addr":"172.21.15.113:6801/1535856914","cluster_addr":"172.21.15.113:6803/1535856914","heartbeat_back_addr":"172.21.15.113:6807/1535856914","heartbeat_front_addr":"172.21.15.113:6805/1535856914","state":["exists","up"]},{"osd":3,"uuid":"83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6811","nonce":2949415020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6813","nonce":2949415020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6817","nonce":2949415020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6815","nonce":2949415020}]},"public_addr":"172.21.15.23:6811/2949415020","cluster_addr":"172.21.15.23:6813/2949415020","heartbeat_back_addr":"172.21.15.23:6817/2949415020","heartbeat_front_addr":"172.21.15.23:6815/2949415020","state":["exists","up"]},{"osd":4,"uuid":"76a90bac-b221-40a3-8238-40e8049cec36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6809","nonce":2615902819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6811","nonce":2615902819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6815","nonce":2615902819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6813","nonce":2615902819}]},"public_addr":"172.21.15.98:6809/2615902819","cluster_addr":"172.21.15.98:6811/2615902819","heartbeat_back_addr":"172.21.15.98:6815/2615902819","heartbeat_front_addr":"172.21.15.98:6813/2615902819","state":["exists","up"]},{"osd":5,"uuid":"b53fdb55-a8df-40a8-bd1b-8bcef634e26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6809","nonce":625271528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6811","nonce":625271528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6815","nonce":625271528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6813","nonce":625271528}]},"public_addr":"172.21.15.113:6809/625271528","cluster_addr":"172.21.15.113:6811/625271528","heartbeat_back_addr":"172.21.15.113:6815/625271528","heartbeat_front_addr":"172.21.15.113:6813/625271528","state":["exists","up"]},{"osd":6,"uuid":"f00283d2-8d67-4e97-a575-9a9d3b55d5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6819","nonce":4188464176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6821","nonce":4188464176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6825","nonce":4188464176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6823","nonce":4188464176}]},"public_addr":"172.21.15.23:6819/4188464176","cluster_addr":"172.21.15.23:6821/4188464176","heartbeat_back_addr":"172.21.15.23:6825/4188464176","heartbeat_front_addr":"172.21.15.23:6823/4188464176","state":["exists","up"]},{"osd":7,"uuid":"edc008a7-b07b-40a8-88b7-1614ba38a775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6817","nonce":1985546028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6819","nonce":1985546028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6823","nonce":1985546028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6821","nonce":1985546028}]},"public_addr":"172.21.15.98:6817/1985546028","cluster_addr":"172.21.15.98:6819/1985546028","heartbeat_back_addr":"172.21.15.98:6823/1985546028","heartbeat_front_addr":"172.21.15.98:6821/1985546028","state":["exists","up"]},{"osd":8,"uuid":"56e7b2d1-81f0-47b2-8638-f29832bed8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6817","nonce":3534876081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6819","nonce":3534876081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6823","nonce":3534876081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6821","nonce":3534876081}]},"public_addr":"172.21.15.113:6817/3534876081","cluster_addr":"172.21.15.113:6819/3534876081","heartbeat_back_addr":"172.21.15.113:6823/3534876081","heartbeat_front_addr":"172.21.15.113:6821/3534876081","state":["exists","up"]},{"osd":9,"uuid":"a2e90caa-6e9c-42bf-9f18-81f7f47880cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6827","nonce":672264207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6829","nonce":672264207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6833","nonce":672264207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6831","nonce":672264207}]},"public_addr":"172.21.15.23:6827/672264207","cluster_addr":"172.21.15.23:6829/672264207","heartbeat_back_addr":"172.21.15.23:6833/672264207","heartbeat_front_addr":"172.21.15.23:6831/672264207","state":["exists","up"]},{"osd":10,"uuid":"0eda917f-020c-437f-95ae-b26203f6c68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6825","nonce":2699920595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6827","nonce":2699920595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6831","nonce":2699920595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6829","nonce":2699920595}]},"public_addr":"172.21.15.98:6825/2699920595","cluster_addr":"172.21.15.98:6827/2699920595","heartbeat_back_addr":"172.21.15.98:6831/2699920595","heartbeat_front_addr":"172.21.15.98:6829/2699920595","state":["exists","up"]},{"osd":11,"uuid":"aae78501-78a8-469a-be20-3006f5ec5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6825","nonce":2113644020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6827","nonce":2113644020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6831","nonce":2113644020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6829","nonce":2113644020}]},"public_addr":"172.21.15.113:6825/2113644020","cluster_addr":"172.21.15.113:6827/2113644020","heartbeat_back_addr":"172.21.15.113:6831/2113644020","heartbeat_front_addr":"172.21.15.113:6829/2113644020","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:15:43.527475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:07.006538+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:30.594176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:55.810999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:20.595811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:45.474138+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:10.500409+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:35.787638+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:01.001188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:26.761115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:52.952752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:20:18.559832+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":7}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:6800/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1876451381":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2253559862":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2653562762":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/3337463830":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6800/1735221490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/2784771635":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2389902490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6801/1735221490":"2024-03-29T05:13:42.579607+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T05:23:19.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.372+0000 7f8e593e8640 1 -- 172.21.15.23:0/1995459745 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8e400a2800 msgr2=0x7f8e400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:19.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.372+0000 7f8e593e8640 1 --2- 172.21.15.23:0/1995459745 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8e400a2800 0x7f8e400a4cc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8e44000c00 tx=0x7f8e44005fd0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.372+0000 7f8e593e8640 1 -- 172.21.15.23:0/1995459745 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e541021f0 msgr2=0x7f8e54103150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:19.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.372+0000 7f8e593e8640 1 --2- 172.21.15.23:0/1995459745 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e541021f0 0x7f8e54103150 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c009fd0 tx=0x7f8e4c00b260 comp rx=0 tx=0).stop 2024-03-28T05:23:19.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.372+0000 7f8e593e8640 1 -- 172.21.15.23:0/1995459745 shutdown_connections 2024-03-28T05:23:19.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.372+0000 7f8e593e8640 1 --2- 172.21.15.23:0/1995459745 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f8e400a2800 0x7f8e400a4cc0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.372+0000 7f8e593e8640 1 --2- 172.21.15.23:0/1995459745 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e541021f0 0x7f8e54103150 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.372+0000 7f8e593e8640 1 --2- 172.21.15.23:0/1995459745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8e54101830 0x7f8e54102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.372+0000 7f8e593e8640 1 --2- 172.21.15.23:0/1995459745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e54100640 0x7f8e54104580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.372+0000 7f8e593e8640 1 -- 172.21.15.23:0/1995459745 >> 172.21.15.23:0/1995459745 conn(0x7f8e540fbbd0 msgr2=0x7f8e540fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:19.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.372+0000 7f8e593e8640 1 -- 172.21.15.23:0/1995459745 shutdown_connections 2024-03-28T05:23:19.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.372+0000 7f8e593e8640 1 -- 172.21.15.23:0/1995459745 wait complete. 2024-03-28T05:23:19.383 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T05:16:32.986974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T05:21:06.103894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T05:21:06.827190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T05:21:10.507418+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T05:23:19.383 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T05:23:19.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.489+0000 7f566b390640 1 -- 172.21.15.23:0/3191729144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56641028c0 msgr2=0x7f5664102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:19.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.489+0000 7f566b390640 1 --2- 172.21.15.23:0/3191729144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56641028c0 0x7f5664102d40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f5658009900 tx=0x7f565802bc30 comp rx=0 tx=0).stop 2024-03-28T05:23:19.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.489+0000 7f566b390640 1 -- 172.21.15.23:0/3191729144 shutdown_connections 2024-03-28T05:23:19.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.489+0000 7f566b390640 1 --2- 172.21.15.23:0/3191729144 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5664103280 0x7f5664109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.489+0000 7f566b390640 1 --2- 172.21.15.23:0/3191729144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56641028c0 0x7f5664102d40 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.489+0000 7f566b390640 1 --2- 172.21.15.23:0/3191729144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56641016d0 0x7f5664101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.489+0000 7f566b390640 1 -- 172.21.15.23:0/3191729144 >> 172.21.15.23:0/3191729144 conn(0x7f56640fcc40 msgr2=0x7f56640ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:19.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.489+0000 7f566b390640 1 -- 172.21.15.23:0/3191729144 shutdown_connections 2024-03-28T05:23:19.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.490+0000 7f566b390640 1 -- 172.21.15.23:0/3191729144 wait complete. 2024-03-28T05:23:19.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.490+0000 7f566b390640 1 Processor -- start 2024-03-28T05:23:19.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.490+0000 7f566b390640 1 -- start start 2024-03-28T05:23:19.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.490+0000 7f566b390640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56641016d0 0x7f56641979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:19.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.490+0000 7f566b390640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56641028c0 0x7f5664197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:19.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.490+0000 7f566b390640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5664103280 0x7f566419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:19.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.490+0000 7f566b390640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566410d330 con 0x7f56641028c0 2024-03-28T05:23:19.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.490+0000 7f566b390640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566410d1b0 con 0x7f56641016d0 2024-03-28T05:23:19.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.490+0000 7f566b390640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566410d4b0 con 0x7f5664103280 2024-03-28T05:23:19.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.490+0000 7f566892b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56641028c0 0x7f5664197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:19.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.490+0000 7f566892b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56641028c0 0x7f5664197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43418/0 (socket says 172.21.15.23:43418) 2024-03-28T05:23:19.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.490+0000 7f566892b640 1 -- 172.21.15.23:0/261375505 learned_addr learned my addr 172.21.15.23:0/261375505 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:19.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.490+0000 7f566992d640 1 --2- 172.21.15.23:0/261375505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5664103280 0x7f566419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:19.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.490+0000 7f566912c640 1 --2- 172.21.15.23:0/261375505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56641016d0 0x7f56641979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:19.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.491+0000 7f566892b640 1 -- 172.21.15.23:0/261375505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5664103280 msgr2=0x7f566419ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:19.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.491+0000 7f566892b640 1 --2- 172.21.15.23:0/261375505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5664103280 0x7f566419ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.491+0000 7f566892b640 1 -- 172.21.15.23:0/261375505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56641016d0 msgr2=0x7f56641979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:19.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.491+0000 7f566892b640 1 --2- 172.21.15.23:0/261375505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56641016d0 0x7f56641979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.491+0000 7f566892b640 1 -- 172.21.15.23:0/261375505 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566419f5f0 con 0x7f56641028c0 2024-03-28T05:23:19.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.491+0000 7f566892b640 1 --2- 172.21.15.23:0/261375505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56641028c0 0x7f5664197f10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f5658009230 tx=0x7f5658005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:19.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.491+0000 7f56627fc640 1 -- 172.21.15.23:0/261375505 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5658041070 con 0x7f56641028c0 2024-03-28T05:23:19.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.491+0000 7f56627fc640 1 -- 172.21.15.23:0/261375505 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56580044d0 con 0x7f56641028c0 2024-03-28T05:23:19.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.491+0000 7f566b390640 1 -- 172.21.15.23:0/261375505 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f566419f880 con 0x7f56641028c0 2024-03-28T05:23:19.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.491+0000 7f56627fc640 1 -- 172.21.15.23:0/261375505 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5658034440 con 0x7f56641028c0 2024-03-28T05:23:19.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.491+0000 7f566b390640 1 -- 172.21.15.23:0/261375505 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f566419fdc0 con 0x7f56641028c0 2024-03-28T05:23:19.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.492+0000 7f566b390640 1 -- 172.21.15.23:0/261375505 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5664108a70 con 0x7f56641028c0 2024-03-28T05:23:19.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.493+0000 7f56627fc640 1 -- 172.21.15.23:0/261375505 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5658044070 con 0x7f56641028c0 2024-03-28T05:23:19.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.493+0000 7f56627fc640 1 --2- 172.21.15.23:0/261375505 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f564c0a2880 0x7f564c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:19.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.494+0000 7f566912c640 1 --2- 172.21.15.23:0/261375505 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f564c0a2880 0x7f564c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:19.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.494+0000 7f56627fc640 1 -- 172.21.15.23:0/261375505 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f56580f0970 con 0x7f56641028c0 2024-03-28T05:23:19.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.494+0000 7f566912c640 1 --2- 172.21.15.23:0/261375505 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f564c0a2880 0x7f564c0a4d40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5664102720 tx=0x7f565400a590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:19.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.498+0000 7f56627fc640 1 -- 172.21.15.23:0/261375505 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f56580bd010 con 0x7f56641028c0 2024-03-28T05:23:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.681+0000 7f566b390640 1 -- 172.21.15.23:0/261375505 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f56640622a0 con 0x7f56641028c0 2024-03-28T05:23:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.682+0000 7f56627fc640 1 -- 172.21.15.23:0/261375505 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f56580c10b0 con 0x7f56641028c0 2024-03-28T05:23:19.681 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-28T05:23:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.684+0000 7f566b390640 1 -- 172.21.15.23:0/261375505 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f564c0a2880 msgr2=0x7f564c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.684+0000 7f566b390640 1 --2- 172.21.15.23:0/261375505 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f564c0a2880 0x7f564c0a4d40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5664102720 tx=0x7f565400a590 comp rx=0 tx=0).stop 2024-03-28T05:23:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.684+0000 7f566b390640 1 -- 172.21.15.23:0/261375505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56641028c0 msgr2=0x7f5664197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.684+0000 7f566b390640 1 --2- 172.21.15.23:0/261375505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56641028c0 0x7f5664197f10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f5658009230 tx=0x7f5658005d50 comp rx=0 tx=0).stop 2024-03-28T05:23:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.684+0000 7f566b390640 1 -- 172.21.15.23:0/261375505 shutdown_connections 2024-03-28T05:23:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.684+0000 7f566b390640 1 --2- 172.21.15.23:0/261375505 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f564c0a2880 0x7f564c0a4d40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.684+0000 7f566b390640 1 --2- 172.21.15.23:0/261375505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5664103280 0x7f566419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.684+0000 7f566b390640 1 --2- 172.21.15.23:0/261375505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56641028c0 0x7f5664197f10 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.684+0000 7f566b390640 1 --2- 172.21.15.23:0/261375505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56641016d0 0x7f56641979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.684+0000 7f566b390640 1 -- 172.21.15.23:0/261375505 >> 172.21.15.23:0/261375505 conn(0x7f56640fcc40 msgr2=0x7f5664105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.684+0000 7f566b390640 1 -- 172.21.15.23:0/261375505 shutdown_connections 2024-03-28T05:23:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.684+0000 7f566b390640 1 -- 172.21.15.23:0/261375505 wait complete. 2024-03-28T05:23:19.695 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T05:23:19.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.801+0000 7f0006577640 1 -- 172.21.15.23:0/3083648900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00001005c0 msgr2=0x7f00001009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:19.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.801+0000 7f0006577640 1 --2- 172.21.15.23:0/3083648900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00001005c0 0x7f00001009e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7efff0009900 tx=0x7efff002bc30 comp rx=0 tx=0).stop 2024-03-28T05:23:19.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.801+0000 7f0006577640 1 -- 172.21.15.23:0/3083648900 shutdown_connections 2024-03-28T05:23:19.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.801+0000 7f0006577640 1 --2- 172.21.15.23:0/3083648900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0000102170 0x7f0000108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.801+0000 7f0006577640 1 --2- 172.21.15.23:0/3083648900 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00001017b0 0x7f0000101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.801+0000 7f0006577640 1 --2- 172.21.15.23:0/3083648900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00001005c0 0x7f00001009e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.801+0000 7f0006577640 1 -- 172.21.15.23:0/3083648900 >> 172.21.15.23:0/3083648900 conn(0x7f00000fbb50 msgr2=0x7f00000fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:19.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.801+0000 7f0006577640 1 -- 172.21.15.23:0/3083648900 shutdown_connections 2024-03-28T05:23:19.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.801+0000 7f0006577640 1 -- 172.21.15.23:0/3083648900 wait complete. 2024-03-28T05:23:19.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.801+0000 7f0006577640 1 Processor -- start 2024-03-28T05:23:19.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.801+0000 7f0006577640 1 -- start start 2024-03-28T05:23:19.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.802+0000 7f0006577640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00001005c0 0x7f000006a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:19.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.802+0000 7f0006577640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00001017b0 0x7f000019c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:19.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.802+0000 7f0006577640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0000102170 0x7f000019ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:19.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.802+0000 7f0006577640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000010c230 con 0x7f00001017b0 2024-03-28T05:23:19.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.802+0000 7f0006577640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000010c0b0 con 0x7f00001005c0 2024-03-28T05:23:19.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.802+0000 7f0006577640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000010c3b0 con 0x7f0000102170 2024-03-28T05:23:19.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.802+0000 7effff7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00001017b0 0x7f000019c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:19.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.802+0000 7effff7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00001017b0 0x7f000019c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43440/0 (socket says 172.21.15.23:43440) 2024-03-28T05:23:19.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.802+0000 7effff7fe640 1 -- 172.21.15.23:0/2811488290 learned_addr learned my addr 172.21.15.23:0/2811488290 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:19.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.802+0000 7f0004b14640 1 --2- 172.21.15.23:0/2811488290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0000102170 0x7f000019ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:19.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.802+0000 7efffffff640 1 --2- 172.21.15.23:0/2811488290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00001005c0 0x7f000006a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:19.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.802+0000 7effff7fe640 1 -- 172.21.15.23:0/2811488290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0000102170 msgr2=0x7f000019ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:19.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.802+0000 7effff7fe640 1 --2- 172.21.15.23:0/2811488290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0000102170 0x7f000019ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.802+0000 7effff7fe640 1 -- 172.21.15.23:0/2811488290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00001005c0 msgr2=0x7f000006a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:19.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.802+0000 7effff7fe640 1 --2- 172.21.15.23:0/2811488290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00001005c0 0x7f000006a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.802+0000 7effff7fe640 1 -- 172.21.15.23:0/2811488290 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f000019f5d0 con 0x7f00001017b0 2024-03-28T05:23:19.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.802+0000 7effff7fe640 1 --2- 172.21.15.23:0/2811488290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00001017b0 0x7f000019c6d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7efff400e9f0 tx=0x7efff400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:19.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.802+0000 7efffd7fa640 1 -- 172.21.15.23:0/2811488290 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff400cd80 con 0x7f00001017b0 2024-03-28T05:23:19.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.802+0000 7efffd7fa640 1 -- 172.21.15.23:0/2811488290 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efff4005500 con 0x7f00001017b0 2024-03-28T05:23:19.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.802+0000 7f0006577640 1 -- 172.21.15.23:0/2811488290 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f000019f8c0 con 0x7f00001017b0 2024-03-28T05:23:19.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.802+0000 7efffd7fa640 1 -- 172.21.15.23:0/2811488290 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff4010690 con 0x7f00001017b0 2024-03-28T05:23:19.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.802+0000 7f0006577640 1 -- 172.21.15.23:0/2811488290 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f000019fdc0 con 0x7f00001017b0 2024-03-28T05:23:19.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.803+0000 7f0006577640 1 -- 172.21.15.23:0/2811488290 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00001078e0 con 0x7f00001017b0 2024-03-28T05:23:19.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.805+0000 7efffd7fa640 1 -- 172.21.15.23:0/2811488290 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7efff40046c0 con 0x7f00001017b0 2024-03-28T05:23:19.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.805+0000 7efffd7fa640 1 --2- 172.21.15.23:0/2811488290 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7effe80a2800 0x7effe80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:19.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.805+0000 7efffd7fa640 1 -- 172.21.15.23:0/2811488290 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7efff40cf490 con 0x7f00001017b0 2024-03-28T05:23:19.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.805+0000 7efffffff640 1 --2- 172.21.15.23:0/2811488290 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7effe80a2800 0x7effe80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:19.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.806+0000 7efffffff640 1 --2- 172.21.15.23:0/2811488290 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7effe80a2800 0x7effe80a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efff002c7b0 tx=0x7efff0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:19.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.809+0000 7efffd7fa640 1 -- 172.21.15.23:0/2811488290 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efff4053740 con 0x7f00001017b0 2024-03-28T05:23:19.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.993+0000 7f0006577640 1 -- 172.21.15.23:0/2811488290 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f000002d2c0 con 0x7f00001017b0 2024-03-28T05:23:19.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.994+0000 7efffd7fa640 1 -- 172.21.15.23:0/2811488290 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7efff409c180 con 0x7f00001017b0 2024-03-28T05:23:19.993 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T05:23:19.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.996+0000 7f0006577640 1 -- 172.21.15.23:0/2811488290 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7effe80a2800 msgr2=0x7effe80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:19.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.996+0000 7f0006577640 1 --2- 172.21.15.23:0/2811488290 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7effe80a2800 0x7effe80a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7efff002c7b0 tx=0x7efff0005fd0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.996+0000 7f0006577640 1 -- 172.21.15.23:0/2811488290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00001017b0 msgr2=0x7f000019c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:19.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.996+0000 7f0006577640 1 --2- 172.21.15.23:0/2811488290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00001017b0 0x7f000019c6d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7efff400e9f0 tx=0x7efff400edf0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.996+0000 7f0006577640 1 -- 172.21.15.23:0/2811488290 shutdown_connections 2024-03-28T05:23:19.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.996+0000 7f0006577640 1 --2- 172.21.15.23:0/2811488290 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7effe80a2800 0x7effe80a4cc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.996+0000 7f0006577640 1 --2- 172.21.15.23:0/2811488290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0000102170 0x7f000019ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.996+0000 7f0006577640 1 --2- 172.21.15.23:0/2811488290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00001017b0 0x7f000019c6d0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.996+0000 7f0006577640 1 --2- 172.21.15.23:0/2811488290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00001005c0 0x7f000006a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:19.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.996+0000 7f0006577640 1 -- 172.21.15.23:0/2811488290 >> 172.21.15.23:0/2811488290 conn(0x7f00000fbb50 msgr2=0x7f0000104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:19.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.996+0000 7f0006577640 1 -- 172.21.15.23:0/2811488290 shutdown_connections 2024-03-28T05:23:19.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:19.996+0000 7f0006577640 1 -- 172.21.15.23:0/2811488290 wait complete. 2024-03-28T05:23:20.005 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:19 smithi023 ceph-mon[31954]: pgmap v358: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 565 B/s rd, 1 op/s; 64 B/s, 0 objects/s recovering 2024-03-28T05:23:20.005 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:19 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3646551435' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:23:20.005 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:19 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1995459745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:23:20.005 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:19 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/261375505' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:23:20.006 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T05:23:20.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.112+0000 7fb1e5177640 1 -- 172.21.15.23:0/4011617502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1e01073a0 msgr2=0x7fb1e01097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:20.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.112+0000 7fb1e5177640 1 --2- 172.21.15.23:0/4011617502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1e01073a0 0x7fb1e01097f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8009ca0 tx=0x7fb1d802c0f0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.112+0000 7fb1e5177640 1 -- 172.21.15.23:0/4011617502 shutdown_connections 2024-03-28T05:23:20.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.112+0000 7fb1e5177640 1 --2- 172.21.15.23:0/4011617502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1e01073a0 0x7fb1e01097f0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.112+0000 7fb1e5177640 1 --2- 172.21.15.23:0/4011617502 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1e0104a70 0x7fb1e0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.112+0000 7fb1e5177640 1 --2- 172.21.15.23:0/4011617502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e0102140 0x7fb1e0104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.112+0000 7fb1e5177640 1 -- 172.21.15.23:0/4011617502 >> 172.21.15.23:0/4011617502 conn(0x7fb1e00fbb50 msgr2=0x7fb1e00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:20.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.112+0000 7fb1e5177640 1 -- 172.21.15.23:0/4011617502 shutdown_connections 2024-03-28T05:23:20.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.112+0000 7fb1e5177640 1 -- 172.21.15.23:0/4011617502 wait complete. 2024-03-28T05:23:20.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.113+0000 7fb1e5177640 1 Processor -- start 2024-03-28T05:23:20.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.113+0000 7fb1e5177640 1 -- start start 2024-03-28T05:23:20.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.113+0000 7fb1e5177640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e0102140 0x7fb1e0197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:20.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.113+0000 7fb1e5177640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1e0104a70 0x7fb1e0197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:20.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.113+0000 7fb1e5177640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1e01073a0 0x7fb1e019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:20.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.113+0000 7fb1e5177640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e010d340 con 0x7fb1e0104a70 2024-03-28T05:23:20.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.113+0000 7fb1e5177640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e010d1c0 con 0x7fb1e0102140 2024-03-28T05:23:20.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.113+0000 7fb1e5177640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e010d4c0 con 0x7fb1e01073a0 2024-03-28T05:23:20.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.114+0000 7fb1de59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1e0104a70 0x7fb1e0197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:20.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.114+0000 7fb1de59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1e0104a70 0x7fb1e0197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43446/0 (socket says 172.21.15.23:43446) 2024-03-28T05:23:20.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.114+0000 7fb1de59c640 1 -- 172.21.15.23:0/3362725596 learned_addr learned my addr 172.21.15.23:0/3362725596 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:20.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.114+0000 7fb1df59e640 1 --2- 172.21.15.23:0/3362725596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1e01073a0 0x7fb1e019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:20.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.114+0000 7fb1ded9d640 1 --2- 172.21.15.23:0/3362725596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e0102140 0x7fb1e0197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:20.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.114+0000 7fb1de59c640 1 -- 172.21.15.23:0/3362725596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1e01073a0 msgr2=0x7fb1e019ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:20.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.114+0000 7fb1de59c640 1 --2- 172.21.15.23:0/3362725596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1e01073a0 0x7fb1e019ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.114+0000 7fb1de59c640 1 -- 172.21.15.23:0/3362725596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e0102140 msgr2=0x7fb1e0197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:20.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.114+0000 7fb1de59c640 1 --2- 172.21.15.23:0/3362725596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e0102140 0x7fb1e0197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.114+0000 7fb1de59c640 1 -- 172.21.15.23:0/3362725596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e019f5a0 con 0x7fb1e0104a70 2024-03-28T05:23:20.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.114+0000 7fb1ded9d640 1 --2- 172.21.15.23:0/3362725596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e0102140 0x7fb1e0197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:23:20.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.114+0000 7fb1de59c640 1 --2- 172.21.15.23:0/3362725596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1e0104a70 0x7fb1e0197ec0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb1d400d770 tx=0x7fb1d400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:20.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.114+0000 7fb1c7fff640 1 -- 172.21.15.23:0/3362725596 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4017070 con 0x7fb1e0104a70 2024-03-28T05:23:20.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.114+0000 7fb1c7fff640 1 -- 172.21.15.23:0/3362725596 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb1d4009c30 con 0x7fb1e0104a70 2024-03-28T05:23:20.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.114+0000 7fb1e5177640 1 -- 172.21.15.23:0/3362725596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e019f890 con 0x7fb1e0104a70 2024-03-28T05:23:20.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.114+0000 7fb1c7fff640 1 -- 172.21.15.23:0/3362725596 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d400f620 con 0x7fb1e0104a70 2024-03-28T05:23:20.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.114+0000 7fb1e5177640 1 -- 172.21.15.23:0/3362725596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e019fde0 con 0x7fb1e0104a70 2024-03-28T05:23:20.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.115+0000 7fb1e5177640 1 -- 172.21.15.23:0/3362725596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e00fd640 con 0x7fb1e0104a70 2024-03-28T05:23:20.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:19 smithi098 ceph-mon[36087]: pgmap v358: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 565 B/s rd, 1 op/s; 64 B/s, 0 objects/s recovering 2024-03-28T05:23:20.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:19 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3646551435' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:23:20.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:19 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1995459745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:23:20.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:19 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/261375505' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:23:20.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.116+0000 7fb1c7fff640 1 -- 172.21.15.23:0/3362725596 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb1d401a070 con 0x7fb1e0104a70 2024-03-28T05:23:20.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.117+0000 7fb1c7fff640 1 --2- 172.21.15.23:0/3362725596 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb1c80a2800 0x7fb1c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:20.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.117+0000 7fb1c7fff640 1 -- 172.21.15.23:0/3362725596 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fb1d40ce8e0 con 0x7fb1e0104a70 2024-03-28T05:23:20.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.117+0000 7fb1ded9d640 1 --2- 172.21.15.23:0/3362725596 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb1c80a2800 0x7fb1c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:20.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.117+0000 7fb1ded9d640 1 --2- 172.21.15.23:0/3362725596 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb1c80a2800 0x7fb1c80a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb1d00058c0 tx=0x7fb1d0005ec0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:20.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.121+0000 7fb1c7fff640 1 -- 172.21.15.23:0/3362725596 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb1d4009de0 con 0x7fb1e0104a70 2024-03-28T05:23:20.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:19 smithi113 ceph-mon[37164]: pgmap v358: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 565 B/s rd, 1 op/s; 64 B/s, 0 objects/s recovering 2024-03-28T05:23:20.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:19 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3646551435' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:23:20.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:19 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1995459745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:23:20.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:19 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/261375505' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:23:20.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.307+0000 7fb1e5177640 1 -- 172.21.15.23:0/3362725596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb1e00622a0 con 0x7fb1e0104a70 2024-03-28T05:23:20.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.308+0000 7fb1c7fff640 1 -- 172.21.15.23:0/3362725596 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb1d409b5d0 con 0x7fb1e0104a70 2024-03-28T05:23:20.307 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-28T05:23:20.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.310+0000 7fb1e5177640 1 -- 172.21.15.23:0/3362725596 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb1c80a2800 msgr2=0x7fb1c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:20.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.310+0000 7fb1e5177640 1 --2- 172.21.15.23:0/3362725596 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb1c80a2800 0x7fb1c80a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb1d00058c0 tx=0x7fb1d0005ec0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.310+0000 7fb1e5177640 1 -- 172.21.15.23:0/3362725596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1e0104a70 msgr2=0x7fb1e0197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:20.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.310+0000 7fb1e5177640 1 --2- 172.21.15.23:0/3362725596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1e0104a70 0x7fb1e0197ec0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb1d400d770 tx=0x7fb1d400dc30 comp rx=0 tx=0).stop 2024-03-28T05:23:20.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.310+0000 7fb1e5177640 1 -- 172.21.15.23:0/3362725596 shutdown_connections 2024-03-28T05:23:20.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.310+0000 7fb1e5177640 1 --2- 172.21.15.23:0/3362725596 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fb1c80a2800 0x7fb1c80a4cc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.310+0000 7fb1e5177640 1 --2- 172.21.15.23:0/3362725596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1e01073a0 0x7fb1e019ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.310+0000 7fb1e5177640 1 --2- 172.21.15.23:0/3362725596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1e0104a70 0x7fb1e0197ec0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.310+0000 7fb1e5177640 1 --2- 172.21.15.23:0/3362725596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb1e0102140 0x7fb1e0197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.310+0000 7fb1e5177640 1 -- 172.21.15.23:0/3362725596 >> 172.21.15.23:0/3362725596 conn(0x7fb1e00fbb50 msgr2=0x7fb1e0102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:20.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.310+0000 7fb1e5177640 1 -- 172.21.15.23:0/3362725596 shutdown_connections 2024-03-28T05:23:20.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.310+0000 7fb1e5177640 1 -- 172.21.15.23:0/3362725596 wait complete. 2024-03-28T05:23:20.320 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T05:23:20.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.426+0000 7fe0796dd640 1 -- 172.21.15.23:0/2878360824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe074107390 msgr2=0x7fe0741097e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:20.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.426+0000 7fe0796dd640 1 --2- 172.21.15.23:0/2878360824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe074107390 0x7fe0741097e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe06c009ca0 tx=0x7fe06c02c0f0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.426+0000 7fe0796dd640 1 -- 172.21.15.23:0/2878360824 shutdown_connections 2024-03-28T05:23:20.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.426+0000 7fe0796dd640 1 --2- 172.21.15.23:0/2878360824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe074107390 0x7fe0741097e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.426+0000 7fe0796dd640 1 --2- 172.21.15.23:0/2878360824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe074104a60 0x7fe074106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.426+0000 7fe0796dd640 1 --2- 172.21.15.23:0/2878360824 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe074102130 0x7fe074104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.426+0000 7fe0796dd640 1 -- 172.21.15.23:0/2878360824 >> 172.21.15.23:0/2878360824 conn(0x7fe0740fbb40 msgr2=0x7fe0740fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:20.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.426+0000 7fe0796dd640 1 -- 172.21.15.23:0/2878360824 shutdown_connections 2024-03-28T05:23:20.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.426+0000 7fe0796dd640 1 -- 172.21.15.23:0/2878360824 wait complete. 2024-03-28T05:23:20.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.426+0000 7fe0796dd640 1 Processor -- start 2024-03-28T05:23:20.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.427+0000 7fe0796dd640 1 -- start start 2024-03-28T05:23:20.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.427+0000 7fe0796dd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe074102130 0x7fe074197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:20.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.427+0000 7fe0796dd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe074104a60 0x7fe074197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:20.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.427+0000 7fe0796dd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe074107390 0x7fe07419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:20.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.427+0000 7fe0796dd640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07410d3c0 con 0x7fe074107390 2024-03-28T05:23:20.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.427+0000 7fe0796dd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07410d240 con 0x7fe074102130 2024-03-28T05:23:20.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.427+0000 7fe0796dd640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07410d540 con 0x7fe074104a60 2024-03-28T05:23:20.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.427+0000 7fe0737fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe074107390 0x7fe07419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:20.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.427+0000 7fe0737fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe074107390 0x7fe07419efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43464/0 (socket says 172.21.15.23:43464) 2024-03-28T05:23:20.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.427+0000 7fe0737fe640 1 -- 172.21.15.23:0/549816186 learned_addr learned my addr 172.21.15.23:0/549816186 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:20.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.427+0000 7fe072ffd640 1 --2- 172.21.15.23:0/549816186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe074102130 0x7fe074197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:20.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.427+0000 7fe06a7fc640 1 --2- 172.21.15.23:0/549816186 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe074104a60 0x7fe074197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:20.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.427+0000 7fe0737fe640 1 -- 172.21.15.23:0/549816186 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe074104a60 msgr2=0x7fe074197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:20.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.427+0000 7fe0737fe640 1 --2- 172.21.15.23:0/549816186 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe074104a60 0x7fe074197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.427+0000 7fe0737fe640 1 -- 172.21.15.23:0/549816186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe074102130 msgr2=0x7fe074197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:20.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.427+0000 7fe0737fe640 1 --2- 172.21.15.23:0/549816186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe074102130 0x7fe074197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.427+0000 7fe0737fe640 1 -- 172.21.15.23:0/549816186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07419f630 con 0x7fe074107390 2024-03-28T05:23:20.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.427+0000 7fe0737fe640 1 --2- 172.21.15.23:0/549816186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe074107390 0x7fe07419efc0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe06c037040 tx=0x7fe06c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:20.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.427+0000 7fe070ff9640 1 -- 172.21.15.23:0/549816186 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06c041020 con 0x7fe074107390 2024-03-28T05:23:20.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.427+0000 7fe0796dd640 1 -- 172.21.15.23:0/549816186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe07419f8c0 con 0x7fe074107390 2024-03-28T05:23:20.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.427+0000 7fe070ff9640 1 -- 172.21.15.23:0/549816186 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe06c034cd0 con 0x7fe074107390 2024-03-28T05:23:20.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.428+0000 7fe070ff9640 1 -- 172.21.15.23:0/549816186 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06c039040 con 0x7fe074107390 2024-03-28T05:23:20.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.428+0000 7fe0796dd640 1 -- 172.21.15.23:0/549816186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe07419fdb0 con 0x7fe074107390 2024-03-28T05:23:20.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.428+0000 7fe0796dd640 1 -- 172.21.15.23:0/549816186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0740fd630 con 0x7fe074107390 2024-03-28T05:23:20.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.429+0000 7fe070ff9640 1 -- 172.21.15.23:0/549816186 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe06c044020 con 0x7fe074107390 2024-03-28T05:23:20.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.430+0000 7fe070ff9640 1 --2- 172.21.15.23:0/549816186 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe0600a2800 0x7fe0600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:20.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.430+0000 7fe072ffd640 1 --2- 172.21.15.23:0/549816186 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe0600a2800 0x7fe0600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:20.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.430+0000 7fe070ff9640 1 -- 172.21.15.23:0/549816186 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fe06c0f0d30 con 0x7fe074107390 2024-03-28T05:23:20.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.431+0000 7fe072ffd640 1 --2- 172.21.15.23:0/549816186 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe0600a2800 0x7fe0600a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe064009bd0 tx=0x7fe064005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:20.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.434+0000 7fe070ff9640 1 -- 172.21.15.23:0/549816186 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe06c04b020 con 0x7fe074107390 2024-03-28T05:23:20.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.616+0000 7fe0796dd640 1 -- 172.21.15.23:0/549816186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe0740622a0 con 0x7fe074107390 2024-03-28T05:23:20.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.616+0000 7fe070ff9640 1 -- 172.21.15.23:0/549816186 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe06c0dbe20 con 0x7fe074107390 2024-03-28T05:23:20.616 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T05:23:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.618+0000 7fe0796dd640 1 -- 172.21.15.23:0/549816186 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe0600a2800 msgr2=0x7fe0600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.618+0000 7fe0796dd640 1 --2- 172.21.15.23:0/549816186 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe0600a2800 0x7fe0600a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe064009bd0 tx=0x7fe064005f90 comp rx=0 tx=0).stop 2024-03-28T05:23:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.618+0000 7fe0796dd640 1 -- 172.21.15.23:0/549816186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe074107390 msgr2=0x7fe07419efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.618+0000 7fe0796dd640 1 --2- 172.21.15.23:0/549816186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe074107390 0x7fe07419efc0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe06c037040 tx=0x7fe06c032f20 comp rx=0 tx=0).stop 2024-03-28T05:23:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.618+0000 7fe0796dd640 1 -- 172.21.15.23:0/549816186 shutdown_connections 2024-03-28T05:23:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.618+0000 7fe0796dd640 1 --2- 172.21.15.23:0/549816186 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7fe0600a2800 0x7fe0600a4cc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.618+0000 7fe0796dd640 1 --2- 172.21.15.23:0/549816186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe074107390 0x7fe07419efc0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.618+0000 7fe0796dd640 1 --2- 172.21.15.23:0/549816186 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe074104a60 0x7fe074197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.618+0000 7fe0796dd640 1 --2- 172.21.15.23:0/549816186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe074102130 0x7fe074197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.618+0000 7fe0796dd640 1 -- 172.21.15.23:0/549816186 >> 172.21.15.23:0/549816186 conn(0x7fe0740fbb40 msgr2=0x7fe074102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:20.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.618+0000 7fe0796dd640 1 -- 172.21.15.23:0/549816186 shutdown_connections 2024-03-28T05:23:20.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.618+0000 7fe0796dd640 1 -- 172.21.15.23:0/549816186 wait complete. 2024-03-28T05:23:20.629 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:23:20.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.732+0000 7f659e545640 1 -- 172.21.15.23:0/123289913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6598102170 msgr2=0x7f6598108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:20.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.732+0000 7f659e545640 1 --2- 172.21.15.23:0/123289913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6598102170 0x7f6598108a00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6590009ca0 tx=0x7f659002c0f0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.732+0000 7f659e545640 1 -- 172.21.15.23:0/123289913 shutdown_connections 2024-03-28T05:23:20.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.732+0000 7f659e545640 1 --2- 172.21.15.23:0/123289913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6598102170 0x7f6598108a00 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.732+0000 7f659e545640 1 --2- 172.21.15.23:0/123289913 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65981017b0 0x7f6598101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.732+0000 7f659e545640 1 --2- 172.21.15.23:0/123289913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65981005c0 0x7f65981009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.732+0000 7f659e545640 1 -- 172.21.15.23:0/123289913 >> 172.21.15.23:0/123289913 conn(0x7f65980fbb30 msgr2=0x7f65980fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:20.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.732+0000 7f659e545640 1 -- 172.21.15.23:0/123289913 shutdown_connections 2024-03-28T05:23:20.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.732+0000 7f659e545640 1 -- 172.21.15.23:0/123289913 wait complete. 2024-03-28T05:23:20.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.733+0000 7f659e545640 1 Processor -- start 2024-03-28T05:23:20.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.733+0000 7f659e545640 1 -- start start 2024-03-28T05:23:20.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.733+0000 7f659e545640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65981005c0 0x7f6598197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:20.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.733+0000 7f659e545640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65981017b0 0x7f6598197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:20.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.733+0000 7f659e545640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6598102170 0x7f659819eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:20.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.733+0000 7f659e545640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659810d360 con 0x7f65981017b0 2024-03-28T05:23:20.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.733+0000 7f659e545640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659810d1e0 con 0x7f6598102170 2024-03-28T05:23:20.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.733+0000 7f659e545640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659810d4e0 con 0x7f65981005c0 2024-03-28T05:23:20.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.733+0000 7f65977fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65981017b0 0x7f6598197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:20.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.733+0000 7f65977fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65981017b0 0x7f6598197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43482/0 (socket says 172.21.15.23:43482) 2024-03-28T05:23:20.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.733+0000 7f65977fe640 1 -- 172.21.15.23:0/3463662266 learned_addr learned my addr 172.21.15.23:0/3463662266 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:20.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.733+0000 7f6597fff640 1 --2- 172.21.15.23:0/3463662266 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65981005c0 0x7f6598197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:20.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.734+0000 7f659cae2640 1 --2- 172.21.15.23:0/3463662266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6598102170 0x7f659819eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:20.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.734+0000 7f65977fe640 1 -- 172.21.15.23:0/3463662266 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65981005c0 msgr2=0x7f6598197940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:20.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.734+0000 7f65977fe640 1 --2- 172.21.15.23:0/3463662266 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65981005c0 0x7f6598197940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.734+0000 7f65977fe640 1 -- 172.21.15.23:0/3463662266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6598102170 msgr2=0x7f659819eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:20.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.734+0000 7f65977fe640 1 --2- 172.21.15.23:0/3463662266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6598102170 0x7f659819eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.734+0000 7f65977fe640 1 -- 172.21.15.23:0/3463662266 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659819f560 con 0x7f65981017b0 2024-03-28T05:23:20.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.734+0000 7f659cae2640 1 --2- 172.21.15.23:0/3463662266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6598102170 0x7f659819eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:23:20.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.734+0000 7f65977fe640 1 --2- 172.21.15.23:0/3463662266 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65981017b0 0x7f6598197e80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f658800ea60 tx=0x7f658800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:20.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.734+0000 7f65957fa640 1 -- 172.21.15.23:0/3463662266 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658800cde0 con 0x7f65981017b0 2024-03-28T05:23:20.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.734+0000 7f65957fa640 1 -- 172.21.15.23:0/3463662266 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f65880035f0 con 0x7f65981017b0 2024-03-28T05:23:20.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.734+0000 7f659e545640 1 -- 172.21.15.23:0/3463662266 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f659819f850 con 0x7f65981017b0 2024-03-28T05:23:20.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.734+0000 7f65957fa640 1 -- 172.21.15.23:0/3463662266 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6588010670 con 0x7f65981017b0 2024-03-28T05:23:20.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.734+0000 7f659e545640 1 -- 172.21.15.23:0/3463662266 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f659819fdf0 con 0x7f65981017b0 2024-03-28T05:23:20.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.735+0000 7f659e545640 1 -- 172.21.15.23:0/3463662266 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6598107880 con 0x7f65981017b0 2024-03-28T05:23:20.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.736+0000 7f65957fa640 1 -- 172.21.15.23:0/3463662266 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f65880037a0 con 0x7f65981017b0 2024-03-28T05:23:20.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.737+0000 7f65957fa640 1 --2- 172.21.15.23:0/3463662266 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f65840a2880 0x7f65840a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:20.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.737+0000 7f6597fff640 1 --2- 172.21.15.23:0/3463662266 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f65840a2880 0x7f65840a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:20.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.737+0000 7f65957fa640 1 -- 172.21.15.23:0/3463662266 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f65880cf130 con 0x7f65981017b0 2024-03-28T05:23:20.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.737+0000 7f6597fff640 1 --2- 172.21.15.23:0/3463662266 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f65840a2880 0x7f65840a4d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6598101610 tx=0x7f658000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:20.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.741+0000 7f65957fa640 1 -- 172.21.15.23:0/3463662266 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f658809be20 con 0x7f65981017b0 2024-03-28T05:23:20.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.968+0000 7f659e545640 1 -- 172.21.15.23:0/3463662266 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f659802d070 con 0x7f65981017b0 2024-03-28T05:23:20.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.969+0000 7f65957fa640 1 -- 172.21.15.23:0/3463662266 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11324 (secure 0 0 0) 0x7f658809b720 con 0x7f65981017b0 2024-03-28T05:23:20.968 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:23:20.968 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/733913400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":733913400},{"type":"v1","addr":"172.21.15.23:6835","nonce":733913400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1972901387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1972901387},{"type":"v1","addr":"172.21.15.23:6837","nonce":1972901387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/2390673788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2390673788},{"type":"v1","addr":"172.21.15.23:6839","nonce":2390673788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/289263156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":289263156},{"type":"v1","addr":"172.21.15.23:6841","nonce":289263156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3313932732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3313932732},{"type":"v1","addr":"172.21.15.98:6835","nonce":3313932732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2695340024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2695340024},{"type":"v1","addr":"172.21.15.113:6835","nonce":2695340024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2750367469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2750367469},{"type":"v1","addr":"172.21.15.98:6837","nonce":2750367469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/3222726274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":3222726274},{"type":"v1","addr":"172.21.15.113:6837","nonce":3222726274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T05:21:07.616553+0000","modified":"2024-03-28T05:21:32.732132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/144113007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":144113007},{"type":"v1","addr":"172.21.15.98:6839","nonce":144113007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.113:6839/212060489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":212060489},{"type":"v1","addr":"172.21.15.113:6839","nonce":212060489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24487,"qdb_cluster":[24487]},"id":1}]} 2024-03-28T05:23:20.969 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 20 2024-03-28T05:23:20.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.971+0000 7f659e545640 1 -- 172.21.15.23:0/3463662266 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f65840a2880 msgr2=0x7f65840a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:20.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.971+0000 7f659e545640 1 --2- 172.21.15.23:0/3463662266 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f65840a2880 0x7f65840a4d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6598101610 tx=0x7f658000b390 comp rx=0 tx=0).stop 2024-03-28T05:23:20.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.971+0000 7f659e545640 1 -- 172.21.15.23:0/3463662266 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65981017b0 msgr2=0x7f6598197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:20.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.971+0000 7f659e545640 1 --2- 172.21.15.23:0/3463662266 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65981017b0 0x7f6598197e80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f658800ea60 tx=0x7f658800ef20 comp rx=0 tx=0).stop 2024-03-28T05:23:20.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.971+0000 7f659e545640 1 -- 172.21.15.23:0/3463662266 shutdown_connections 2024-03-28T05:23:20.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.971+0000 7f659e545640 1 --2- 172.21.15.23:0/3463662266 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f65840a2880 0x7f65840a4d40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.971+0000 7f659e545640 1 --2- 172.21.15.23:0/3463662266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6598102170 0x7f659819eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.971+0000 7f659e545640 1 --2- 172.21.15.23:0/3463662266 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65981017b0 0x7f6598197e80 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.971+0000 7f659e545640 1 --2- 172.21.15.23:0/3463662266 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f65981005c0 0x7f6598197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:20.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.971+0000 7f659e545640 1 -- 172.21.15.23:0/3463662266 >> 172.21.15.23:0/3463662266 conn(0x7f65980fbb30 msgr2=0x7f6598104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:20.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.971+0000 7f659e545640 1 -- 172.21.15.23:0/3463662266 shutdown_connections 2024-03-28T05:23:20.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:20.971+0000 7f659e545640 1 -- 172.21.15.23:0/3463662266 wait complete. 2024-03-28T05:23:20.980 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:20 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2811488290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:23:20.981 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:20 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3362725596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:23:20.981 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:20 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/549816186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:23:20.982 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T05:23:21.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.086+0000 7f058e5c4640 1 -- 172.21.15.23:0/709844498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05881028a0 msgr2=0x7f0588102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:21.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.086+0000 7f058e5c4640 1 --2- 172.21.15.23:0/709844498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05881028a0 0x7f0588102d20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0578009900 tx=0x7f057802bc30 comp rx=0 tx=0).stop 2024-03-28T05:23:21.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.086+0000 7f058e5c4640 1 -- 172.21.15.23:0/709844498 shutdown_connections 2024-03-28T05:23:21.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.086+0000 7f058e5c4640 1 --2- 172.21.15.23:0/709844498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0588103260 0x7f0588109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.086+0000 7f058e5c4640 1 --2- 172.21.15.23:0/709844498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05881028a0 0x7f0588102d20 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.086+0000 7f058e5c4640 1 --2- 172.21.15.23:0/709844498 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05881016b0 0x7f0588101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.086+0000 7f058e5c4640 1 -- 172.21.15.23:0/709844498 >> 172.21.15.23:0/709844498 conn(0x7f05880fcc80 msgr2=0x7f05880ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:21.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.086+0000 7f058e5c4640 1 -- 172.21.15.23:0/709844498 shutdown_connections 2024-03-28T05:23:21.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.086+0000 7f058e5c4640 1 -- 172.21.15.23:0/709844498 wait complete. 2024-03-28T05:23:21.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.087+0000 7f058e5c4640 1 Processor -- start 2024-03-28T05:23:21.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.087+0000 7f058e5c4640 1 -- start start 2024-03-28T05:23:21.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.087+0000 7f058e5c4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05881016b0 0x7f05881979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:21.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.087+0000 7f058e5c4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05881028a0 0x7f0588197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:21.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.087+0000 7f058e5c4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0588103260 0x7f058819ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:21.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.087+0000 7f058e5c4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058810d330 con 0x7f0588103260 2024-03-28T05:23:21.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.087+0000 7f058e5c4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058810d1b0 con 0x7f05881028a0 2024-03-28T05:23:21.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.087+0000 7f058e5c4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058810d4b0 con 0x7f05881016b0 2024-03-28T05:23:21.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.087+0000 7f058cb61640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0588103260 0x7f058819ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:21.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.088+0000 7f0587fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05881016b0 0x7f05881979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:21.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.088+0000 7f058cb61640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0588103260 0x7f058819ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43506/0 (socket says 172.21.15.23:43506) 2024-03-28T05:23:21.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.088+0000 7f058cb61640 1 -- 172.21.15.23:0/2284081646 learned_addr learned my addr 172.21.15.23:0/2284081646 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:21.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.088+0000 7f05877fe640 1 --2- 172.21.15.23:0/2284081646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05881028a0 0x7f0588197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:21.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.088+0000 7f058cb61640 1 -- 172.21.15.23:0/2284081646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05881016b0 msgr2=0x7f05881979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.088+0000 7f058cb61640 1 --2- 172.21.15.23:0/2284081646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05881016b0 0x7f05881979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.088+0000 7f058cb61640 1 -- 172.21.15.23:0/2284081646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05881028a0 msgr2=0x7f0588197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.088+0000 7f058cb61640 1 --2- 172.21.15.23:0/2284081646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05881028a0 0x7f0588197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.088+0000 7f058cb61640 1 -- 172.21.15.23:0/2284081646 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058819f5f0 con 0x7f0588103260 2024-03-28T05:23:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.088+0000 7f0587fff640 1 --2- 172.21.15.23:0/2284081646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05881016b0 0x7f05881979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:23:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.088+0000 7f058cb61640 1 --2- 172.21.15.23:0/2284081646 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0588103260 0x7f058819ef80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f058000eb10 tx=0x7f058000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.088+0000 7f05857fa640 1 -- 172.21.15.23:0/2284081646 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0580004b20 con 0x7f0588103260 2024-03-28T05:23:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.088+0000 7f05857fa640 1 -- 172.21.15.23:0/2284081646 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0580004cc0 con 0x7f0588103260 2024-03-28T05:23:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.088+0000 7f058e5c4640 1 -- 172.21.15.23:0/2284081646 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f058819f8e0 con 0x7f0588103260 2024-03-28T05:23:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.088+0000 7f05857fa640 1 -- 172.21.15.23:0/2284081646 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0580010760 con 0x7f0588103260 2024-03-28T05:23:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.088+0000 7f058e5c4640 1 -- 172.21.15.23:0/2284081646 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f058819fdb0 con 0x7f0588103260 2024-03-28T05:23:21.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.089+0000 7f058e5c4640 1 -- 172.21.15.23:0/2284081646 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05881089d0 con 0x7f0588103260 2024-03-28T05:23:21.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.090+0000 7f05857fa640 1 -- 172.21.15.23:0/2284081646 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0580005020 con 0x7f0588103260 2024-03-28T05:23:21.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.091+0000 7f05857fa640 1 --2- 172.21.15.23:0/2284081646 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f05740a2800 0x7f05740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:21.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.091+0000 7f0587fff640 1 --2- 172.21.15.23:0/2284081646 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f05740a2800 0x7f05740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:21.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.091+0000 7f05857fa640 1 -- 172.21.15.23:0/2284081646 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f05800cf390 con 0x7f0588103260 2024-03-28T05:23:21.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.091+0000 7f0587fff640 1 --2- 172.21.15.23:0/2284081646 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f05740a2800 0x7f05740a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0588102700 tx=0x7f0570005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:21.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.095+0000 7f05857fa640 1 -- 172.21.15.23:0/2284081646 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f058009c120 con 0x7f0588103260 2024-03-28T05:23:21.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:20 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2811488290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:23:21.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:20 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3362725596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:23:21.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:20 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/549816186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:23:21.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:20 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2811488290' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:23:21.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:20 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3362725596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:23:21.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:20 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/549816186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:23:21.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.279+0000 7f058e5c4640 1 -- 172.21.15.23:0/2284081646 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f058802d290 con 0x7f0588103260 2024-03-28T05:23:21.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.281+0000 7f05857fa640 1 -- 172.21.15.23:0/2284081646 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24695 (secure 0 0 0) 0x7f058009ba20 con 0x7f0588103260 2024-03-28T05:23:21.280 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:23:21.281 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":93,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","created":"2024-03-28T05:13:13.453653+0000","modified":"2024-03-28T05:23:10.695543+0000","last_up_change":"2024-03-28T05:20:20.221527+0000","last_in_change":"2024-03-28T05:20:00.365478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T05:16:32.986974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T05:21:06.103894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T05:21:06.827190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T05:21:10.507418+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"18150942-59a4-4c3b-83d4-c1fc3ccf1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6803","nonce":1219956881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6805","nonce":1219956881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6809","nonce":1219956881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6807","nonce":1219956881}]},"public_addr":"172.21.15.23:6803/1219956881","cluster_addr":"172.21.15.23:6805/1219956881","heartbeat_back_addr":"172.21.15.23:6809/1219956881","heartbeat_front_addr":"172.21.15.23:6807/1219956881","state":["exists","up"]},{"osd":1,"uuid":"ba6d1a95-5135-4670-95f9-7878c0e7bd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6801","nonce":555491234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6803","nonce":555491234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6807","nonce":555491234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6805","nonce":555491234}]},"public_addr":"172.21.15.98:6801/555491234","cluster_addr":"172.21.15.98:6803/555491234","heartbeat_back_addr":"172.21.15.98:6807/555491234","heartbeat_front_addr":"172.21.15.98:6805/555491234","state":["exists","up"]},{"osd":2,"uuid":"873bdd60-2a47-403b-b1f4-ee9226a95067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6801","nonce":1535856914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6803","nonce":1535856914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6807","nonce":1535856914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6805","nonce":1535856914}]},"public_addr":"172.21.15.113:6801/1535856914","cluster_addr":"172.21.15.113:6803/1535856914","heartbeat_back_addr":"172.21.15.113:6807/1535856914","heartbeat_front_addr":"172.21.15.113:6805/1535856914","state":["exists","up"]},{"osd":3,"uuid":"83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6811","nonce":2949415020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6813","nonce":2949415020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6817","nonce":2949415020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6815","nonce":2949415020}]},"public_addr":"172.21.15.23:6811/2949415020","cluster_addr":"172.21.15.23:6813/2949415020","heartbeat_back_addr":"172.21.15.23:6817/2949415020","heartbeat_front_addr":"172.21.15.23:6815/2949415020","state":["exists","up"]},{"osd":4,"uuid":"76a90bac-b221-40a3-8238-40e8049cec36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6809","nonce":2615902819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6811","nonce":2615902819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6815","nonce":2615902819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6813","nonce":2615902819}]},"public_addr":"172.21.15.98:6809/2615902819","cluster_addr":"172.21.15.98:6811/2615902819","heartbeat_back_addr":"172.21.15.98:6815/2615902819","heartbeat_front_addr":"172.21.15.98:6813/2615902819","state":["exists","up"]},{"osd":5,"uuid":"b53fdb55-a8df-40a8-bd1b-8bcef634e26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6809","nonce":625271528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6811","nonce":625271528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6815","nonce":625271528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6813","nonce":625271528}]},"public_addr":"172.21.15.113:6809/625271528","cluster_addr":"172.21.15.113:6811/625271528","heartbeat_back_addr":"172.21.15.113:6815/625271528","heartbeat_front_addr":"172.21.15.113:6813/625271528","state":["exists","up"]},{"osd":6,"uuid":"f00283d2-8d67-4e97-a575-9a9d3b55d5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6819","nonce":4188464176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6821","nonce":4188464176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6825","nonce":4188464176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6823","nonce":4188464176}]},"public_addr":"172.21.15.23:6819/4188464176","cluster_addr":"172.21.15.23:6821/4188464176","heartbeat_back_addr":"172.21.15.23:6825/4188464176","heartbeat_front_addr":"172.21.15.23:6823/4188464176","state":["exists","up"]},{"osd":7,"uuid":"edc008a7-b07b-40a8-88b7-1614ba38a775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6817","nonce":1985546028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6819","nonce":1985546028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6823","nonce":1985546028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6821","nonce":1985546028}]},"public_addr":"172.21.15.98:6817/1985546028","cluster_addr":"172.21.15.98:6819/1985546028","heartbeat_back_addr":"172.21.15.98:6823/1985546028","heartbeat_front_addr":"172.21.15.98:6821/1985546028","state":["exists","up"]},{"osd":8,"uuid":"56e7b2d1-81f0-47b2-8638-f29832bed8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6817","nonce":3534876081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6819","nonce":3534876081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6823","nonce":3534876081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6821","nonce":3534876081}]},"public_addr":"172.21.15.113:6817/3534876081","cluster_addr":"172.21.15.113:6819/3534876081","heartbeat_back_addr":"172.21.15.113:6823/3534876081","heartbeat_front_addr":"172.21.15.113:6821/3534876081","state":["exists","up"]},{"osd":9,"uuid":"a2e90caa-6e9c-42bf-9f18-81f7f47880cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6827","nonce":672264207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6829","nonce":672264207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6833","nonce":672264207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6831","nonce":672264207}]},"public_addr":"172.21.15.23:6827/672264207","cluster_addr":"172.21.15.23:6829/672264207","heartbeat_back_addr":"172.21.15.23:6833/672264207","heartbeat_front_addr":"172.21.15.23:6831/672264207","state":["exists","up"]},{"osd":10,"uuid":"0eda917f-020c-437f-95ae-b26203f6c68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6825","nonce":2699920595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6827","nonce":2699920595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6831","nonce":2699920595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6829","nonce":2699920595}]},"public_addr":"172.21.15.98:6825/2699920595","cluster_addr":"172.21.15.98:6827/2699920595","heartbeat_back_addr":"172.21.15.98:6831/2699920595","heartbeat_front_addr":"172.21.15.98:6829/2699920595","state":["exists","up"]},{"osd":11,"uuid":"aae78501-78a8-469a-be20-3006f5ec5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6825","nonce":2113644020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6827","nonce":2113644020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6831","nonce":2113644020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6829","nonce":2113644020}]},"public_addr":"172.21.15.113:6825/2113644020","cluster_addr":"172.21.15.113:6827/2113644020","heartbeat_back_addr":"172.21.15.113:6831/2113644020","heartbeat_front_addr":"172.21.15.113:6829/2113644020","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:15:43.527475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:07.006538+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:30.594176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:55.810999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:20.595811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:45.474138+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:10.500409+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:35.787638+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:01.001188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:26.761115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:52.952752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:20:18.559832+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":7}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:6800/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1876451381":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2253559862":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2653562762":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/3337463830":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6800/1735221490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/2784771635":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2389902490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6801/1735221490":"2024-03-29T05:13:42.579607+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T05:23:21.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.283+0000 7f058e5c4640 1 -- 172.21.15.23:0/2284081646 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f05740a2800 msgr2=0x7f05740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:21.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.283+0000 7f058e5c4640 1 --2- 172.21.15.23:0/2284081646 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f05740a2800 0x7f05740a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0588102700 tx=0x7f0570005f90 comp rx=0 tx=0).stop 2024-03-28T05:23:21.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.283+0000 7f058e5c4640 1 -- 172.21.15.23:0/2284081646 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0588103260 msgr2=0x7f058819ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:21.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.283+0000 7f058e5c4640 1 --2- 172.21.15.23:0/2284081646 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0588103260 0x7f058819ef80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f058000eb10 tx=0x7f058000c2d0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.283+0000 7f058e5c4640 1 -- 172.21.15.23:0/2284081646 shutdown_connections 2024-03-28T05:23:21.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.283+0000 7f058e5c4640 1 --2- 172.21.15.23:0/2284081646 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f05740a2800 0x7f05740a4cc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.283+0000 7f058e5c4640 1 --2- 172.21.15.23:0/2284081646 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0588103260 0x7f058819ef80 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.283+0000 7f058e5c4640 1 --2- 172.21.15.23:0/2284081646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05881028a0 0x7f0588197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.283+0000 7f058e5c4640 1 --2- 172.21.15.23:0/2284081646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05881016b0 0x7f05881979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.283+0000 7f058e5c4640 1 -- 172.21.15.23:0/2284081646 >> 172.21.15.23:0/2284081646 conn(0x7f05880fcc80 msgr2=0x7f0588105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:21.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.283+0000 7f058e5c4640 1 -- 172.21.15.23:0/2284081646 shutdown_connections 2024-03-28T05:23:21.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.283+0000 7f058e5c4640 1 -- 172.21.15.23:0/2284081646 wait complete. 2024-03-28T05:23:21.294 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T05:21:07.616553+0000', 'modified': '2024-03-28T05:21:32.732132+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24487}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24473': {'gid': 24473, 'name': 'b', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6839/144113007', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6838', 'nonce': 144113007}, {'type': 'v1', 'addr': '172.21.15.98:6839', 'nonce': 144113007}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24487': {'gid': 24487, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.113:6839/212060489', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6838', 'nonce': 212060489}, {'type': 'v1', 'addr': '172.21.15.113:6839', 'nonce': 212060489}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24487, 'qdb_cluster': [24487]} 2024-03-28T05:23:21.294 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T05:23:21.294 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-28T05:23:21.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.398+0000 7f48c5329640 1 -- 172.21.15.23:0/3111346656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48c0107400 msgr2=0x7f48c01097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:21.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.398+0000 7f48c5329640 1 --2- 172.21.15.23:0/3111346656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48c0107400 0x7f48c01097f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f48b0009ca0 tx=0x7f48b002c0f0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.398+0000 7f48c5329640 1 -- 172.21.15.23:0/3111346656 shutdown_connections 2024-03-28T05:23:21.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.398+0000 7f48c5329640 1 --2- 172.21.15.23:0/3111346656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48c0107400 0x7f48c01097f0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.398+0000 7f48c5329640 1 --2- 172.21.15.23:0/3111346656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48c0104ad0 0x7f48c0106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.398+0000 7f48c5329640 1 --2- 172.21.15.23:0/3111346656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48c01021a0 0x7f48c0104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.398+0000 7f48c5329640 1 -- 172.21.15.23:0/3111346656 >> 172.21.15.23:0/3111346656 conn(0x7f48c00fbb30 msgr2=0x7f48c00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:21.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.398+0000 7f48c5329640 1 -- 172.21.15.23:0/3111346656 shutdown_connections 2024-03-28T05:23:21.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.398+0000 7f48c5329640 1 -- 172.21.15.23:0/3111346656 wait complete. 2024-03-28T05:23:21.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.399+0000 7f48c5329640 1 Processor -- start 2024-03-28T05:23:21.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.399+0000 7f48c5329640 1 -- start start 2024-03-28T05:23:21.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.399+0000 7f48c5329640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48c01021a0 0x7f48c01979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:21.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.399+0000 7f48c5329640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48c0104ad0 0x7f48c0197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:21.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.399+0000 7f48c5329640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48c0107400 0x7f48c019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:21.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.399+0000 7f48c5329640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c010d320 con 0x7f48c0107400 2024-03-28T05:23:21.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.399+0000 7f48c5329640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c010d1a0 con 0x7f48c0104ad0 2024-03-28T05:23:21.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.399+0000 7f48c5329640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c010d4a0 con 0x7f48c01021a0 2024-03-28T05:23:21.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.399+0000 7f48bf7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48c0107400 0x7f48c019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:21.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.399+0000 7f48bf7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48c0107400 0x7f48c019ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43534/0 (socket says 172.21.15.23:43534) 2024-03-28T05:23:21.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.399+0000 7f48bf7fe640 1 -- 172.21.15.23:0/286129388 learned_addr learned my addr 172.21.15.23:0/286129388 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:21.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.399+0000 7f48beffd640 1 --2- 172.21.15.23:0/286129388 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48c01021a0 0x7f48c01979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:21.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.399+0000 7f48be7fc640 1 --2- 172.21.15.23:0/286129388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48c0104ad0 0x7f48c0197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:21.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.400+0000 7f48bf7fe640 1 -- 172.21.15.23:0/286129388 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48c01021a0 msgr2=0x7f48c01979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:21.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.400+0000 7f48bf7fe640 1 --2- 172.21.15.23:0/286129388 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48c01021a0 0x7f48c01979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.400+0000 7f48bf7fe640 1 -- 172.21.15.23:0/286129388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48c0104ad0 msgr2=0x7f48c0197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:21.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.400+0000 7f48bf7fe640 1 --2- 172.21.15.23:0/286129388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48c0104ad0 0x7f48c0197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.400+0000 7f48bf7fe640 1 -- 172.21.15.23:0/286129388 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c019f5f0 con 0x7f48c0107400 2024-03-28T05:23:21.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.400+0000 7f48be7fc640 1 --2- 172.21.15.23:0/286129388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48c0104ad0 0x7f48c0197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:23:21.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.400+0000 7f48bf7fe640 1 --2- 172.21.15.23:0/286129388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48c0107400 0x7f48c019ef80 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f48b0037040 tx=0x7f48b0032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:21.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.400+0000 7f48a7fff640 1 -- 172.21.15.23:0/286129388 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b0041020 con 0x7f48c0107400 2024-03-28T05:23:21.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.400+0000 7f48c5329640 1 -- 172.21.15.23:0/286129388 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c019f880 con 0x7f48c0107400 2024-03-28T05:23:21.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.400+0000 7f48a7fff640 1 -- 172.21.15.23:0/286129388 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48b0034d90 con 0x7f48c0107400 2024-03-28T05:23:21.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.400+0000 7f48a7fff640 1 -- 172.21.15.23:0/286129388 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b0039040 con 0x7f48c0107400 2024-03-28T05:23:21.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.400+0000 7f48c5329640 1 -- 172.21.15.23:0/286129388 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c019fd70 con 0x7f48c0107400 2024-03-28T05:23:21.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.403+0000 7f48a7fff640 1 -- 172.21.15.23:0/286129388 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f48b0044020 con 0x7f48c0107400 2024-03-28T05:23:21.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.403+0000 7f48c7ff2740 1 Processor -- start 2024-03-28T05:23:21.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.403+0000 7f48c7ff2740 1 -- start start 2024-03-28T05:23:21.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.403+0000 7f48c7ff2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09b740 0x55d19c09bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:21.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.403+0000 7f48c7ff2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d19c09c130 0x55d19c0a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:21.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.403+0000 7f48c7ff2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d19c0a12a0 0x55d19c0a3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:21.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.403+0000 7f48c7ff2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d19c000c20 con 0x55d19c09b740 2024-03-28T05:23:21.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.403+0000 7f48c7ff2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d19bfa9a40 con 0x55d19c09c130 2024-03-28T05:23:21.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.403+0000 7f48c7ff2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d19bffccc0 con 0x55d19c0a12a0 2024-03-28T05:23:21.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.403+0000 7f48be7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09b740 0x55d19c09bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:21.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.403+0000 7f48be7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09b740 0x55d19c09bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43540/0 (socket says 172.21.15.23:43540) 2024-03-28T05:23:21.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.403+0000 7f48be7fc640 1 -- 172.21.15.23:0/2857399821 learned_addr learned my addr 172.21.15.23:0/2857399821 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:21.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.403+0000 7f48be7fc640 1 --2- 172.21.15.23:0/2857399821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d19c0a12a0 0x55d19c0a3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:21.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.403+0000 7f48a7fff640 1 --2- 172.21.15.23:0/286129388 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f48a80a2800 0x7f48a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:21.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.403+0000 7f48beffd640 1 --2- 172.21.15.23:0/2857399821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d19c09c130 0x55d19c0a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:21.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.403+0000 7f48be7fc640 1 -- 172.21.15.23:0/2857399821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d19c0a12a0 msgr2=0x55d19c0a3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:23:21.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.403+0000 7f48be7fc640 1 --2- 172.21.15.23:0/2857399821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d19c0a12a0 0x55d19c0a3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.403+0000 7f48be7fc640 1 -- 172.21.15.23:0/2857399821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d19c09c130 msgr2=0x55d19c0a0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:23:21.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.403+0000 7f48be7fc640 1 --2- 172.21.15.23:0/2857399821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d19c09c130 0x55d19c0a0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.403+0000 7f48be7fc640 1 -- 172.21.15.23:0/2857399821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d19bfcbf90 con 0x55d19c09b740 2024-03-28T05:23:21.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.403+0000 7f48beffd640 1 --2- 172.21.15.23:0/2857399821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d19c09c130 0x55d19c0a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:23:21.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.404+0000 7f48bf7fe640 1 --2- 172.21.15.23:0/286129388 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f48a80a2800 0x7f48a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:21.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.404+0000 7f48a7fff640 1 -- 172.21.15.23:0/286129388 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f48b00f0d80 con 0x7f48c0107400 2024-03-28T05:23:21.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.404+0000 7f48be7fc640 1 --2- 172.21.15.23:0/2857399821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09b740 0x55d19c09bb60 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f48b4014490 tx=0x7f48b40096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f6a03f354ad2df3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:21.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.404+0000 7f48a57fa640 1 -- 172.21.15.23:0/2857399821 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b401a030 con 0x55d19c09b740 2024-03-28T05:23:21.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.404+0000 7f48a57fa640 1 -- 172.21.15.23:0/2857399821 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48b4003610 con 0x55d19c09b740 2024-03-28T05:23:21.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.404+0000 7f48a57fa640 1 -- 172.21.15.23:0/2857399821 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b4012670 con 0x55d19c09b740 2024-03-28T05:23:21.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.404+0000 7f48c7ff2740 1 -- 172.21.15.23:0/2857399821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09b740 msgr2=0x55d19c09bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:21.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.404+0000 7f48bf7fe640 1 --2- 172.21.15.23:0/286129388 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f48a80a2800 0x7f48a80a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f48b00b97d0 tx=0x7f48b00dd040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:21.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.404+0000 7f48c7ff2740 1 --2- 172.21.15.23:0/2857399821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09b740 0x55d19c09bb60 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f48b4014490 tx=0x7f48b40096a0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.405+0000 7f48c7ff2740 1 -- 172.21.15.23:0/2857399821 shutdown_connections 2024-03-28T05:23:21.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.405+0000 7f48c7ff2740 1 --2- 172.21.15.23:0/2857399821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d19c0a12a0 0x55d19c0a3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.405+0000 7f48c7ff2740 1 --2- 172.21.15.23:0/2857399821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d19c09c130 0x55d19c0a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.405+0000 7f48c7ff2740 1 --2- 172.21.15.23:0/2857399821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09b740 0x55d19c09bb60 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.405+0000 7f48c7ff2740 1 -- 172.21.15.23:0/2857399821 >> 172.21.15.23:0/2857399821 conn(0x55d19c063a60 msgr2=0x55d19c0970b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:21.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.405+0000 7f48c7ff2740 1 -- 172.21.15.23:0/2857399821 shutdown_connections 2024-03-28T05:23:21.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.405+0000 7f48c7ff2740 1 -- 172.21.15.23:0/2857399821 wait complete. 2024-03-28T05:23:21.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.405+0000 7f48c7ff2740 1 -- start start 2024-03-28T05:23:21.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.405+0000 7f48c7ff2740 1 Processor -- start 2024-03-28T05:23:21.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.405+0000 7f48c7ff2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09b740 0x55d19c0a5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:21.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.405+0000 7f48c7ff2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d19c09c130 0x55d19c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:21.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.405+0000 7f48c7ff2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d19c0a12a0 0x55d19c0ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:21.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.405+0000 7f48c7ff2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d19c0a4250 con 0x55d19c09b740 2024-03-28T05:23:21.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.405+0000 7f48c7ff2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d19bfbce90 con 0x55d19c0a12a0 2024-03-28T05:23:21.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.405+0000 7f48c7ff2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d19c0a43d0 con 0x55d19c09c130 2024-03-28T05:23:21.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.405+0000 7f48c7ff2740 10 client.0 _ll_register_callbacks cb 0x55d19bd14ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:23:21.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.405+0000 7f48be7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09b740 0x55d19c0a5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:21.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.405+0000 7f48be7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09b740 0x55d19c0a5c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43556/0 (socket says 172.21.15.23:43556) 2024-03-28T05:23:21.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.405+0000 7f48be7fc640 1 -- 172.21.15.23:0/113076697 learned_addr learned my addr 172.21.15.23:0/113076697 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:21.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.405+0000 7f48beffd640 1 --2- 172.21.15.23:0/113076697 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d19c09c130 0x55d19c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:21.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.405+0000 7f48be7fc640 1 --2- 172.21.15.23:0/113076697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d19c0a12a0 0x55d19c0ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:21.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.406+0000 7f48be7fc640 1 -- 172.21.15.23:0/113076697 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d19c09c130 msgr2=0x55d19c0a6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:23:21.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.406+0000 7f48be7fc640 1 --2- 172.21.15.23:0/113076697 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d19c09c130 0x55d19c0a6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.406+0000 7f48be7fc640 1 -- 172.21.15.23:0/113076697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d19c0a12a0 msgr2=0x55d19c0ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:23:21.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.406+0000 7f48be7fc640 1 --2- 172.21.15.23:0/113076697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d19c0a12a0 0x55d19c0ad1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.406+0000 7f48be7fc640 1 -- 172.21.15.23:0/113076697 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d19bfc99d0 con 0x55d19c09b740 2024-03-28T05:23:21.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.406+0000 7f48be7fc640 1 --2- 172.21.15.23:0/113076697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09b740 0x55d19c0a5c50 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f48b40098f0 tx=0x7f48b4009420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0e52941a65dd832 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:21.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.406+0000 7f48a2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:23:21.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.406+0000 7f48a2ff5640 1 -- 172.21.15.23:0/113076697 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b401a050 con 0x55d19c09b740 2024-03-28T05:23:21.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.406+0000 7f48a2ff5640 1 -- 172.21.15.23:0/113076697 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f48b401f660 con 0x55d19c09b740 2024-03-28T05:23:21.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.406+0000 7f48c7ff2740 1 -- 172.21.15.23:0/113076697 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f48b4003610 con 0x55d19c09b740 2024-03-28T05:23:21.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.406+0000 7f48a2ff5640 1 -- 172.21.15.23:0/113076697 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48b4035050 con 0x55d19c09b740 2024-03-28T05:23:21.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.406+0000 7f48c7ff2740 10 client.15366 fetch_fsmap learned FSMap version 20 2024-03-28T05:23:21.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.406+0000 7f48c7ff2740 1 -- 172.21.15.23:0/113076697 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55d19bd152d0 con 0x55d19c09b740 2024-03-28T05:23:21.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.406+0000 7f48a2ff5640 1 -- 172.21.15.23:0/113076697 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48b4003610 con 0x55d19c09b740 2024-03-28T05:23:21.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.406+0000 7f48a2ff5640 1 -- 172.21.15.23:0/113076697 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f48b400f040 con 0x55d19c09b740 2024-03-28T05:23:21.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.406+0000 7f48c7ff2740 10 client.15366 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:23:21.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.406+0000 7f48c7ff2740 10 client.15366 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T05:23:21.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.406+0000 7f48c7ff2740 20 client.15366 populate_metadata read hostname 'smithi023' 2024-03-28T05:23:21.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.406+0000 7f48c7ff2740 1 --2- 172.21.15.23:0/113076697 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d19c0b20b0 0x55d19c0b1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:21.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.407+0000 7f48c7ff2740 4 client.15366 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:23:21.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.407+0000 7f48c7ff2740 1 -- 172.21.15.23:0/113076697 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d19c00e160 con 0x55d19c0b20b0 2024-03-28T05:23:21.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.407+0000 7f48beffd640 1 --2- 172.21.15.23:0/113076697 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d19c0b20b0 0x55d19c0b1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:21.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.407+0000 7f48a2ff5640 10 client.15366 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:23:21.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.407+0000 7f48beffd640 1 --2- 172.21.15.23:0/113076697 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d19c0b20b0 0x55d19c0b1360 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d18b66b9e7e04732 server_cookie=e8e71eb9978b8f28 in_seq=0 out_seq=0 2024-03-28T05:23:21.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.408+0000 7f48a2ff5640 1 -- 172.21.15.23:0/113076697 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d19c00e160 con 0x55d19c0b20b0 2024-03-28T05:23:21.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.408+0000 7f48a2ff5640 10 client.15366 handle_command_reply: tid=0 2024-03-28T05:23:21.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.408+0000 7f48c7ff2740 1 client.15366 shutdown 2024-03-28T05:23:21.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.408+0000 7f48c7ff2740 1 -- 172.21.15.23:0/113076697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09b740 msgr2=0x55d19c0a5c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:21.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.408+0000 7f48c7ff2740 1 --2- 172.21.15.23:0/113076697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09b740 0x55d19c0a5c50 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f48b40098f0 tx=0x7f48b4009420 comp rx=0 tx=0).stop 2024-03-28T05:23:21.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.408+0000 7f48c7ff2740 1 -- 172.21.15.23:0/113076697 shutdown_connections 2024-03-28T05:23:21.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.408+0000 7f48c7ff2740 1 --2- 172.21.15.23:0/113076697 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d19c0b20b0 0x55d19c0b1360 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.408+0000 7f48c7ff2740 1 --2- 172.21.15.23:0/113076697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d19c0a12a0 0x55d19c0ad1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.408+0000 7f48c7ff2740 1 --2- 172.21.15.23:0/113076697 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d19c09c130 0x55d19c0a6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.408+0000 7f48c7ff2740 1 --2- 172.21.15.23:0/113076697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09b740 0x55d19c0a5c50 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.408+0000 7f48a2ff5640 0 client.15366 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:23:21.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.408+0000 7f48c7ff2740 1 -- 172.21.15.23:0/113076697 >> 172.21.15.23:0/113076697 conn(0x55d19c063a60 msgr2=0x55d19c0962f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:21.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.408+0000 7f48c7ff2740 1 -- 172.21.15.23:0/113076697 shutdown_connections 2024-03-28T05:23:21.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.408+0000 7f48c7ff2740 1 -- 172.21.15.23:0/113076697 wait complete. 2024-03-28T05:23:21.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.408+0000 7f48c7ff2740 20 client.15366 trim_cache size 0 max 16384 2024-03-28T05:23:21.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.425+0000 7f48c7ff2740 1 Processor -- start 2024-03-28T05:23:21.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.425+0000 7f48c7ff2740 1 -- start start 2024-03-28T05:23:21.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.425+0000 7f48c7ff2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d19c09b740 0x55d19c0a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:21.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.425+0000 7f48c7ff2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09c130 0x55d19c0ad160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:21.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.425+0000 7f48c7ff2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d19c0a12a0 0x55d19c09a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:21.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.425+0000 7f48c7ff2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d19c0b18d0 con 0x55d19c09c130 2024-03-28T05:23:21.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.425+0000 7f48c7ff2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d19c0b1f30 con 0x55d19c0a12a0 2024-03-28T05:23:21.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.425+0000 7f48c7ff2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d19c0b2590 con 0x55d19c09b740 2024-03-28T05:23:21.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.425+0000 7f48beffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09c130 0x55d19c0ad160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:21.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.426+0000 7f48beffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09c130 0x55d19c0ad160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43568/0 (socket says 172.21.15.23:43568) 2024-03-28T05:23:21.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.426+0000 7f48beffd640 1 -- 172.21.15.23:0/1150628641 learned_addr learned my addr 172.21.15.23:0/1150628641 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:21.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.426+0000 7f48be7fc640 1 --2- 172.21.15.23:0/1150628641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d19c09b740 0x55d19c0a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:21.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.426+0000 7f48beffd640 1 -- 172.21.15.23:0/1150628641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d19c09b740 msgr2=0x55d19c0a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:23:21.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.426+0000 7f48beffd640 1 --2- 172.21.15.23:0/1150628641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d19c09b740 0x55d19c0a4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.426+0000 7f48beffd640 1 -- 172.21.15.23:0/1150628641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d19c0a12a0 msgr2=0x55d19c09a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:23:21.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.426+0000 7f48beffd640 1 --2- 172.21.15.23:0/1150628641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d19c0a12a0 0x55d19c09a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.426+0000 7f48beffd640 1 -- 172.21.15.23:0/1150628641 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d19bddf6b0 con 0x55d19c09c130 2024-03-28T05:23:21.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.426+0000 7f48beffd640 1 --2- 172.21.15.23:0/1150628641 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09c130 0x55d19c0ad160 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f48b800b7a0 tx=0x7f48b801eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4c2a25eb95c0102 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:21.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.426+0000 7f48a4ff9640 1 -- 172.21.15.23:0/1150628641 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b801e050 con 0x55d19c09c130 2024-03-28T05:23:21.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.426+0000 7f48a4ff9640 1 -- 172.21.15.23:0/1150628641 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48b801e1f0 con 0x55d19c09c130 2024-03-28T05:23:21.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.426+0000 7f48a4ff9640 1 -- 172.21.15.23:0/1150628641 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b8012910 con 0x55d19c09c130 2024-03-28T05:23:21.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48c7ff2740 1 -- 172.21.15.23:0/1150628641 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09c130 msgr2=0x55d19c0ad160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:21.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48c7ff2740 1 --2- 172.21.15.23:0/1150628641 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09c130 0x55d19c0ad160 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f48b800b7a0 tx=0x7f48b801eac0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48c7ff2740 1 -- 172.21.15.23:0/1150628641 shutdown_connections 2024-03-28T05:23:21.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48c7ff2740 1 --2- 172.21.15.23:0/1150628641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d19c0a12a0 0x55d19c09a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48c7ff2740 1 --2- 172.21.15.23:0/1150628641 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09c130 0x55d19c0ad160 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48c7ff2740 1 --2- 172.21.15.23:0/1150628641 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d19c09b740 0x55d19c0a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48c7ff2740 1 -- 172.21.15.23:0/1150628641 >> 172.21.15.23:0/1150628641 conn(0x55d19c063a60 msgr2=0x55d19c0a4770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:21.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48c7ff2740 1 -- 172.21.15.23:0/1150628641 shutdown_connections 2024-03-28T05:23:21.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48c7ff2740 1 -- 172.21.15.23:0/1150628641 wait complete. 2024-03-28T05:23:21.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48c7ff2740 1 -- start start 2024-03-28T05:23:21.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48c7ff2740 1 Processor -- start 2024-03-28T05:23:21.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48c7ff2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d19c09b740 0x55d19c10b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:21.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48c7ff2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09c130 0x55d19c10d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:21.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48c7ff2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d19c0a12a0 0x55d19c10ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:21.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48c7ff2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d19c0b3250 con 0x55d19c09c130 2024-03-28T05:23:21.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48c7ff2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d19c0b2bf0 con 0x55d19c0a12a0 2024-03-28T05:23:21.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48c7ff2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d19c0b38b0 con 0x55d19c09b740 2024-03-28T05:23:21.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48c7ff2740 10 client.0 _ll_register_callbacks cb 0x7f48b4009f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:23:21.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48beffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09c130 0x55d19c10d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:21.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48beffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09c130 0x55d19c10d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43570/0 (socket says 172.21.15.23:43570) 2024-03-28T05:23:21.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48beffd640 1 -- 172.21.15.23:0/545531501 learned_addr learned my addr 172.21.15.23:0/545531501 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:21.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48be7fc640 1 --2- 172.21.15.23:0/545531501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d19c09b740 0x55d19c10b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:21.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.427+0000 7f48be7fc640 1 --2- 172.21.15.23:0/545531501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d19c0a12a0 0x55d19c10ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:21.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.428+0000 7f48beffd640 1 -- 172.21.15.23:0/545531501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d19c09b740 msgr2=0x55d19c10b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:23:21.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.428+0000 7f48beffd640 1 --2- 172.21.15.23:0/545531501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d19c09b740 0x55d19c10b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.428+0000 7f48beffd640 1 -- 172.21.15.23:0/545531501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d19c0a12a0 msgr2=0x55d19c10ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:23:21.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.428+0000 7f48beffd640 1 --2- 172.21.15.23:0/545531501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d19c0a12a0 0x55d19c10ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.428+0000 7f48beffd640 1 -- 172.21.15.23:0/545531501 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d19c110760 con 0x55d19c09c130 2024-03-28T05:23:21.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.428+0000 7f48be7fc640 1 --2- 172.21.15.23:0/545531501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d19c0a12a0 0x55d19c10ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:23:21.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.428+0000 7f48beffd640 1 --2- 172.21.15.23:0/545531501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09c130 0x55d19c10d880 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f48b8000c00 tx=0x7f48b801e140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeb924313f0c4993 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:21.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.428+0000 7f48a2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:23:21.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.428+0000 7f48a2ff5640 1 -- 172.21.15.23:0/545531501 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b8012690 con 0x55d19c09c130 2024-03-28T05:23:21.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.428+0000 7f48a2ff5640 1 -- 172.21.15.23:0/545531501 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f48b8012830 con 0x55d19c09c130 2024-03-28T05:23:21.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.428+0000 7f48c7ff2740 1 -- 172.21.15.23:0/545531501 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f48b801e1f0 con 0x55d19c09c130 2024-03-28T05:23:21.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.428+0000 7f48a2ff5640 1 -- 172.21.15.23:0/545531501 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48b8038050 con 0x55d19c09c130 2024-03-28T05:23:21.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.428+0000 7f48c7ff2740 10 client.15372 fetch_fsmap learned FSMap version 20 2024-03-28T05:23:21.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.428+0000 7f48c7ff2740 1 -- 172.21.15.23:0/545531501 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55d19c111a90 con 0x55d19c09c130 2024-03-28T05:23:21.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.428+0000 7f48a2ff5640 1 -- 172.21.15.23:0/545531501 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48b801e1f0 con 0x55d19c09c130 2024-03-28T05:23:21.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.428+0000 7f48a2ff5640 1 -- 172.21.15.23:0/545531501 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f48b8027420 con 0x55d19c09c130 2024-03-28T05:23:21.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.428+0000 7f48c7ff2740 10 client.15372 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:23:21.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.428+0000 7f48c7ff2740 10 client.15372 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T05:23:21.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.428+0000 7f48c7ff2740 20 client.15372 populate_metadata read hostname 'smithi023' 2024-03-28T05:23:21.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.428+0000 7f48c7ff2740 1 --2- 172.21.15.23:0/545531501 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d19c115fe0 0x55d19c111dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:21.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.429+0000 7f48c7ff2740 4 client.15372 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T05:23:21.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.429+0000 7f48c7ff2740 1 -- 172.21.15.23:0/545531501 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d19c0084b0 con 0x55d19c115fe0 2024-03-28T05:23:21.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.429+0000 7f48be7fc640 1 --2- 172.21.15.23:0/545531501 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d19c115fe0 0x55d19c111dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:21.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.429+0000 7f48a2ff5640 10 client.15372 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:23:21.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.429+0000 7f48be7fc640 1 --2- 172.21.15.23:0/545531501 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d19c115fe0 0x55d19c111dd0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b33df80f85dc6807 server_cookie=bf15efe1761a8ce1 in_seq=0 out_seq=0 2024-03-28T05:23:21.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.429+0000 7f48a2ff5640 1 -- 172.21.15.23:0/545531501 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55d19c0084b0 con 0x55d19c115fe0 2024-03-28T05:23:21.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.429+0000 7f48a2ff5640 10 client.15372 handle_command_reply: tid=0 2024-03-28T05:23:21.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.429+0000 7f48c7ff2740 1 client.15372 shutdown 2024-03-28T05:23:21.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.429+0000 7f48c7ff2740 1 -- 172.21.15.23:0/545531501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09c130 msgr2=0x55d19c10d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:21.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.429+0000 7f48c7ff2740 1 --2- 172.21.15.23:0/545531501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09c130 0x55d19c10d880 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f48b8000c00 tx=0x7f48b801e140 comp rx=0 tx=0).stop 2024-03-28T05:23:21.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.429+0000 7f48c7ff2740 1 -- 172.21.15.23:0/545531501 shutdown_connections 2024-03-28T05:23:21.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.429+0000 7f48c7ff2740 1 --2- 172.21.15.23:0/545531501 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d19c115fe0 0x55d19c111dd0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.429+0000 7f48c7ff2740 1 --2- 172.21.15.23:0/545531501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d19c0a12a0 0x55d19c10ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.429+0000 7f48c7ff2740 1 --2- 172.21.15.23:0/545531501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d19c09c130 0x55d19c10d880 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.429+0000 7f48c7ff2740 1 --2- 172.21.15.23:0/545531501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d19c09b740 0x55d19c10b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.429+0000 7f48a2ff5640 0 client.15372 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:23:21.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.429+0000 7f48c7ff2740 1 -- 172.21.15.23:0/545531501 >> 172.21.15.23:0/545531501 conn(0x55d19c063a60 msgr2=0x55d19c0ba550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:21.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.429+0000 7f48c7ff2740 1 -- 172.21.15.23:0/545531501 shutdown_connections 2024-03-28T05:23:21.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.429+0000 7f48c7ff2740 1 -- 172.21.15.23:0/545531501 wait complete. 2024-03-28T05:23:21.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.429+0000 7f48c7ff2740 20 client.15372 trim_cache size 0 max 16384 2024-03-28T05:23:21.439 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T05:23:21.439 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "bf37ebc4-ecc1-11ee-95d0-87774f69a715", 2024-03-28T05:23:21.439 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 0, 2024-03-28T05:23:21.439 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24487, 2024-03-28T05:23:21.439 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-28T05:23:21.439 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-28T05:23:21.439 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-28T05:23:21.440 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 133.18235813000001, 2024-03-28T05:23:21.440 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 20, 2024-03-28T05:23:21.440 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 93, 2024-03-28T05:23:21.440 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 89, 2024-03-28T05:23:21.440 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 145.93600996800001 2024-03-28T05:23:21.440 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T05:23:21.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.430+0000 7f48c5329640 1 -- 172.21.15.23:0/286129388 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f48a80a2800 msgr2=0x7f48a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:21.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.430+0000 7f48c5329640 1 --2- 172.21.15.23:0/286129388 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f48a80a2800 0x7f48a80a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f48b00b97d0 tx=0x7f48b00dd040 comp rx=0 tx=0).stop 2024-03-28T05:23:21.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.430+0000 7f48c5329640 1 -- 172.21.15.23:0/286129388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48c0107400 msgr2=0x7f48c019ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:21.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.430+0000 7f48c5329640 1 --2- 172.21.15.23:0/286129388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48c0107400 0x7f48c019ef80 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f48b0037040 tx=0x7f48b0032f20 comp rx=0 tx=0).stop 2024-03-28T05:23:21.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.430+0000 7f48c5329640 1 -- 172.21.15.23:0/286129388 shutdown_connections 2024-03-28T05:23:21.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.430+0000 7f48c5329640 1 --2- 172.21.15.23:0/286129388 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f48a80a2800 0x7f48a80a4cc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.430+0000 7f48c5329640 1 --2- 172.21.15.23:0/286129388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48c0107400 0x7f48c019ef80 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.430+0000 7f48c5329640 1 --2- 172.21.15.23:0/286129388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48c0104ad0 0x7f48c0197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.430+0000 7f48c5329640 1 --2- 172.21.15.23:0/286129388 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48c01021a0 0x7f48c01979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:21.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.430+0000 7f48c5329640 1 -- 172.21.15.23:0/286129388 >> 172.21.15.23:0/286129388 conn(0x7f48c00fbb30 msgr2=0x7f48c0102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:21.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.430+0000 7f48c5329640 1 -- 172.21.15.23:0/286129388 shutdown_connections 2024-03-28T05:23:21.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:21.430+0000 7f48c5329640 1 -- 172.21.15.23:0/286129388 wait complete. 2024-03-28T05:23:21.443 INFO:teuthology.run_tasks:Running task kclient... 2024-03-28T05:23:21.458 INFO:tasks.kclient:Mounting kernel clients... 2024-03-28T05:23:21.458 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-28T05:23:21.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:23:21.458 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2024-03-28T05:23:21.474 INFO:teuthology.orchestra.run.smithi023.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-28T05:23:21.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:23:21.475 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T05:23:21.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:23:21.543 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link delete ceph-brx 2024-03-28T05:23:21.644 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-28T05:23:21.644 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T05:23:21.644 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T05:23:21.644 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph fs ls 2024-03-28T05:23:21.890 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:23:22.011 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:21 smithi023 ceph-mon[31954]: pgmap v359: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1018 B/s rd, 1 op/s; 57 B/s, 0 objects/s recovering 2024-03-28T05:23:22.012 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:21 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3463662266' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:23:22.012 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:21 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2284081646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:23:22.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:21 smithi098 ceph-mon[36087]: pgmap v359: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1018 B/s rd, 1 op/s; 57 B/s, 0 objects/s recovering 2024-03-28T05:23:22.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:21 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3463662266' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:23:22.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:21 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2284081646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:23:22.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:21 smithi113 ceph-mon[37164]: pgmap v359: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1018 B/s rd, 1 op/s; 57 B/s, 0 objects/s recovering 2024-03-28T05:23:22.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:21 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3463662266' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:23:22.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:21 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2284081646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:23:23.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.519+0000 7f16c0c97700 1 -- 172.21.15.23:0/998358229 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16bc1016e0 msgr2=0x7f16bc106e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:23.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.519+0000 7f16c0c97700 1 --2- 172.21.15.23:0/998358229 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16bc1016e0 0x7f16bc106e20 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f16a40099c0 tx=0x7f16a4009cd0 comp rx=0 tx=0).stop 2024-03-28T05:23:23.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.519+0000 7f16c0c97700 1 -- 172.21.15.23:0/998358229 shutdown_connections 2024-03-28T05:23:23.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.519+0000 7f16c0c97700 1 --2- 172.21.15.23:0/998358229 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16bc107550 0x7f16bc10f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:23.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.519+0000 7f16c0c97700 1 --2- 172.21.15.23:0/998358229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16bc100d30 0x7f16bc101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:23.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.519+0000 7f16c0c97700 1 --2- 172.21.15.23:0/998358229 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16bc1016e0 0x7f16bc106e20 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:23.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.519+0000 7f16c0c97700 1 -- 172.21.15.23:0/998358229 >> 172.21.15.23:0/998358229 conn(0x7f16bc0763d0 msgr2=0x7f16bc0767e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:23.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.519+0000 7f16c0c97700 1 -- 172.21.15.23:0/998358229 shutdown_connections 2024-03-28T05:23:23.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.519+0000 7f16c0c97700 1 -- 172.21.15.23:0/998358229 wait complete. 2024-03-28T05:23:23.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.520+0000 7f16c0c97700 1 Processor -- start 2024-03-28T05:23:23.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.520+0000 7f16c0c97700 1 -- start start 2024-03-28T05:23:23.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.520+0000 7f16c0c97700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16bc100d30 0x7f16bc19e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:23.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.520+0000 7f16c0c97700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16bc1016e0 0x7f16bc19eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:23.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.520+0000 7f16c0c97700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16bc107550 0x7f16bc1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:23.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.520+0000 7f16c0c97700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc1124c0 con 0x7f16bc1016e0 2024-03-28T05:23:23.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.520+0000 7f16c0c97700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc112340 con 0x7f16bc100d30 2024-03-28T05:23:23.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.520+0000 7f16c0c97700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc112640 con 0x7f16bc107550 2024-03-28T05:23:23.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.520+0000 7f16b9d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16bc1016e0 0x7f16bc19eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:23.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.520+0000 7f16b9d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16bc1016e0 0x7f16bc19eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37182/0 (socket says 172.21.15.23:37182) 2024-03-28T05:23:23.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.520+0000 7f16b9d9b700 1 -- 172.21.15.23:0/3620789987 learned_addr learned my addr 172.21.15.23:0/3620789987 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:23.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.520+0000 7f16bad9d700 1 --2- 172.21.15.23:0/3620789987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16bc107550 0x7f16bc1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:23.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.520+0000 7f16ba59c700 1 --2- 172.21.15.23:0/3620789987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16bc100d30 0x7f16bc19e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:23.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.520+0000 7f16b9d9b700 1 -- 172.21.15.23:0/3620789987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16bc107550 msgr2=0x7f16bc1986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:23.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.520+0000 7f16b9d9b700 1 --2- 172.21.15.23:0/3620789987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16bc107550 0x7f16bc1986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:23.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.520+0000 7f16b9d9b700 1 -- 172.21.15.23:0/3620789987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16bc100d30 msgr2=0x7f16bc19e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:23.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.520+0000 7f16b9d9b700 1 --2- 172.21.15.23:0/3620789987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16bc100d30 0x7f16bc19e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:23.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.520+0000 7f16b9d9b700 1 -- 172.21.15.23:0/3620789987 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16bc198fc0 con 0x7f16bc1016e0 2024-03-28T05:23:23.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.520+0000 7f16ba59c700 1 --2- 172.21.15.23:0/3620789987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16bc100d30 0x7f16bc19e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:23:23.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.521+0000 7f16b9d9b700 1 --2- 172.21.15.23:0/3620789987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16bc1016e0 0x7f16bc19eb50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f16a40098e0 tx=0x7f16a400fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:23.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.521+0000 7f16b37fe700 1 -- 172.21.15.23:0/3620789987 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a4027070 con 0x7f16bc1016e0 2024-03-28T05:23:23.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.521+0000 7f16c0c97700 1 -- 172.21.15.23:0/3620789987 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16bc199250 con 0x7f16bc1016e0 2024-03-28T05:23:23.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.521+0000 7f16b37fe700 1 -- 172.21.15.23:0/3620789987 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16a4003680 con 0x7f16bc1016e0 2024-03-28T05:23:23.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.521+0000 7f16c0c97700 1 -- 172.21.15.23:0/3620789987 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16bc1a5590 con 0x7f16bc1016e0 2024-03-28T05:23:23.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.521+0000 7f16b37fe700 1 -- 172.21.15.23:0/3620789987 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a40173f0 con 0x7f16bc1016e0 2024-03-28T05:23:23.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.522+0000 7f16c0c97700 1 -- 172.21.15.23:0/3620789987 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16bc06a380 con 0x7f16bc1016e0 2024-03-28T05:23:23.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.523+0000 7f16b37fe700 1 -- 172.21.15.23:0/3620789987 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f16a402a070 con 0x7f16bc1016e0 2024-03-28T05:23:23.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.523+0000 7f16b37fe700 1 --2- 172.21.15.23:0/3620789987 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f16a80a2890 0x7f16a80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:23.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.524+0000 7f16ba59c700 1 --2- 172.21.15.23:0/3620789987 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f16a80a2890 0x7f16a80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:23.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.524+0000 7f16b37fe700 1 -- 172.21.15.23:0/3620789987 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f16a40cfa80 con 0x7f16bc1016e0 2024-03-28T05:23:23.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.524+0000 7f16ba59c700 1 --2- 172.21.15.23:0/3620789987 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f16a80a2890 0x7f16a80a4d50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f16b400e300 tx=0x7f16b4009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:23.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.528+0000 7f16b37fe700 1 -- 172.21.15.23:0/3620789987 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f16a40d1e20 con 0x7f16bc1016e0 2024-03-28T05:23:23.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.783+0000 7f16c0c97700 1 -- 172.21.15.23:0/3620789987 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f16bc068e50 con 0x7f16bc1016e0 2024-03-28T05:23:23.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.783+0000 7f16b37fe700 1 -- 172.21.15.23:0/3620789987 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) v1 ==== 53+0+88 (secure 0 0 0) 0x7f16a4013070 con 0x7f16bc1016e0 2024-03-28T05:23:23.783 INFO:teuthology.orchestra.run.smithi023.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-28T05:23:23.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.785+0000 7f16c0c97700 1 -- 172.21.15.23:0/3620789987 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f16a80a2890 msgr2=0x7f16a80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:23.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.785+0000 7f16c0c97700 1 --2- 172.21.15.23:0/3620789987 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f16a80a2890 0x7f16a80a4d50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f16b400e300 tx=0x7f16b4009040 comp rx=0 tx=0).stop 2024-03-28T05:23:23.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.785+0000 7f16c0c97700 1 -- 172.21.15.23:0/3620789987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16bc1016e0 msgr2=0x7f16bc19eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:23.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.785+0000 7f16c0c97700 1 --2- 172.21.15.23:0/3620789987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16bc1016e0 0x7f16bc19eb50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f16a40098e0 tx=0x7f16a400fa00 comp rx=0 tx=0).stop 2024-03-28T05:23:23.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.785+0000 7f16c0c97700 1 -- 172.21.15.23:0/3620789987 shutdown_connections 2024-03-28T05:23:23.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.785+0000 7f16c0c97700 1 --2- 172.21.15.23:0/3620789987 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f16a80a2890 0x7f16a80a4d50 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:23.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.785+0000 7f16c0c97700 1 --2- 172.21.15.23:0/3620789987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f16bc107550 0x7f16bc1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:23.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.785+0000 7f16c0c97700 1 --2- 172.21.15.23:0/3620789987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16bc100d30 0x7f16bc19e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:23.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.785+0000 7f16c0c97700 1 --2- 172.21.15.23:0/3620789987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16bc1016e0 0x7f16bc19eb50 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:23.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.785+0000 7f16c0c97700 1 -- 172.21.15.23:0/3620789987 >> 172.21.15.23:0/3620789987 conn(0x7f16bc0763d0 msgr2=0x7f16bc0fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:23.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.785+0000 7f16c0c97700 1 -- 172.21.15.23:0/3620789987 shutdown_connections 2024-03-28T05:23:23.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:23.785+0000 7f16c0c97700 1 -- 172.21.15.23:0/3620789987 wait complete. 2024-03-28T05:23:24.102 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:23 smithi023 ceph-mon[31954]: pgmap v360: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s; 48 B/s, 0 objects/s recovering 2024-03-28T05:23:24.102 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:23 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3620789987' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T05:23:24.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:23 smithi098 ceph-mon[36087]: pgmap v360: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s; 48 B/s, 0 objects/s recovering 2024-03-28T05:23:24.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:23 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3620789987' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T05:23:24.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:23 smithi113 ceph-mon[37164]: pgmap v360: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s; 48 B/s, 0 objects/s recovering 2024-03-28T05:23:24.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:23 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3620789987' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T05:23:24.399 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-28T05:23:24.399 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T05:23:24.399 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi023.front.sepia.ceph.com 2024-03-28T05:23:24.399 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-28T05:23:24.399 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T05:23:24.399 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-28T05:23:24.399 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T05:23:24.399 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T05:23:24.399 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-28T05:23:24.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:23:24.400 DEBUG:teuthology.orchestra.run.smithi023:> ip addr 2024-03-28T05:23:24.416 INFO:teuthology.orchestra.run.smithi023.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T05:23:24.416 INFO:teuthology.orchestra.run.smithi023.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T05:23:24.416 INFO:teuthology.orchestra.run.smithi023.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T05:23:24.416 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-28T05:23:24.416 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 ::1/128 scope host 2024-03-28T05:23:24.416 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-28T05:23:24.416 INFO:teuthology.orchestra.run.smithi023.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T05:23:24.416 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:ae brd ff:ff:ff:ff:ff:ff 2024-03-28T05:23:24.416 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f0 2024-03-28T05:23:24.416 INFO:teuthology.orchestra.run.smithi023.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T05:23:24.417 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:af brd ff:ff:ff:ff:ff:ff 2024-03-28T05:23:24.417 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f1 2024-03-28T05:23:24.417 INFO:teuthology.orchestra.run.smithi023.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T05:23:24.417 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:42 brd ff:ff:ff:ff:ff:ff 2024-03-28T05:23:24.417 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f0 2024-03-28T05:23:24.417 INFO:teuthology.orchestra.run.smithi023.stdout: inet 172.21.15.23/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-28T05:23:24.417 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft 42103sec preferred_lft 42103sec 2024-03-28T05:23:24.417 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 fe80::ec4:7aff:fe59:ce42/64 scope link noprefixroute 2024-03-28T05:23:24.417 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-28T05:23:24.417 INFO:teuthology.orchestra.run.smithi023.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T05:23:24.417 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:43 brd ff:ff:ff:ff:ff:ff 2024-03-28T05:23:24.417 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f1 2024-03-28T05:23:24.417 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T05:23:24.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:23:24.418 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-28T05:23:24.418 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-28T05:23:24.418 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add name ceph-brx type bridge 2024-03-28T05:23:24.418 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr flush dev ceph-brx 2024-03-28T05:23:24.418 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set ceph-brx up 2024-03-28T05:23:24.418 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T05:23:24.418 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-28T05:23:24.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:23:24.528 DEBUG:teuthology.orchestra.run.smithi023:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T05:23:24.554 INFO:teuthology.orchestra.run.smithi023.stdout:1 2024-03-28T05:23:24.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:23:24.556 DEBUG:teuthology.orchestra.run.smithi023:> route 2024-03-28T05:23:24.613 INFO:teuthology.orchestra.run.smithi023.stdout:Kernel IP routing table 2024-03-28T05:23:24.613 INFO:teuthology.orchestra.run.smithi023.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T05:23:24.613 INFO:teuthology.orchestra.run.smithi023.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-28T05:23:24.613 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-28T05:23:24.613 INFO:teuthology.orchestra.run.smithi023.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T05:23:24.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:23:24.613 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-28T05:23:24.613 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-28T05:23:24.613 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-28T05:23:24.614 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-28T05:23:24.614 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-28T05:23:24.614 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-28T05:23:24.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:23:24.710 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2024-03-28T05:23:24.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:23:24.725 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list-id 2024-03-28T05:23:24.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:23:24.780 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-28T05:23:24.780 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-28T05:23:24.780 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T05:23:24.780 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-28T05:23:24.780 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-28T05:23:24.863 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-28T05:23:24.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:23:24.863 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-28T05:23:24.864 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-28T05:23:24.864 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-28T05:23:24.864 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T05:23:24.864 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-28T05:23:24.864 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-28T05:23:24.864 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-28T05:23:24.864 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-28T05:23:24.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:23:24.989 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-28T05:23:24.989 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-28T05:23:24.989 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set brx.0 up 2024-03-28T05:23:24.989 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T05:23:24.989 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-28T05:23:25.030 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T05:23:25.030 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-28T05:23:25.085 INFO:teuthology.orchestra.run.smithi023.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-28T05:23:25.085 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T05:23:25.085 DEBUG:teuthology.orchestra.run.smithi023:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-28T05:23:25.139 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-28T05:23:25.139 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-28T05:23:25.140 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/30671fb5-9586-458a-b54e-fbeb280f6667 2024-03-28T05:23:25.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:23:25.140 DEBUG:teuthology.orchestra.run.smithi023:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/30671fb5-9586-458a-b54e-fbeb280f6667 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-03-28T05:23:25.223 INFO:teuthology.orchestra.run.smithi023.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-03-28T05:23:25.223 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2024-03-28T05:23:25.339 INFO:teuthology.orchestra.run.smithi023.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-03-28T05:23:25.339 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2024-03-28T05:23:25.339 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: resolved to: "172.21.15.23:3300,172.21.15.98:3300,172.21.15.113:3300" 2024-03-28T05:23:25.339 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: trying mount with new device syntax: 0@bf37ebc4-ecc1-11ee-95d0-87774f69a715.cephfs=/volumes/qa/sv_1/30671fb5-9586-458a-b54e-fbeb280f6667 2024-03-28T05:23:25.339 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync,name=0,key=0,mon_addr=172.21.15.23:3300/172.21.15.98:3300/172.21.15.113:3300" will pass to kernel 2024-03-28T05:23:25.340 INFO:teuthology.orchestra.run.smithi023.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-28T05:23:25.340 INFO:teuthology.orchestra.run.smithi023.stdout: You just mounted a file system that supports labels which does not 2024-03-28T05:23:25.340 INFO:teuthology.orchestra.run.smithi023.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-28T05:23:25.340 INFO:teuthology.orchestra.run.smithi023.stdout: applications will generate AVC messages and not be allowed access to 2024-03-28T05:23:25.340 INFO:teuthology.orchestra.run.smithi023.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-28T05:23:25.342 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-28T05:23:25.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:23:25.342 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-28T05:23:25.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:23:25.369 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T05:23:25.425 INFO:teuthology.orchestra.run.smithi023.stdout:client25219 2024-03-28T05:23:25.426 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T05:23:25.426 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T05:23:25.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:23:25.482 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T05:23:25.538 INFO:teuthology.orchestra.run.smithi023.stdout:client25219 2024-03-28T05:23:25.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:23:25.538 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T05:23:25.593 INFO:teuthology.orchestra.run.smithi023.stdout:client25219 2024-03-28T05:23:25.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:23:25.594 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715.client25219/status') 2024-03-28T05:23:25.660 INFO:teuthology.orchestra.run.smithi023.stdout:instance: client.25219 (3)172.21.15.23:0/3758255652 2024-03-28T05:23:25.661 INFO:teuthology.orchestra.run.smithi023.stdout:blocklisted: false 2024-03-28T05:23:25.661 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2024-03-28T05:23:25.661 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2024-03-28T05:23:25.661 INFO:teuthology.orchestra.run.smithi023.stderr:71 bytes copied, 2.1897e-05 s, 3.2 MB/s 2024-03-28T05:23:25.662 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-28T05:23:25.673 INFO:teuthology.run_tasks:Running task exec... 2024-03-28T05:23:25.693 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T05:23:25.693 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-28T05:23:25.693 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-28T05:23:25.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.805+0000 7f6deff1b640 1 -- 172.21.15.23:0/1374731209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6de8101800 msgr2=0x7f6de8101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:25.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.805+0000 7f6deff1b640 1 --2- 172.21.15.23:0/1374731209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6de8101800 0x7f6de8101c80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8009900 tx=0x7f6dd802bc30 comp rx=0 tx=0).stop 2024-03-28T05:23:25.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.805+0000 7f6deff1b640 1 -- 172.21.15.23:0/1374731209 shutdown_connections 2024-03-28T05:23:25.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.805+0000 7f6deff1b640 1 --2- 172.21.15.23:0/1374731209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6de81021c0 0x7f6de8108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:25.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.805+0000 7f6deff1b640 1 --2- 172.21.15.23:0/1374731209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6de8101800 0x7f6de8101c80 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:25.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.805+0000 7f6deff1b640 1 --2- 172.21.15.23:0/1374731209 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6de8100610 0x7f6de8100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:25.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.805+0000 7f6deff1b640 1 -- 172.21.15.23:0/1374731209 >> 172.21.15.23:0/1374731209 conn(0x7f6de80fbb80 msgr2=0x7f6de80fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:25.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.805+0000 7f6deff1b640 1 -- 172.21.15.23:0/1374731209 shutdown_connections 2024-03-28T05:23:25.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.805+0000 7f6deff1b640 1 -- 172.21.15.23:0/1374731209 wait complete. 2024-03-28T05:23:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.806+0000 7f6deff1b640 1 Processor -- start 2024-03-28T05:23:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.806+0000 7f6deff1b640 1 -- start start 2024-03-28T05:23:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.806+0000 7f6deff1b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6de8100610 0x7f6de81979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.806+0000 7f6deff1b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6de8101800 0x7f6de8197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.806+0000 7f6deff1b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6de81021c0 0x7f6de819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.806+0000 7f6deff1b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de810c360 con 0x7f6de8101800 2024-03-28T05:23:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.806+0000 7f6deff1b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de810c1e0 con 0x7f6de8100610 2024-03-28T05:23:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.806+0000 7f6deff1b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de810c4e0 con 0x7f6de81021c0 2024-03-28T05:23:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.806+0000 7f6ded4b6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6de8101800 0x7f6de8197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.806+0000 7f6ded4b6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6de8101800 0x7f6de8197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37194/0 (socket says 172.21.15.23:37194) 2024-03-28T05:23:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.806+0000 7f6ded4b6640 1 -- 172.21.15.23:0/884146654 learned_addr learned my addr 172.21.15.23:0/884146654 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.806+0000 7f6dee4b8640 1 --2- 172.21.15.23:0/884146654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6de81021c0 0x7f6de819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.806+0000 7f6dedcb7640 1 --2- 172.21.15.23:0/884146654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6de8100610 0x7f6de81979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.806+0000 7f6ded4b6640 1 -- 172.21.15.23:0/884146654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6de81021c0 msgr2=0x7f6de819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.806+0000 7f6ded4b6640 1 --2- 172.21.15.23:0/884146654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6de81021c0 0x7f6de819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:25.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.806+0000 7f6ded4b6640 1 -- 172.21.15.23:0/884146654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6de8100610 msgr2=0x7f6de81979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:25.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.806+0000 7f6ded4b6640 1 --2- 172.21.15.23:0/884146654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6de8100610 0x7f6de81979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:25.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.806+0000 7f6ded4b6640 1 -- 172.21.15.23:0/884146654 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de819f600 con 0x7f6de8101800 2024-03-28T05:23:25.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.806+0000 7f6dedcb7640 1 --2- 172.21.15.23:0/884146654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6de8100610 0x7f6de81979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:23:25.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.807+0000 7f6ded4b6640 1 --2- 172.21.15.23:0/884146654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6de8101800 0x7f6de8197f20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8005ab0 tx=0x7f6dd802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:25.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.807+0000 7f6ddeffd640 1 -- 172.21.15.23:0/884146654 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd8043020 con 0x7f6de8101800 2024-03-28T05:23:25.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.807+0000 7f6ddeffd640 1 -- 172.21.15.23:0/884146654 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6dd80054e0 con 0x7f6de8101800 2024-03-28T05:23:25.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.807+0000 7f6ddeffd640 1 -- 172.21.15.23:0/884146654 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd8003ce0 con 0x7f6de8101800 2024-03-28T05:23:25.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.807+0000 7f6deff1b640 1 -- 172.21.15.23:0/884146654 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de819f890 con 0x7f6de8101800 2024-03-28T05:23:25.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.807+0000 7f6deff1b640 1 -- 172.21.15.23:0/884146654 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de819fe30 con 0x7f6de8101800 2024-03-28T05:23:25.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.808+0000 7f6deff1b640 1 -- 172.21.15.23:0/884146654 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de804f400 con 0x7f6de8101800 2024-03-28T05:23:25.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.809+0000 7f6ddeffd640 1 -- 172.21.15.23:0/884146654 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6dd8046020 con 0x7f6de8101800 2024-03-28T05:23:25.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.810+0000 7f6ddeffd640 1 --2- 172.21.15.23:0/884146654 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f6dd40a2920 0x7f6dd40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:25.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.810+0000 7f6dedcb7640 1 --2- 172.21.15.23:0/884146654 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f6dd40a2920 0x7f6dd40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:25.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.810+0000 7f6ddeffd640 1 -- 172.21.15.23:0/884146654 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f6dd80f0fb0 con 0x7f6de8101800 2024-03-28T05:23:25.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.810+0000 7f6dedcb7640 1 --2- 172.21.15.23:0/884146654 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f6dd40a2920 0x7f6dd40a4de0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6de8101660 tx=0x7f6de4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:25.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:25.814+0000 7f6ddeffd640 1 -- 172.21.15.23:0/884146654 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6dd804d020 con 0x7f6de8101800 2024-03-28T05:23:26.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:25 smithi023 ceph-mon[31954]: pgmap v361: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 48 B/s, 0 objects/s recovering 2024-03-28T05:23:26.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.050+0000 7f6deff1b640 1 -- 172.21.15.23:0/884146654 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f6de81a0760 con 0x7f6de8101800 2024-03-28T05:23:26.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:25 smithi098 ceph-mon[36087]: pgmap v361: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 48 B/s, 0 objects/s recovering 2024-03-28T05:23:26.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:25 smithi113 ceph-mon[37164]: pgmap v361: 129 pgs: 129 active+clean; 639 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 48 B/s, 0 objects/s recovering 2024-03-28T05:23:26.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.852+0000 7f6ddeffd640 1 -- 172.21.15.23:0/884146654 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7f6dd80bdca0 con 0x7f6de8101800 2024-03-28T05:23:26.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.856+0000 7f6deff1b640 1 -- 172.21.15.23:0/884146654 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f6dd40a2920 msgr2=0x7f6dd40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:26.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.856+0000 7f6deff1b640 1 --2- 172.21.15.23:0/884146654 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f6dd40a2920 0x7f6dd40a4de0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6de8101660 tx=0x7f6de4005f90 comp rx=0 tx=0).stop 2024-03-28T05:23:26.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.856+0000 7f6deff1b640 1 -- 172.21.15.23:0/884146654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6de8101800 msgr2=0x7f6de8197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:26.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.856+0000 7f6deff1b640 1 --2- 172.21.15.23:0/884146654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6de8101800 0x7f6de8197f20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8005ab0 tx=0x7f6dd802c6c0 comp rx=0 tx=0).stop 2024-03-28T05:23:26.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.856+0000 7f6deff1b640 1 -- 172.21.15.23:0/884146654 shutdown_connections 2024-03-28T05:23:26.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.856+0000 7f6deff1b640 1 --2- 172.21.15.23:0/884146654 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f6dd40a2920 0x7f6dd40a4de0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:26.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.856+0000 7f6deff1b640 1 --2- 172.21.15.23:0/884146654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6de81021c0 0x7f6de819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:26.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.856+0000 7f6deff1b640 1 --2- 172.21.15.23:0/884146654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6de8101800 0x7f6de8197f20 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:26.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.856+0000 7f6deff1b640 1 --2- 172.21.15.23:0/884146654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6de8100610 0x7f6de81979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:26.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.856+0000 7f6deff1b640 1 -- 172.21.15.23:0/884146654 >> 172.21.15.23:0/884146654 conn(0x7f6de80fbb80 msgr2=0x7f6de80fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:26.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.856+0000 7f6deff1b640 1 -- 172.21.15.23:0/884146654 shutdown_connections 2024-03-28T05:23:26.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.856+0000 7f6deff1b640 1 -- 172.21.15.23:0/884146654 wait complete. 2024-03-28T05:23:26.873 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-28T05:23:26.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.985+0000 7f5ae3787640 1 -- 172.21.15.23:0/496910038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5adc102140 msgr2=0x7f5adc104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:26.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.985+0000 7f5ae3787640 1 --2- 172.21.15.23:0/496910038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5adc102140 0x7f5adc104530 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5acc009900 tx=0x7f5acc02bc30 comp rx=0 tx=0).stop 2024-03-28T05:23:26.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.985+0000 7f5ae3787640 1 -- 172.21.15.23:0/496910038 shutdown_connections 2024-03-28T05:23:26.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.985+0000 7f5ae3787640 1 --2- 172.21.15.23:0/496910038 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5adc1073a0 0x7f5adc1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:26.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.985+0000 7f5ae3787640 1 --2- 172.21.15.23:0/496910038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5adc104a70 0x7f5adc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:26.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.985+0000 7f5ae3787640 1 --2- 172.21.15.23:0/496910038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5adc102140 0x7f5adc104530 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:26.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.985+0000 7f5ae3787640 1 -- 172.21.15.23:0/496910038 >> 172.21.15.23:0/496910038 conn(0x7f5adc0fbb30 msgr2=0x7f5adc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:26.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.985+0000 7f5ae3787640 1 -- 172.21.15.23:0/496910038 shutdown_connections 2024-03-28T05:23:26.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.985+0000 7f5ae3787640 1 -- 172.21.15.23:0/496910038 wait complete. 2024-03-28T05:23:26.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.986+0000 7f5ae3787640 1 Processor -- start 2024-03-28T05:23:26.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.986+0000 7f5ae3787640 1 -- start start 2024-03-28T05:23:26.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.986+0000 7f5ae3787640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5adc102140 0x7f5adc1979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:26.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.986+0000 7f5ae3787640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5adc104a70 0x7f5adc197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:26.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.986+0000 7f5ae3787640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5adc1073a0 0x7f5adc19ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:26.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.986+0000 7f5ae3787640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc10d390 con 0x7f5adc1073a0 2024-03-28T05:23:26.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.986+0000 7f5ae3787640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc10d210 con 0x7f5adc104a70 2024-03-28T05:23:26.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.986+0000 7f5ae3787640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc10d510 con 0x7f5adc102140 2024-03-28T05:23:26.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.986+0000 7f5ae1d24640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5adc1073a0 0x7f5adc19ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:26.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.986+0000 7f5ae1d24640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5adc1073a0 0x7f5adc19ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37236/0 (socket says 172.21.15.23:37236) 2024-03-28T05:23:26.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.986+0000 7f5ae1d24640 1 -- 172.21.15.23:0/2409694744 learned_addr learned my addr 172.21.15.23:0/2409694744 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:26.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.986+0000 7f5ae1523640 1 --2- 172.21.15.23:0/2409694744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5adc102140 0x7f5adc1979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:26.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.986+0000 7f5ae0d22640 1 --2- 172.21.15.23:0/2409694744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5adc104a70 0x7f5adc197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:26.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.986+0000 7f5ae1d24640 1 -- 172.21.15.23:0/2409694744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5adc102140 msgr2=0x7f5adc1979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:26.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.986+0000 7f5ae1d24640 1 --2- 172.21.15.23:0/2409694744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5adc102140 0x7f5adc1979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:26.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.986+0000 7f5ae1d24640 1 -- 172.21.15.23:0/2409694744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5adc104a70 msgr2=0x7f5adc197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:26.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.986+0000 7f5ae1d24640 1 --2- 172.21.15.23:0/2409694744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5adc104a70 0x7f5adc197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:26.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.986+0000 7f5ae1d24640 1 -- 172.21.15.23:0/2409694744 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5adc19f5c0 con 0x7f5adc1073a0 2024-03-28T05:23:26.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.987+0000 7f5ae1d24640 1 --2- 172.21.15.23:0/2409694744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5adc1073a0 0x7f5adc19ef50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4009fd0 tx=0x7f5ad400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:26.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.987+0000 7f5ad27fc640 1 -- 172.21.15.23:0/2409694744 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad4017070 con 0x7f5adc1073a0 2024-03-28T05:23:26.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.987+0000 7f5ae3787640 1 -- 172.21.15.23:0/2409694744 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5adc19f8b0 con 0x7f5adc1073a0 2024-03-28T05:23:26.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.987+0000 7f5ad27fc640 1 -- 172.21.15.23:0/2409694744 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ad4003e50 con 0x7f5adc1073a0 2024-03-28T05:23:26.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.987+0000 7f5ad27fc640 1 -- 172.21.15.23:0/2409694744 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad40106c0 con 0x7f5adc1073a0 2024-03-28T05:23:26.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.987+0000 7f5ae3787640 1 -- 172.21.15.23:0/2409694744 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5adc19fdb0 con 0x7f5adc1073a0 2024-03-28T05:23:26.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.988+0000 7f5ae3787640 1 -- 172.21.15.23:0/2409694744 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5adc0fd620 con 0x7f5adc1073a0 2024-03-28T05:23:26.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.989+0000 7f5ad27fc640 1 -- 172.21.15.23:0/2409694744 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 137307+0+0 (secure 0 0 0) 0x7f5ad4003970 con 0x7f5adc1073a0 2024-03-28T05:23:26.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.990+0000 7f5ad27fc640 1 --2- 172.21.15.23:0/2409694744 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5ac80a28f0 0x7f5ac80a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:26.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.990+0000 7f5ae1523640 1 -- 172.21.15.23:0/2409694744 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5ac80a28f0 msgr2=0x7f5ac80a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/222178108 2024-03-28T05:23:26.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.990+0000 7f5ad27fc640 1 -- 172.21.15.23:0/2409694744 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f5ad40cea90 con 0x7f5adc1073a0 2024-03-28T05:23:26.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.990+0000 7f5ae1523640 1 --2- 172.21.15.23:0/2409694744 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5ac80a28f0 0x7f5ac80a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T05:23:26.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:26.998+0000 7f5ad27fc640 1 -- 172.21.15.23:0/2409694744 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5ad409b130 con 0x7f5adc1073a0 2024-03-28T05:23:27.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:26 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/884146654' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T05:23:27.115 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:26 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: ignoring --setuser ceph since I am not root 2024-03-28T05:23:27.115 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:26 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: ignoring --setgroup ceph since I am not root 2024-03-28T05:23:27.115 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:26 smithi098 ceph-mgr[37483]: -- 172.21.15.98:0/813981194 <== mon.1 v2:172.21.15.113:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5578afdb85a0 con 0x5578afd8d000 2024-03-28T05:23:27.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.182+0000 7f5ae3787640 1 -- 172.21.15.23:0/2409694744 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f5adc02d070 con 0x7f5adc1073a0 2024-03-28T05:23:27.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.186+0000 7f5ad27fc640 1 -- 172.21.15.23:0/2409694744 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v28)=0 v28) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5ad409f1d0 con 0x7f5adc1073a0 2024-03-28T05:23:27.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.189+0000 7f5ae3787640 1 -- 172.21.15.23:0/2409694744 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5ac80a28f0 msgr2=0x7f5ac80a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T05:23:27.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.189+0000 7f5ae3787640 1 --2- 172.21.15.23:0/2409694744 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5ac80a28f0 0x7f5ac80a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:27.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.189+0000 7f5ae3787640 1 -- 172.21.15.23:0/2409694744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5adc1073a0 msgr2=0x7f5adc19ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:27.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.189+0000 7f5ae3787640 1 --2- 172.21.15.23:0/2409694744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5adc1073a0 0x7f5adc19ef50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4009fd0 tx=0x7f5ad400b260 comp rx=0 tx=0).stop 2024-03-28T05:23:27.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.189+0000 7f5ae3787640 1 -- 172.21.15.23:0/2409694744 shutdown_connections 2024-03-28T05:23:27.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.189+0000 7f5ae3787640 1 --2- 172.21.15.23:0/2409694744 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f5ac80a28f0 0x7f5ac80a4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:27.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.189+0000 7f5ae3787640 1 --2- 172.21.15.23:0/2409694744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5adc1073a0 0x7f5adc19ef50 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:27.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.189+0000 7f5ae3787640 1 --2- 172.21.15.23:0/2409694744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5adc104a70 0x7f5adc197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:27.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.189+0000 7f5ae3787640 1 --2- 172.21.15.23:0/2409694744 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5adc102140 0x7f5adc1979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:27.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.189+0000 7f5ae3787640 1 -- 172.21.15.23:0/2409694744 >> 172.21.15.23:0/2409694744 conn(0x7f5adc0fbb30 msgr2=0x7f5adc102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:27.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.189+0000 7f5ae3787640 1 -- 172.21.15.23:0/2409694744 shutdown_connections 2024-03-28T05:23:27.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.189+0000 7f5ae3787640 1 -- 172.21.15.23:0/2409694744 wait complete. 2024-03-28T05:23:27.202 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:26 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/884146654' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T05:23:27.202 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:26 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: ignoring --setuser ceph since I am not root 2024-03-28T05:23:27.203 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:26 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: ignoring --setgroup ceph since I am not root 2024-03-28T05:23:27.203 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-28T05:23:27.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:26 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/884146654' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T05:23:27.242 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:26 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: ignoring --setuser ceph since I am not root 2024-03-28T05:23:27.242 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:26 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: ignoring --setgroup ceph since I am not root 2024-03-28T05:23:27.243 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:26 smithi113 ceph-mgr[39034]: -- 172.21.15.113:0/32875532 <== mon.1 v2:172.21.15.113:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5601d804e5a0 con 0x5601d8022800 2024-03-28T05:23:27.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.309+0000 7f354c1ab640 1 -- 172.21.15.23:0/1745872506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35441017b0 msgr2=0x7f3544101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:27.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.309+0000 7f354c1ab640 1 --2- 172.21.15.23:0/1745872506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35441017b0 0x7f3544101c30 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3534009900 tx=0x7f353402bc30 comp rx=0 tx=0).stop 2024-03-28T05:23:27.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.309+0000 7f354c1ab640 1 -- 172.21.15.23:0/1745872506 shutdown_connections 2024-03-28T05:23:27.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.309+0000 7f354c1ab640 1 --2- 172.21.15.23:0/1745872506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3544102170 0x7f3544108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:27.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.309+0000 7f354c1ab640 1 --2- 172.21.15.23:0/1745872506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35441017b0 0x7f3544101c30 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:27.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.309+0000 7f354c1ab640 1 --2- 172.21.15.23:0/1745872506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35441005c0 0x7f35441009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:27.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.309+0000 7f354c1ab640 1 -- 172.21.15.23:0/1745872506 >> 172.21.15.23:0/1745872506 conn(0x7f35440fbb30 msgr2=0x7f35440fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:27.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.309+0000 7f354c1ab640 1 -- 172.21.15.23:0/1745872506 shutdown_connections 2024-03-28T05:23:27.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.309+0000 7f354c1ab640 1 -- 172.21.15.23:0/1745872506 wait complete. 2024-03-28T05:23:27.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.309+0000 7f354c1ab640 1 Processor -- start 2024-03-28T05:23:27.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.309+0000 7f354c1ab640 1 -- start start 2024-03-28T05:23:27.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.310+0000 7f354c1ab640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35441005c0 0x7f35441979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:27.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.310+0000 7f354c1ab640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35441017b0 0x7f3544197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:27.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.310+0000 7f354c1ab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3544102170 0x7f354419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:27.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.310+0000 7f354c1ab640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354410c240 con 0x7f35441017b0 2024-03-28T05:23:27.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.310+0000 7f354c1ab640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354410c0c0 con 0x7f3544102170 2024-03-28T05:23:27.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.310+0000 7f354c1ab640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354410c3c0 con 0x7f35441005c0 2024-03-28T05:23:27.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.310+0000 7f3549746640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35441017b0 0x7f3544197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:27.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.310+0000 7f3549746640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35441017b0 0x7f3544197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37256/0 (socket says 172.21.15.23:37256) 2024-03-28T05:23:27.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.310+0000 7f3549746640 1 -- 172.21.15.23:0/2305324235 learned_addr learned my addr 172.21.15.23:0/2305324235 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:27.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.310+0000 7f3549f47640 1 --2- 172.21.15.23:0/2305324235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35441005c0 0x7f35441979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:27.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.310+0000 7f354a748640 1 --2- 172.21.15.23:0/2305324235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3544102170 0x7f354419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:27.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.310+0000 7f3549746640 1 -- 172.21.15.23:0/2305324235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35441005c0 msgr2=0x7f35441979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:27.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.310+0000 7f3549746640 1 --2- 172.21.15.23:0/2305324235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35441005c0 0x7f35441979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:27.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.310+0000 7f3549746640 1 -- 172.21.15.23:0/2305324235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3544102170 msgr2=0x7f354419ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:27.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.310+0000 7f3549746640 1 --2- 172.21.15.23:0/2305324235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3544102170 0x7f354419ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:27.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.310+0000 7f3549746640 1 -- 172.21.15.23:0/2305324235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354419f580 con 0x7f35441017b0 2024-03-28T05:23:27.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.310+0000 7f3549746640 1 --2- 172.21.15.23:0/2305324235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35441017b0 0x7f3544197ef0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f3534009230 tx=0x7f353402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:27.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.310+0000 7f353affd640 1 -- 172.21.15.23:0/2305324235 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3534041070 con 0x7f35441017b0 2024-03-28T05:23:27.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.310+0000 7f353affd640 1 -- 172.21.15.23:0/2305324235 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3534004450 con 0x7f35441017b0 2024-03-28T05:23:27.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.310+0000 7f354c1ab640 1 -- 172.21.15.23:0/2305324235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f354419f810 con 0x7f35441017b0 2024-03-28T05:23:27.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.310+0000 7f353affd640 1 -- 172.21.15.23:0/2305324235 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3534003e20 con 0x7f35441017b0 2024-03-28T05:23:27.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.311+0000 7f354c1ab640 1 -- 172.21.15.23:0/2305324235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f354419fd00 con 0x7f35441017b0 2024-03-28T05:23:27.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.311+0000 7f354c1ab640 1 -- 172.21.15.23:0/2305324235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f354419ff60 con 0x7f35441017b0 2024-03-28T05:23:27.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.312+0000 7f353affd640 1 -- 172.21.15.23:0/2305324235 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 137307+0+0 (secure 0 0 0) 0x7f3534044070 con 0x7f35441017b0 2024-03-28T05:23:27.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.313+0000 7f353affd640 1 --2- 172.21.15.23:0/2305324235 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f35300a28f0 0x7f35300a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:27.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.313+0000 7f3549f47640 1 -- 172.21.15.23:0/2305324235 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f35300a28f0 msgr2=0x7f35300a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/222178108 2024-03-28T05:23:27.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.313+0000 7f353affd640 1 -- 172.21.15.23:0/2305324235 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f35340f0a20 con 0x7f35441017b0 2024-03-28T05:23:27.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.313+0000 7f3549f47640 1 --2- 172.21.15.23:0/2305324235 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f35300a28f0 0x7f35300a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T05:23:27.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.317+0000 7f353affd640 1 -- 172.21.15.23:0/2305324235 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f35340bd0c0 con 0x7f35441017b0 2024-03-28T05:23:27.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.502+0000 7f354c1ab640 1 -- 172.21.15.23:0/2305324235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f35441a0700 con 0x7f35441017b0 2024-03-28T05:23:27.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.507+0000 7f353affd640 1 -- 172.21.15.23:0/2305324235 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v29)=0 v29) v1 ==== 133+0+0 (secure 0 0 0) 0x7f35340c1160 con 0x7f35441017b0 2024-03-28T05:23:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.509+0000 7f354c1ab640 1 -- 172.21.15.23:0/2305324235 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f35300a28f0 msgr2=0x7f35300a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T05:23:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.509+0000 7f354c1ab640 1 --2- 172.21.15.23:0/2305324235 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f35300a28f0 0x7f35300a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.509+0000 7f354c1ab640 1 -- 172.21.15.23:0/2305324235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35441017b0 msgr2=0x7f3544197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.509+0000 7f354c1ab640 1 --2- 172.21.15.23:0/2305324235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35441017b0 0x7f3544197ef0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f3534009230 tx=0x7f353402c650 comp rx=0 tx=0).stop 2024-03-28T05:23:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.509+0000 7f354c1ab640 1 -- 172.21.15.23:0/2305324235 shutdown_connections 2024-03-28T05:23:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.509+0000 7f354c1ab640 1 --2- 172.21.15.23:0/2305324235 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7f35300a28f0 0x7f35300a4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.509+0000 7f354c1ab640 1 --2- 172.21.15.23:0/2305324235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3544102170 0x7f354419ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.509+0000 7f354c1ab640 1 --2- 172.21.15.23:0/2305324235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35441017b0 0x7f3544197ef0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:27.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.509+0000 7f354c1ab640 1 --2- 172.21.15.23:0/2305324235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35441005c0 0x7f35441979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:27.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.509+0000 7f354c1ab640 1 -- 172.21.15.23:0/2305324235 >> 172.21.15.23:0/2305324235 conn(0x7f35440fbb30 msgr2=0x7f35440fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:27.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.509+0000 7f354c1ab640 1 -- 172.21.15.23:0/2305324235 shutdown_connections 2024-03-28T05:23:27.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.510+0000 7f354c1ab640 1 -- 172.21.15.23:0/2305324235 wait complete. 2024-03-28T05:23:27.520 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-28T05:23:27.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.628+0000 7ff57f259640 1 -- 172.21.15.23:0/1352395628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5781005c0 msgr2=0x7ff5781009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:27.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.628+0000 7ff57f259640 1 --2- 172.21.15.23:0/1352395628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5781005c0 0x7ff5781009e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff570009ca0 tx=0x7ff57002c0f0 comp rx=0 tx=0).stop 2024-03-28T05:23:27.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.628+0000 7ff57f259640 1 -- 172.21.15.23:0/1352395628 shutdown_connections 2024-03-28T05:23:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.628+0000 7ff57f259640 1 --2- 172.21.15.23:0/1352395628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff578102170 0x7ff578108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.628+0000 7ff57f259640 1 --2- 172.21.15.23:0/1352395628 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff5781017b0 0x7ff578101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.628+0000 7ff57f259640 1 --2- 172.21.15.23:0/1352395628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5781005c0 0x7ff5781009e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.628+0000 7ff57f259640 1 -- 172.21.15.23:0/1352395628 >> 172.21.15.23:0/1352395628 conn(0x7ff5780fbb30 msgr2=0x7ff5780fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.628+0000 7ff57f259640 1 -- 172.21.15.23:0/1352395628 shutdown_connections 2024-03-28T05:23:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.628+0000 7ff57f259640 1 -- 172.21.15.23:0/1352395628 wait complete. 2024-03-28T05:23:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.628+0000 7ff57f259640 1 Processor -- start 2024-03-28T05:23:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.628+0000 7ff57f259640 1 -- start start 2024-03-28T05:23:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.628+0000 7ff57f259640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5781005c0 0x7ff578196850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.628+0000 7ff57f259640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5781017b0 0x7ff578196d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:27.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.628+0000 7ff57f259640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff578102170 0x7ff57819de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:27.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.628+0000 7ff57f259640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57810c210 con 0x7ff5781005c0 2024-03-28T05:23:27.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.628+0000 7ff57f259640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57810c090 con 0x7ff5781017b0 2024-03-28T05:23:27.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.628+0000 7ff57f259640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57810c390 con 0x7ff578102170 2024-03-28T05:23:27.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.629+0000 7ff57cff5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5781005c0 0x7ff578196850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:27.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.629+0000 7ff57cff5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5781005c0 0x7ff578196850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37276/0 (socket says 172.21.15.23:37276) 2024-03-28T05:23:27.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.629+0000 7ff57cff5640 1 -- 172.21.15.23:0/1186100775 learned_addr learned my addr 172.21.15.23:0/1186100775 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:27.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.629+0000 7ff57d7f6640 1 --2- 172.21.15.23:0/1186100775 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff578102170 0x7ff57819de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:27.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.629+0000 7ff577fff640 1 --2- 172.21.15.23:0/1186100775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5781017b0 0x7ff578196d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:27.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.629+0000 7ff57cff5640 1 -- 172.21.15.23:0/1186100775 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff578102170 msgr2=0x7ff57819de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:27.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.629+0000 7ff57cff5640 1 --2- 172.21.15.23:0/1186100775 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff578102170 0x7ff57819de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:27.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.629+0000 7ff57cff5640 1 -- 172.21.15.23:0/1186100775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5781017b0 msgr2=0x7ff578196d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:27.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.629+0000 7ff57cff5640 1 --2- 172.21.15.23:0/1186100775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff5781017b0 0x7ff578196d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:27.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.629+0000 7ff57cff5640 1 -- 172.21.15.23:0/1186100775 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57819e470 con 0x7ff5781005c0 2024-03-28T05:23:27.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.629+0000 7ff57cff5640 1 --2- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5781005c0 0x7ff578196850 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff5700324c0 tx=0x7ff570032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:27.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.629+0000 7ff575ffb640 1 -- 172.21.15.23:0/1186100775 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff570041020 con 0x7ff5781005c0 2024-03-28T05:23:27.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.629+0000 7ff575ffb640 1 -- 172.21.15.23:0/1186100775 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff570034cd0 con 0x7ff5781005c0 2024-03-28T05:23:27.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.629+0000 7ff57f259640 1 -- 172.21.15.23:0/1186100775 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff57819e700 con 0x7ff5781005c0 2024-03-28T05:23:27.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.629+0000 7ff575ffb640 1 -- 172.21.15.23:0/1186100775 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff570039040 con 0x7ff5781005c0 2024-03-28T05:23:27.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.630+0000 7ff57f259640 1 -- 172.21.15.23:0/1186100775 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff57819ebf0 con 0x7ff5781005c0 2024-03-28T05:23:27.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.630+0000 7ff57f259640 1 -- 172.21.15.23:0/1186100775 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff578107900 con 0x7ff5781005c0 2024-03-28T05:23:27.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.632+0000 7ff575ffb640 1 -- 172.21.15.23:0/1186100775 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 137307+0+0 (secure 0 0 0) 0x7ff570044020 con 0x7ff5781005c0 2024-03-28T05:23:27.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.632+0000 7ff575ffb640 1 --2- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff5600a28f0 0x7ff5600a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:27.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.632+0000 7ff577fff640 1 -- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff5600a28f0 msgr2=0x7ff5600a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/222178108 2024-03-28T05:23:27.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.632+0000 7ff575ffb640 1 -- 172.21.15.23:0/1186100775 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7ff5700f10d0 con 0x7ff5781005c0 2024-03-28T05:23:27.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.632+0000 7ff577fff640 1 --2- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff5600a28f0 0x7ff5600a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T05:23:27.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.636+0000 7ff575ffb640 1 -- 172.21.15.23:0/1186100775 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff57004b020 con 0x7ff5781005c0 2024-03-28T05:23:27.742 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:27 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:27.481+0000 7f051ae7d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T05:23:27.742 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:27 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:27.595+0000 7f051ae7d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T05:23:27.780 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:27 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:27.460+0000 7fb18b183200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T05:23:27.780 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:27 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:27.560+0000 7fb18b183200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T05:23:27.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.833+0000 7ff577fff640 1 -- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff5600a28f0 msgr2=0x7ff5600a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/222178108 2024-03-28T05:23:27.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.833+0000 7ff577fff640 1 --2- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff5600a28f0 0x7ff5600a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T05:23:27.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:27.841+0000 7ff57f259640 1 -- 172.21.15.23:0/1186100775 --> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff578061ae0 con 0x7ff5600a28f0 2024-03-28T05:23:27.853 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:27 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:27.416+0000 7f18d0b81200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T05:23:27.853 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:27 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:27.510+0000 7f18d0b81200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T05:23:28.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:27 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/884146654' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T05:23:28.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:27 smithi098 ceph-mon[36087]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-28T05:23:28.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:27 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2409694744' entity='client.admin' 2024-03-28T05:23:28.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:27 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2305324235' entity='client.admin' 2024-03-28T05:23:28.225 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:27 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/884146654' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T05:23:28.225 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:27 smithi023 ceph-mon[31954]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-28T05:23:28.226 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:27 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2409694744' entity='client.admin' 2024-03-28T05:23:28.226 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:27 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2305324235' entity='client.admin' 2024-03-28T05:23:28.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:28.234+0000 7ff577fff640 1 -- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff5600a28f0 msgr2=0x7ff5600a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/222178108 2024-03-28T05:23:28.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:28.234+0000 7ff577fff640 1 --2- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff5600a28f0 0x7ff5600a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T05:23:28.235 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:27 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/884146654' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T05:23:28.236 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:27 smithi113 ceph-mon[37164]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-28T05:23:28.236 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:27 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2409694744' entity='client.admin' 2024-03-28T05:23:28.236 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:27 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2305324235' entity='client.admin' 2024-03-28T05:23:28.471 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:28 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:28.143+0000 7f18d0b81200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T05:23:28.471 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:28 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:28.235+0000 7f18d0b81200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T05:23:28.471 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:28 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:28.378+0000 7f18d0b81200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T05:23:28.471 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:28 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:28.473+0000 7f18d0b81200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T05:23:28.492 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:28 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:28.237+0000 7f051ae7d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T05:23:28.492 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:28 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:28.353+0000 7f051ae7d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T05:23:28.530 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:28 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:28.226+0000 7fb18b183200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T05:23:28.531 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:28 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:28.323+0000 7fb18b183200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T05:23:28.531 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:28 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:28.468+0000 7fb18b183200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T05:23:28.814 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:28 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:28.555+0000 7f18d0b81200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T05:23:28.814 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:28 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:28.815+0000 7f18d0b81200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T05:23:28.922 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:28 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:28.566+0000 7fb18b183200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T05:23:28.922 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:28 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:28.652+0000 7fb18b183200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T05:23:28.922 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:28 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:28.922+0000 7fb18b183200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T05:23:28.992 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:28 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:28.550+0000 7f051ae7d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T05:23:28.992 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:28 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:28.689+0000 7f051ae7d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T05:23:28.992 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:28 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:28.780+0000 7f051ae7d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T05:23:29.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:29.035+0000 7ff577fff640 1 -- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff5600a28f0 msgr2=0x7ff5600a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/222178108 2024-03-28T05:23:29.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:29.035+0000 7ff577fff640 1 --2- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff5600a28f0 0x7ff5600a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T05:23:29.103 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:28 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:28.924+0000 7f18d0b81200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T05:23:29.103 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:29 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:29.010+0000 7f18d0b81200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T05:23:29.103 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:29 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:29.104+0000 7f18d0b81200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T05:23:29.220 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:29 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:29.035+0000 7fb18b183200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T05:23:29.220 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:29 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:29.123+0000 7fb18b183200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T05:23:29.220 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:29 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:29.220+0000 7fb18b183200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T05:23:29.365 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:29 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:29.280+0000 7f18d0b81200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T05:23:29.413 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:29 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:29.050+0000 7f051ae7d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T05:23:29.413 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:29 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:29.178+0000 7f051ae7d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T05:23:29.413 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:29 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:29.295+0000 7f051ae7d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T05:23:29.530 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:29 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:29.400+0000 7fb18b183200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T05:23:29.742 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:29 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:29.415+0000 7f051ae7d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T05:23:29.742 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:29 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:29.592+0000 7f051ae7d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T05:23:30.615 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:30 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:30.235+0000 7f18d0b81200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T05:23:30.615 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:30 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:30.365+0000 7f18d0b81200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T05:23:30.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:30.636+0000 7ff577fff640 1 -- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff5600a28f0 msgr2=0x7ff5600a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/222178108 2024-03-28T05:23:30.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:30.636+0000 7ff577fff640 1 --2- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff5600a28f0 0x7ff5600a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T05:23:30.780 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:30 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:30.392+0000 7fb18b183200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T05:23:30.780 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:30 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:30.528+0000 7fb18b183200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T05:23:30.977 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:30 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:30.698+0000 7f18d0b81200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T05:23:30.978 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:30 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:30.786+0000 7f18d0b81200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T05:23:30.992 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:30 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:30.571+0000 7f051ae7d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T05:23:30.992 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:30 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:30.726+0000 7f051ae7d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T05:23:31.165 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:30 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:30.870+0000 7fb18b183200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T05:23:31.166 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:30 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:30.963+0000 7fb18b183200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T05:23:31.365 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:30 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:30.979+0000 7f18d0b81200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T05:23:31.365 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:31 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:31.164+0000 7f18d0b81200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T05:23:31.396 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:31 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:31.107+0000 7f051ae7d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T05:23:31.396 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:31 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:31.196+0000 7f051ae7d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T05:23:31.396 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:31 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:31.396+0000 7f051ae7d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T05:23:31.530 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:31 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:31.166+0000 7fb18b183200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T05:23:31.530 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:31 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:31.360+0000 7fb18b183200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T05:23:31.702 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:31 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:31.448+0000 7f18d0b81200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T05:23:31.742 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:31 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:31.584+0000 7f051ae7d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T05:23:31.922 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:31 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:31.658+0000 7fb18b183200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T05:23:32.115 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:31 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:31.704+0000 7f18d0b81200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T05:23:32.115 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:31 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:31.938+0000 7f18d0b81200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T05:23:32.160 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:31 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:31.896+0000 7f051ae7d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T05:23:32.172 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:31 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:31.922+0000 7fb18b183200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T05:23:32.440 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:32 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:32.160+0000 7f051ae7d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T05:23:32.530 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:32 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:32.172+0000 7fb18b183200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T05:23:32.742 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:32 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:32.441+0000 7f051ae7d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T05:23:33.115 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:32 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:32.741+0000 7f18d0b81200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T05:23:33.280 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:33 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:33.012+0000 7fb18b183200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T05:23:33.615 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:33 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:33.309+0000 7f18d0b81200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T05:23:33.615 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:33 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:33.407+0000 7f18d0b81200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T05:23:33.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:33.700+0000 7ff575ffb640 1 -- 172.21.15.23:0/1186100775 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 20) v1 ==== 137307+0+0 (secure 0 0 0) 0x7ff5700b9070 con 0x7ff5781005c0 2024-03-28T05:23:33.702 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:33 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:33.306+0000 7f051ae7d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T05:23:33.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:33.840+0000 7ff577fff640 1 -- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff5600a28f0 msgr2=0x7ff5600a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/222178108 2024-03-28T05:23:33.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:33.840+0000 7ff577fff640 1 --2- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff5600a28f0 0x7ff5600a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-28T05:23:33.935 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:33 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:33.599+0000 7fb18b183200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T05:23:33.935 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:33 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:33.707+0000 7fb18b183200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T05:23:33.935 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:33 smithi023 ceph-mon[31954]: Standby manager daemon y restarted 2024-03-28T05:23:33.935 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:33 smithi023 ceph-mon[31954]: Standby manager daemon y started 2024-03-28T05:23:33.935 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:33 smithi023 ceph-mon[31954]: from='mgr.? 172.21.15.98:0/2332708165' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T05:23:33.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:33 smithi023 ceph-mon[31954]: from='mgr.? 172.21.15.98:0/2332708165' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T05:23:33.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:33 smithi023 ceph-mon[31954]: from='mgr.? 172.21.15.98:0/2332708165' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T05:23:33.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:33 smithi023 ceph-mon[31954]: from='mgr.? 172.21.15.98:0/2332708165' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T05:23:33.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:33.963+0000 7ff575ffb640 1 -- 172.21.15.23:0/1186100775 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mgrmap(e 21) v1 ==== 136975+0+0 (secure 0 0 0) 0x7ff5700b9dc0 con 0x7ff5781005c0 2024-03-28T05:23:33.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:33.963+0000 7ff575ffb640 1 -- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff5600a28f0 msgr2=0x7ff5600a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T05:23:33.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:33.963+0000 7ff575ffb640 1 --2- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:6800/222178108,v1:172.21.15.23:6801/222178108] conn(0x7ff5600a28f0 0x7ff5600a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:33.992 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:33 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:33.886+0000 7f051ae7d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T05:23:33.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:33 smithi113 ceph-mon[37164]: Standby manager daemon y restarted 2024-03-28T05:23:33.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:33 smithi113 ceph-mon[37164]: Standby manager daemon y started 2024-03-28T05:23:33.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:33 smithi113 ceph-mon[37164]: from='mgr.? 172.21.15.98:0/2332708165' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T05:23:33.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:33 smithi113 ceph-mon[37164]: from='mgr.? 172.21.15.98:0/2332708165' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T05:23:33.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:33 smithi113 ceph-mon[37164]: from='mgr.? 172.21.15.98:0/2332708165' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T05:23:33.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:33 smithi113 ceph-mon[37164]: from='mgr.? 172.21.15.98:0/2332708165' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T05:23:34.115 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:23:33 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y[37479]: 2024-03-28T05:23:33.624+0000 7f18d0b81200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T05:23:34.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:33 smithi098 ceph-mon[36087]: Standby manager daemon y restarted 2024-03-28T05:23:34.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:33 smithi098 ceph-mon[36087]: Standby manager daemon y started 2024-03-28T05:23:34.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:33 smithi098 ceph-mon[36087]: from='mgr.? 172.21.15.98:0/2332708165' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T05:23:34.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:33 smithi098 ceph-mon[36087]: from='mgr.? 172.21.15.98:0/2332708165' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T05:23:34.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:33 smithi098 ceph-mon[36087]: from='mgr.? 172.21.15.98:0/2332708165' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T05:23:34.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:33 smithi098 ceph-mon[36087]: from='mgr.? 172.21.15.98:0/2332708165' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T05:23:34.247 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:33 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:33.935+0000 7fb18b183200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T05:23:34.266 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:33 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:33.993+0000 7f051ae7d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T05:23:34.266 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z[39029]: 2024-03-28T05:23:34.227+0000 7f051ae7d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T05:23:34.531 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:34.247+0000 7fb0eab15700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T05:23:34.531 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:34.247+0000 7fb0eab15700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T05:23:34.531 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:34.247+0000 7fb0eab15700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T05:23:34.531 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:34.247+0000 7fb0eab15700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T05:23:34.531 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:34.247+0000 7fb0eab15700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T05:23:34.531 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:34.263+0000 7fb0edb1b700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T05:23:34.531 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:34.263+0000 7fb0edb1b700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T05:23:34.531 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:34.263+0000 7fb0edb1b700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T05:23:34.531 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:34.263+0000 7fb0edb1b700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T05:23:34.531 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:23:34.263+0000 7fb0edb1b700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T05:23:34.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:34.968+0000 7ff575ffb640 1 -- 172.21.15.23:0/1186100775 <== mon.0 v2:172.21.15.23:3300/0 9 ==== mgrmap(e 22) v1 ==== 137102+0+0 (secure 0 0 0) 0x7ff5700ee870 con 0x7ff5781005c0 2024-03-28T05:23:34.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:34.969+0000 7ff575ffb640 1 --2- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7ff560035170 0x7ff560035550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:34.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:34.969+0000 7ff575ffb640 1 -- 172.21.15.23:0/1186100775 --> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff570037e70 con 0x7ff560035170 2024-03-28T05:23:34.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:34.971+0000 7ff577fff640 1 --2- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7ff560035170 0x7ff560035550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:34.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:34.972+0000 7ff577fff640 1 --2- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7ff560035170 0x7ff560035550 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff578197d60 tx=0x7ff56c00b410 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:34.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-03-28T05:23:34.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: Active manager daemon x restarted 2024-03-28T05:23:34.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: Activating manager daemon x 2024-03-28T05:23:34.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T05:23:34.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: mgrmap e21: x(active, starting, since 0.0147257s), standbys: z, y 2024-03-28T05:23:34.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T05:23:34.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:23:34.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:23:34.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T05:23:34.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T05:23:34.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T05:23:34.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T05:23:34.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T05:23:34.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T05:23:34.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T05:23:34.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T05:23:34.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T05:23:34.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T05:23:34.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T05:23:34.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T05:23:34.995 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T05:23:34.995 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T05:23:34.995 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T05:23:34.995 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T05:23:34.995 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T05:23:34.995 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T05:23:34.995 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T05:23:34.995 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T05:23:34.995 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T05:23:34.995 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T05:23:34.995 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T05:23:34.995 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T05:23:34.996 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T05:23:34.996 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T05:23:34.996 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T05:23:34.996 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T05:23:34.996 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T05:23:34.996 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T05:23:34.996 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: Manager daemon x is now available 2024-03-28T05:23:34.996 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:23:34.996 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:23:34.996 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T05:23:34.996 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T05:23:34.996 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: Standby manager daemon z restarted 2024-03-28T05:23:34.996 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: Standby manager daemon z started 2024-03-28T05:23:34.996 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.? 172.21.15.113:0/3088043873' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T05:23:34.996 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.? 172.21.15.113:0/3088043873' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T05:23:34.996 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.? 172.21.15.113:0/3088043873' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T05:23:34.996 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:34 smithi113 ceph-mon[37164]: from='mgr.? 172.21.15.113:0/3088043873' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T05:23:35.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-03-28T05:23:35.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: Active manager daemon x restarted 2024-03-28T05:23:35.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: Activating manager daemon x 2024-03-28T05:23:35.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T05:23:35.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: mgrmap e21: x(active, starting, since 0.0147257s), standbys: z, y 2024-03-28T05:23:35.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T05:23:35.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:23:35.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:23:35.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T05:23:35.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T05:23:35.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T05:23:35.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T05:23:35.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T05:23:35.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T05:23:35.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T05:23:35.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T05:23:35.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T05:23:35.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T05:23:35.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T05:23:35.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T05:23:35.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T05:23:35.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T05:23:35.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T05:23:35.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T05:23:35.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T05:23:35.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T05:23:35.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T05:23:35.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T05:23:35.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T05:23:35.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T05:23:35.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T05:23:35.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T05:23:35.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T05:23:35.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T05:23:35.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T05:23:35.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T05:23:35.033 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T05:23:35.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T05:23:35.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: Manager daemon x is now available 2024-03-28T05:23:35.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:23:35.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:23:35.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T05:23:35.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T05:23:35.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: Standby manager daemon z restarted 2024-03-28T05:23:35.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: Standby manager daemon z started 2024-03-28T05:23:35.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.? 172.21.15.113:0/3088043873' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T05:23:35.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.? 172.21.15.113:0/3088043873' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T05:23:35.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.? 172.21.15.113:0/3088043873' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T05:23:35.034 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:34 smithi023 ceph-mon[31954]: from='mgr.? 172.21.15.113:0/3088043873' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T05:23:35.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.042+0000 7ff575ffb640 1 -- 172.21.15.23:0/1186100775 <== mgr.15402 v2:172.21.15.23:6800/1385031613 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ff570037e70 con 0x7ff560035170 2024-03-28T05:23:35.041 INFO:teuthology.orchestra.run.smithi023.stdout:Schedule set for path / 2024-03-28T05:23:35.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.044+0000 7ff57f259640 1 -- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7ff560035170 msgr2=0x7ff560035550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:35.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.044+0000 7ff57f259640 1 --2- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7ff560035170 0x7ff560035550 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff578197d60 tx=0x7ff56c00b410 comp rx=0 tx=0).stop 2024-03-28T05:23:35.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.044+0000 7ff57f259640 1 -- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5781005c0 msgr2=0x7ff578196850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:35.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.044+0000 7ff57f259640 1 --2- 172.21.15.23:0/1186100775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5781005c0 0x7ff578196850 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff5700324c0 tx=0x7ff570032f40 comp rx=0 tx=0).stop 2024-03-28T05:23:35.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.044+0000 7ff57d7f6640 1 -- 172.21.15.23:0/1186100775 reap_dead start 2024-03-28T05:23:35.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.044+0000 7ff57f259640 1 -- 172.21.15.23:0/1186100775 shutdown_connections 2024-03-28T05:23:35.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.044+0000 7ff57f259640 1 -- 172.21.15.23:0/1186100775 >> 172.21.15.23:0/1186100775 conn(0x7ff5780fbb30 msgr2=0x7ff578104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:35.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.044+0000 7ff57f259640 1 -- 172.21.15.23:0/1186100775 shutdown_connections 2024-03-28T05:23:35.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.044+0000 7ff57f259640 1 -- 172.21.15.23:0/1186100775 wait complete. 2024-03-28T05:23:35.055 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-28T05:23:35.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-03-28T05:23:35.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: Active manager daemon x restarted 2024-03-28T05:23:35.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: Activating manager daemon x 2024-03-28T05:23:35.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T05:23:35.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: mgrmap e21: x(active, starting, since 0.0147257s), standbys: z, y 2024-03-28T05:23:35.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T05:23:35.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T05:23:35.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T05:23:35.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T05:23:35.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T05:23:35.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T05:23:35.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T05:23:35.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T05:23:35.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T05:23:35.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T05:23:35.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T05:23:35.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T05:23:35.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T05:23:35.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T05:23:35.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T05:23:35.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T05:23:35.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T05:23:35.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T05:23:35.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T05:23:35.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T05:23:35.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T05:23:35.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T05:23:35.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T05:23:35.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T05:23:35.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T05:23:35.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T05:23:35.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T05:23:35.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T05:23:35.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T05:23:35.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T05:23:35.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T05:23:35.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T05:23:35.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T05:23:35.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: Manager daemon x is now available 2024-03-28T05:23:35.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:23:35.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:23:35.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T05:23:35.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T05:23:35.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: Standby manager daemon z restarted 2024-03-28T05:23:35.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: Standby manager daemon z started 2024-03-28T05:23:35.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.? 172.21.15.113:0/3088043873' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T05:23:35.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.? 172.21.15.113:0/3088043873' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T05:23:35.119 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.? 172.21.15.113:0/3088043873' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T05:23:35.119 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:34 smithi098 ceph-mon[36087]: from='mgr.? 172.21.15.113:0/3088043873' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T05:23:35.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.163+0000 7eff49590640 1 -- 172.21.15.23:0/3882791623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff441073c0 msgr2=0x7eff44109810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:35.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.163+0000 7eff49590640 1 --2- 172.21.15.23:0/3882791623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff441073c0 0x7eff44109810 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7eff3c009ca0 tx=0x7eff3c02c0f0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.163+0000 7eff49590640 1 -- 172.21.15.23:0/3882791623 shutdown_connections 2024-03-28T05:23:35.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.163+0000 7eff49590640 1 --2- 172.21.15.23:0/3882791623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff441073c0 0x7eff44109810 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.163+0000 7eff49590640 1 --2- 172.21.15.23:0/3882791623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff44104a90 0x7eff44106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.163+0000 7eff49590640 1 --2- 172.21.15.23:0/3882791623 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff44102160 0x7eff44104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.163+0000 7eff49590640 1 -- 172.21.15.23:0/3882791623 >> 172.21.15.23:0/3882791623 conn(0x7eff440fbb50 msgr2=0x7eff440fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:35.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.163+0000 7eff49590640 1 -- 172.21.15.23:0/3882791623 shutdown_connections 2024-03-28T05:23:35.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.163+0000 7eff49590640 1 -- 172.21.15.23:0/3882791623 wait complete. 2024-03-28T05:23:35.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.164+0000 7eff49590640 1 Processor -- start 2024-03-28T05:23:35.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.164+0000 7eff49590640 1 -- start start 2024-03-28T05:23:35.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.164+0000 7eff49590640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff44102160 0x7eff441978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:35.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.164+0000 7eff49590640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff44104a90 0x7eff44197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:35.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.164+0000 7eff49590640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff441073c0 0x7eff4419ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:35.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.164+0000 7eff49590640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4410d380 con 0x7eff44102160 2024-03-28T05:23:35.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.164+0000 7eff49590640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4410d200 con 0x7eff44104a90 2024-03-28T05:23:35.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.164+0000 7eff49590640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4410d500 con 0x7eff441073c0 2024-03-28T05:23:35.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.164+0000 7eff42ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff44102160 0x7eff441978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:35.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.164+0000 7eff42ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff44102160 0x7eff441978e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48944/0 (socket says 172.21.15.23:48944) 2024-03-28T05:23:35.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.164+0000 7eff42ffd640 1 -- 172.21.15.23:0/576059078 learned_addr learned my addr 172.21.15.23:0/576059078 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:35.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.164+0000 7eff3bfff640 1 --2- 172.21.15.23:0/576059078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff44104a90 0x7eff44197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:35.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.164+0000 7eff437fe640 1 --2- 172.21.15.23:0/576059078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff441073c0 0x7eff4419ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:35.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.164+0000 7eff42ffd640 1 -- 172.21.15.23:0/576059078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff441073c0 msgr2=0x7eff4419ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:35.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.164+0000 7eff42ffd640 1 --2- 172.21.15.23:0/576059078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff441073c0 0x7eff4419ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.164+0000 7eff42ffd640 1 -- 172.21.15.23:0/576059078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff44104a90 msgr2=0x7eff44197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:35.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.164+0000 7eff42ffd640 1 --2- 172.21.15.23:0/576059078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff44104a90 0x7eff44197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.164+0000 7eff42ffd640 1 -- 172.21.15.23:0/576059078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff4419f500 con 0x7eff44102160 2024-03-28T05:23:35.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.164+0000 7eff3bfff640 1 --2- 172.21.15.23:0/576059078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff44104a90 0x7eff44197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:23:35.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.165+0000 7eff42ffd640 1 --2- 172.21.15.23:0/576059078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff44102160 0x7eff441978e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7eff3400e9f0 tx=0x7eff3400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:35.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.165+0000 7eff40ff9640 1 -- 172.21.15.23:0/576059078 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3400cd80 con 0x7eff44102160 2024-03-28T05:23:35.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.165+0000 7eff40ff9640 1 -- 172.21.15.23:0/576059078 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff34005500 con 0x7eff44102160 2024-03-28T05:23:35.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.165+0000 7eff40ff9640 1 -- 172.21.15.23:0/576059078 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff34010690 con 0x7eff44102160 2024-03-28T05:23:35.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.165+0000 7eff49590640 1 -- 172.21.15.23:0/576059078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff4419f7f0 con 0x7eff44102160 2024-03-28T05:23:35.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.165+0000 7eff49590640 1 -- 172.21.15.23:0/576059078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff4419fcf0 con 0x7eff44102160 2024-03-28T05:23:35.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.166+0000 7eff49590640 1 -- 172.21.15.23:0/576059078 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff440fd640 con 0x7eff44102160 2024-03-28T05:23:35.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.167+0000 7eff40ff9640 1 -- 172.21.15.23:0/576059078 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 22) v1 ==== 137102+0+0 (secure 0 0 0) 0x7eff340046c0 con 0x7eff44102160 2024-03-28T05:23:35.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.167+0000 7eff40ff9640 1 --2- 172.21.15.23:0/576059078 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7eff300a26b0 0x7eff300a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:35.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.168+0000 7eff3bfff640 1 --2- 172.21.15.23:0/576059078 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7eff300a26b0 0x7eff300a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:35.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.168+0000 7eff40ff9640 1 -- 172.21.15.23:0/576059078 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7eff340cef00 con 0x7eff44102160 2024-03-28T05:23:35.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.168+0000 7eff3bfff640 1 --2- 172.21.15.23:0/576059078 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7eff300a26b0 0x7eff300a4b70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7eff44198df0 tx=0x7eff2c005f90 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:35.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.172+0000 7eff40ff9640 1 -- 172.21.15.23:0/576059078 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7eff3409bde0 con 0x7eff44102160 2024-03-28T05:23:35.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.381+0000 7eff49590640 1 -- 172.21.15.23:0/576059078 --> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) v1 -- 0x7eff44061b80 con 0x7eff300a26b0 2024-03-28T05:23:35.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.440+0000 7eff40ff9640 1 -- 172.21.15.23:0/576059078 <== mgr.15402 v2:172.21.15.23:6800/1385031613 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7eff44061b80 con 0x7eff300a26b0 2024-03-28T05:23:35.440 INFO:teuthology.orchestra.run.smithi023.stdout:Retention added to path / 2024-03-28T05:23:35.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.442+0000 7eff49590640 1 -- 172.21.15.23:0/576059078 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7eff300a26b0 msgr2=0x7eff300a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:35.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.442+0000 7eff49590640 1 --2- 172.21.15.23:0/576059078 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7eff300a26b0 0x7eff300a4b70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7eff44198df0 tx=0x7eff2c005f90 comp rx=0 tx=0).stop 2024-03-28T05:23:35.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.442+0000 7eff49590640 1 -- 172.21.15.23:0/576059078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff44102160 msgr2=0x7eff441978e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:35.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.442+0000 7eff49590640 1 --2- 172.21.15.23:0/576059078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff44102160 0x7eff441978e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7eff3400e9f0 tx=0x7eff3400edf0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.443+0000 7eff49590640 1 -- 172.21.15.23:0/576059078 shutdown_connections 2024-03-28T05:23:35.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.443+0000 7eff49590640 1 --2- 172.21.15.23:0/576059078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff441073c0 0x7eff4419ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.443+0000 7eff49590640 1 --2- 172.21.15.23:0/576059078 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7eff300a26b0 0x7eff300a4b70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.443+0000 7eff49590640 1 --2- 172.21.15.23:0/576059078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff44104a90 0x7eff44197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.443+0000 7eff49590640 1 --2- 172.21.15.23:0/576059078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff44102160 0x7eff441978e0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.443+0000 7eff49590640 1 -- 172.21.15.23:0/576059078 >> 172.21.15.23:0/576059078 conn(0x7eff440fbb50 msgr2=0x7eff44102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:35.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.443+0000 7eff49590640 1 -- 172.21.15.23:0/576059078 shutdown_connections 2024-03-28T05:23:35.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.443+0000 7eff49590640 1 -- 172.21.15.23:0/576059078 wait complete. 2024-03-28T05:23:35.453 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-28T05:23:35.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.558+0000 7f4997f53640 1 -- 172.21.15.23:0/1093470471 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4990104ad0 msgr2=0x7f4990106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:35.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.558+0000 7f4997f53640 1 --2- 172.21.15.23:0/1093470471 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4990104ad0 0x7f4990106ec0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4980009900 tx=0x7f498002bc30 comp rx=0 tx=0).stop 2024-03-28T05:23:35.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.558+0000 7f4997f53640 1 -- 172.21.15.23:0/1093470471 shutdown_connections 2024-03-28T05:23:35.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.558+0000 7f4997f53640 1 --2- 172.21.15.23:0/1093470471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4990107400 0x7f49901097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.558+0000 7f4997f53640 1 --2- 172.21.15.23:0/1093470471 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4990104ad0 0x7f4990106ec0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.558+0000 7f4997f53640 1 --2- 172.21.15.23:0/1093470471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49901021a0 0x7f4990104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.558+0000 7f4997f53640 1 -- 172.21.15.23:0/1093470471 >> 172.21.15.23:0/1093470471 conn(0x7f49900fbb30 msgr2=0x7f49900fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:35.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.558+0000 7f4997f53640 1 -- 172.21.15.23:0/1093470471 shutdown_connections 2024-03-28T05:23:35.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.558+0000 7f4997f53640 1 -- 172.21.15.23:0/1093470471 wait complete. 2024-03-28T05:23:35.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.558+0000 7f4997f53640 1 Processor -- start 2024-03-28T05:23:35.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.558+0000 7f4997f53640 1 -- start start 2024-03-28T05:23:35.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.558+0000 7f4997f53640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49901021a0 0x7f4990197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:35.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.558+0000 7f4997f53640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4990104ad0 0x7f4990197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:35.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.558+0000 7f4997f53640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4990107400 0x7f499019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:35.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.558+0000 7f4997f53640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499010d310 con 0x7f4990104ad0 2024-03-28T05:23:35.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.558+0000 7f4997f53640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499010d190 con 0x7f49901021a0 2024-03-28T05:23:35.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.558+0000 7f4997f53640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499010d490 con 0x7f4990107400 2024-03-28T05:23:35.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.559+0000 7f49954ee640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4990104ad0 0x7f4990197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:35.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.559+0000 7f49954ee640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4990104ad0 0x7f4990197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48968/0 (socket says 172.21.15.23:48968) 2024-03-28T05:23:35.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.559+0000 7f49954ee640 1 -- 172.21.15.23:0/2660842180 learned_addr learned my addr 172.21.15.23:0/2660842180 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:35.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.559+0000 7f49964f0640 1 --2- 172.21.15.23:0/2660842180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4990107400 0x7f499019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:35.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.559+0000 7f4995cef640 1 --2- 172.21.15.23:0/2660842180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49901021a0 0x7f4990197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:35.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.559+0000 7f49954ee640 1 -- 172.21.15.23:0/2660842180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4990107400 msgr2=0x7f499019ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:35.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.559+0000 7f49954ee640 1 --2- 172.21.15.23:0/2660842180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4990107400 0x7f499019ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.559+0000 7f49954ee640 1 -- 172.21.15.23:0/2660842180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49901021a0 msgr2=0x7f4990197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:35.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.559+0000 7f49954ee640 1 --2- 172.21.15.23:0/2660842180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49901021a0 0x7f4990197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.559+0000 7f49954ee640 1 -- 172.21.15.23:0/2660842180 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499019f5a0 con 0x7f4990104ad0 2024-03-28T05:23:35.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.559+0000 7f49954ee640 1 --2- 172.21.15.23:0/2660842180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4990104ad0 0x7f4990197ec0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4980000c00 tx=0x7f4980005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:35.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.559+0000 7f4986ffd640 1 -- 172.21.15.23:0/2660842180 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4980041070 con 0x7f4990104ad0 2024-03-28T05:23:35.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.559+0000 7f4997f53640 1 -- 172.21.15.23:0/2660842180 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f499019f830 con 0x7f4990104ad0 2024-03-28T05:23:35.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.559+0000 7f4986ffd640 1 -- 172.21.15.23:0/2660842180 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49800044d0 con 0x7f4990104ad0 2024-03-28T05:23:35.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.559+0000 7f4986ffd640 1 -- 172.21.15.23:0/2660842180 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4980034440 con 0x7f4990104ad0 2024-03-28T05:23:35.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.559+0000 7f4997f53640 1 -- 172.21.15.23:0/2660842180 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f499019fd70 con 0x7f4990104ad0 2024-03-28T05:23:35.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.560+0000 7f4997f53640 1 -- 172.21.15.23:0/2660842180 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49900fd620 con 0x7f4990104ad0 2024-03-28T05:23:35.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.561+0000 7f4986ffd640 1 -- 172.21.15.23:0/2660842180 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 22) v1 ==== 137102+0+0 (secure 0 0 0) 0x7f4980044070 con 0x7f4990104ad0 2024-03-28T05:23:35.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.562+0000 7f4986ffd640 1 --2- 172.21.15.23:0/2660842180 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f497c0a26b0 0x7f497c0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:35.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.562+0000 7f4995cef640 1 --2- 172.21.15.23:0/2660842180 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f497c0a26b0 0x7f497c0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:35.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.562+0000 7f4986ffd640 1 -- 172.21.15.23:0/2660842180 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f49800f08d0 con 0x7f4990104ad0 2024-03-28T05:23:35.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.563+0000 7f4995cef640 1 --2- 172.21.15.23:0/2660842180 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f497c0a26b0 0x7f497c0a4b70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f498c005d00 tx=0x7f498c005c50 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:35.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.566+0000 7f4986ffd640 1 -- 172.21.15.23:0/2660842180 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f49800bd7b0 con 0x7f4990104ad0 2024-03-28T05:23:35.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.774+0000 7f4997f53640 1 -- 172.21.15.23:0/2660842180 --> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f4990061ae0 con 0x7f497c0a26b0 2024-03-28T05:23:35.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.782+0000 7f4986ffd640 1 -- 172.21.15.23:0/2660842180 <== mgr.15402 v2:172.21.15.23:6800/1385031613 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+298 (secure 0 0 0) 0x7f4990061ae0 con 0x7f497c0a26b0 2024-03-28T05:23:35.781 INFO:teuthology.orchestra.run.smithi023.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-28T00:00:00", "created": "2024-03-28T05:23:34", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-28T05:23:35.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.784+0000 7f4997f53640 1 -- 172.21.15.23:0/2660842180 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f497c0a26b0 msgr2=0x7f497c0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:35.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.784+0000 7f4997f53640 1 --2- 172.21.15.23:0/2660842180 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f497c0a26b0 0x7f497c0a4b70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f498c005d00 tx=0x7f498c005c50 comp rx=0 tx=0).stop 2024-03-28T05:23:35.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.784+0000 7f4997f53640 1 -- 172.21.15.23:0/2660842180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4990104ad0 msgr2=0x7f4990197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:35.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.784+0000 7f4997f53640 1 --2- 172.21.15.23:0/2660842180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4990104ad0 0x7f4990197ec0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4980000c00 tx=0x7f4980005d50 comp rx=0 tx=0).stop 2024-03-28T05:23:35.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.784+0000 7f4997f53640 1 -- 172.21.15.23:0/2660842180 shutdown_connections 2024-03-28T05:23:35.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.784+0000 7f4997f53640 1 --2- 172.21.15.23:0/2660842180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4990107400 0x7f499019ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.784+0000 7f4997f53640 1 --2- 172.21.15.23:0/2660842180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4990104ad0 0x7f4990197ec0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.784+0000 7f4997f53640 1 --2- 172.21.15.23:0/2660842180 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f497c0a26b0 0x7f497c0a4b70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.784+0000 7f4997f53640 1 --2- 172.21.15.23:0/2660842180 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49901021a0 0x7f4990197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.784+0000 7f4997f53640 1 -- 172.21.15.23:0/2660842180 >> 172.21.15.23:0/2660842180 conn(0x7f49900fbb30 msgr2=0x7f4990102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:35.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.784+0000 7f4997f53640 1 -- 172.21.15.23:0/2660842180 shutdown_connections 2024-03-28T05:23:35.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.784+0000 7f4997f53640 1 -- 172.21.15.23:0/2660842180 wait complete. 2024-03-28T05:23:35.795 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-28T05:23:35.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.906+0000 7fdf23722640 1 -- 172.21.15.23:0/3182075686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf1c1028e0 msgr2=0x7fdf1c102d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:35.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.906+0000 7fdf23722640 1 --2- 172.21.15.23:0/3182075686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf1c1028e0 0x7fdf1c102d60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c009900 tx=0x7fdf0c02bc30 comp rx=0 tx=0).stop 2024-03-28T05:23:35.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.906+0000 7fdf23722640 1 -- 172.21.15.23:0/3182075686 shutdown_connections 2024-03-28T05:23:35.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.906+0000 7fdf23722640 1 --2- 172.21.15.23:0/3182075686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf1c1032a0 0x7fdf1c109b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.906+0000 7fdf23722640 1 --2- 172.21.15.23:0/3182075686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf1c1028e0 0x7fdf1c102d60 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.906+0000 7fdf23722640 1 --2- 172.21.15.23:0/3182075686 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf1c1016f0 0x7fdf1c101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.906+0000 7fdf23722640 1 -- 172.21.15.23:0/3182075686 >> 172.21.15.23:0/3182075686 conn(0x7fdf1c0fcc60 msgr2=0x7fdf1c0ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:35.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.906+0000 7fdf23722640 1 -- 172.21.15.23:0/3182075686 shutdown_connections 2024-03-28T05:23:35.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.906+0000 7fdf23722640 1 -- 172.21.15.23:0/3182075686 wait complete. 2024-03-28T05:23:35.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.907+0000 7fdf23722640 1 Processor -- start 2024-03-28T05:23:35.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.907+0000 7fdf23722640 1 -- start start 2024-03-28T05:23:35.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.907+0000 7fdf23722640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf1c1016f0 0x7fdf1c1979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:35.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.907+0000 7fdf23722640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf1c1028e0 0x7fdf1c197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:35.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.907+0000 7fdf23722640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf1c1032a0 0x7fdf1c19ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:35.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.907+0000 7fdf214be640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf1c1016f0 0x7fdf1c1979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:35.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.907+0000 7fdf214be640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf1c1016f0 0x7fdf1c1979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48990/0 (socket says 172.21.15.23:48990) 2024-03-28T05:23:35.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.907+0000 7fdf214be640 1 -- 172.21.15.23:0/3607287484 learned_addr learned my addr 172.21.15.23:0/3607287484 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:35.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.907+0000 7fdf23722640 1 -- 172.21.15.23:0/3607287484 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c10d370 con 0x7fdf1c1016f0 2024-03-28T05:23:35.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.907+0000 7fdf23722640 1 -- 172.21.15.23:0/3607287484 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c10d1f0 con 0x7fdf1c1028e0 2024-03-28T05:23:35.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.907+0000 7fdf23722640 1 -- 172.21.15.23:0/3607287484 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c10d4f0 con 0x7fdf1c1032a0 2024-03-28T05:23:35.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.907+0000 7fdf21cbf640 1 --2- 172.21.15.23:0/3607287484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf1c1032a0 0x7fdf1c19ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:35.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.907+0000 7fdf20cbd640 1 --2- 172.21.15.23:0/3607287484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf1c1028e0 0x7fdf1c197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:35.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.908+0000 7fdf214be640 1 -- 172.21.15.23:0/3607287484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf1c1032a0 msgr2=0x7fdf1c19ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:35.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.908+0000 7fdf214be640 1 --2- 172.21.15.23:0/3607287484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf1c1032a0 0x7fdf1c19ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.908+0000 7fdf214be640 1 -- 172.21.15.23:0/3607287484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf1c1028e0 msgr2=0x7fdf1c197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:35.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.908+0000 7fdf214be640 1 --2- 172.21.15.23:0/3607287484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf1c1028e0 0x7fdf1c197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:35.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.908+0000 7fdf214be640 1 -- 172.21.15.23:0/3607287484 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1c19f580 con 0x7fdf1c1016f0 2024-03-28T05:23:35.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.908+0000 7fdf214be640 1 --2- 172.21.15.23:0/3607287484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf1c1016f0 0x7fdf1c1979b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fdf1800e9c0 tx=0x7fdf1800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:35.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.908+0000 7fdf127fc640 1 -- 172.21.15.23:0/3607287484 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1800cd80 con 0x7fdf1c1016f0 2024-03-28T05:23:35.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.908+0000 7fdf23722640 1 -- 172.21.15.23:0/3607287484 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf1c19f870 con 0x7fdf1c1016f0 2024-03-28T05:23:35.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.908+0000 7fdf127fc640 1 -- 172.21.15.23:0/3607287484 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdf180035f0 con 0x7fdf1c1016f0 2024-03-28T05:23:35.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.908+0000 7fdf127fc640 1 -- 172.21.15.23:0/3607287484 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf18010690 con 0x7fdf1c1016f0 2024-03-28T05:23:35.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.908+0000 7fdf23722640 1 -- 172.21.15.23:0/3607287484 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf1c19fd40 con 0x7fdf1c1016f0 2024-03-28T05:23:35.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.909+0000 7fdf23722640 1 -- 172.21.15.23:0/3607287484 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf1c108a10 con 0x7fdf1c1016f0 2024-03-28T05:23:35.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.910+0000 7fdf127fc640 1 -- 172.21.15.23:0/3607287484 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 22) v1 ==== 137102+0+0 (secure 0 0 0) 0x7fdf180037a0 con 0x7fdf1c1016f0 2024-03-28T05:23:35.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.911+0000 7fdf127fc640 1 --2- 172.21.15.23:0/3607287484 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fdf080a26b0 0x7fdf080a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:35.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.911+0000 7fdf127fc640 1 -- 172.21.15.23:0/3607287484 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fdf180cef40 con 0x7fdf1c1016f0 2024-03-28T05:23:35.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.911+0000 7fdf20cbd640 1 --2- 172.21.15.23:0/3607287484 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fdf080a26b0 0x7fdf080a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:35.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.912+0000 7fdf20cbd640 1 --2- 172.21.15.23:0/3607287484 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fdf080a26b0 0x7fdf080a4b70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c02c7b0 tx=0x7fdf0c005fd0 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:35.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:35.915+0000 7fdf127fc640 1 -- 172.21.15.23:0/3607287484 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdf1809be20 con 0x7fdf1c1016f0 2024-03-28T05:23:35.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:35 smithi113 ceph-mon[37164]: [28/Mar/2024:05:23:34] ENGINE Bus STARTING 2024-03-28T05:23:35.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:35 smithi113 ceph-mon[37164]: [28/Mar/2024:05:23:34] ENGINE Serving on https://172.21.15.23:7150 2024-03-28T05:23:35.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:35 smithi113 ceph-mon[37164]: [28/Mar/2024:05:23:34] ENGINE Serving on http://172.21.15.23:8765 2024-03-28T05:23:35.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:35 smithi113 ceph-mon[37164]: [28/Mar/2024:05:23:34] ENGINE Bus STARTED 2024-03-28T05:23:35.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:35 smithi113 ceph-mon[37164]: mgrmap e22: x(active, since 1.02322s), standbys: y, z 2024-03-28T05:23:36.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:35 smithi023 ceph-mon[31954]: [28/Mar/2024:05:23:34] ENGINE Bus STARTING 2024-03-28T05:23:36.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:35 smithi023 ceph-mon[31954]: [28/Mar/2024:05:23:34] ENGINE Serving on https://172.21.15.23:7150 2024-03-28T05:23:36.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:35 smithi023 ceph-mon[31954]: [28/Mar/2024:05:23:34] ENGINE Serving on http://172.21.15.23:8765 2024-03-28T05:23:36.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:35 smithi023 ceph-mon[31954]: [28/Mar/2024:05:23:34] ENGINE Bus STARTED 2024-03-28T05:23:36.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:35 smithi023 ceph-mon[31954]: mgrmap e22: x(active, since 1.02322s), standbys: y, z 2024-03-28T05:23:36.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:35 smithi098 ceph-mon[36087]: [28/Mar/2024:05:23:34] ENGINE Bus STARTING 2024-03-28T05:23:36.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:35 smithi098 ceph-mon[36087]: [28/Mar/2024:05:23:34] ENGINE Serving on https://172.21.15.23:7150 2024-03-28T05:23:36.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:35 smithi098 ceph-mon[36087]: [28/Mar/2024:05:23:34] ENGINE Serving on http://172.21.15.23:8765 2024-03-28T05:23:36.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:35 smithi098 ceph-mon[36087]: [28/Mar/2024:05:23:34] ENGINE Bus STARTED 2024-03-28T05:23:36.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:35 smithi098 ceph-mon[36087]: mgrmap e22: x(active, since 1.02322s), standbys: y, z 2024-03-28T05:23:36.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:36.167+0000 7fdf23722640 1 -- 172.21.15.23:0/3607287484 --> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fdf1c061ae0 con 0x7fdf080a26b0 2024-03-28T05:23:36.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:36.183+0000 7fdf127fc640 1 -- 172.21.15.23:0/3607287484 <== mgr.15402 v2:172.21.15.23:6800/1385031613 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7fdf1c061ae0 con 0x7fdf080a26b0 2024-03-28T05:23:36.183 INFO:teuthology.orchestra.run.smithi023.stdout:/ 1m 6m3h 2024-03-28T05:23:36.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:36.186+0000 7fdf23722640 1 -- 172.21.15.23:0/3607287484 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fdf080a26b0 msgr2=0x7fdf080a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:36.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:36.186+0000 7fdf23722640 1 --2- 172.21.15.23:0/3607287484 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fdf080a26b0 0x7fdf080a4b70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c02c7b0 tx=0x7fdf0c005fd0 comp rx=0 tx=0).stop 2024-03-28T05:23:36.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:36.186+0000 7fdf23722640 1 -- 172.21.15.23:0/3607287484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf1c1016f0 msgr2=0x7fdf1c1979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:36.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:36.186+0000 7fdf23722640 1 --2- 172.21.15.23:0/3607287484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf1c1016f0 0x7fdf1c1979b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fdf1800e9c0 tx=0x7fdf1800ee80 comp rx=0 tx=0).stop 2024-03-28T05:23:36.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:36.186+0000 7fdf23722640 1 -- 172.21.15.23:0/3607287484 shutdown_connections 2024-03-28T05:23:36.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:36.186+0000 7fdf23722640 1 --2- 172.21.15.23:0/3607287484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf1c1032a0 0x7fdf1c19ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:36.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:36.186+0000 7fdf23722640 1 --2- 172.21.15.23:0/3607287484 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fdf080a26b0 0x7fdf080a4b70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:36.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:36.186+0000 7fdf23722640 1 --2- 172.21.15.23:0/3607287484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf1c1028e0 0x7fdf1c197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:36.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:36.186+0000 7fdf23722640 1 --2- 172.21.15.23:0/3607287484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf1c1016f0 0x7fdf1c1979b0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:36.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:36.186+0000 7fdf23722640 1 -- 172.21.15.23:0/3607287484 >> 172.21.15.23:0/3607287484 conn(0x7fdf1c0fcc60 msgr2=0x7fdf1c105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:36.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:36.186+0000 7fdf23722640 1 -- 172.21.15.23:0/3607287484 shutdown_connections 2024-03-28T05:23:36.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:23:36.186+0000 7fdf23722640 1 -- 172.21.15.23:0/3607287484 wait complete. 2024-03-28T05:23:36.197 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-28T05:23:36.227 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-28T05:23:36.237 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-28T05:23:36.266 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-28T05:23:36.266 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-28T05:23:36.269 INFO:teuthology.run_tasks:Running task workunit... 2024-03-28T05:23:36.280 INFO:tasks.workunit:Pulling workunits from ref 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T05:23:36.281 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-28T05:23:36.281 INFO:tasks.workunit:timeout=3h 2024-03-28T05:23:36.281 INFO:tasks.workunit:cleanup=True 2024-03-28T05:23:36.281 DEBUG:teuthology.orchestra.run.smithi023:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-28T05:23:36.298 INFO:teuthology.orchestra.run.smithi023.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-28T05:23:36.298 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-28T05:23:36.298 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-03-28T05:23:36.298 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T05:23:36.298 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-28T05:23:36.298 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-28 05:21:34.802180983 +0000 2024-03-28T05:23:36.298 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-28 05:21:34.802180983 +0000 2024-03-28T05:23:36.298 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-28 05:23:25.367909053 +0000 2024-03-28T05:23:36.298 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-28 05:21:34.802180983 +0000 2024-03-28T05:23:36.299 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-28T05:23:36.299 DEBUG:teuthology.orchestra.run.smithi023:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-28T05:23:36.370 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T05:23:36.425 INFO:tasks.workunit.client.0.smithi023.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-28T05:23:36.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:36 smithi113 ceph-mon[37164]: from='client.15420 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:36.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:36 smithi113 ceph-mon[37164]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:23:36.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:36 smithi113 ceph-mon[37164]: from='client.15462 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:36.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:36 smithi113 ceph-mon[37164]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:36.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:36 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:36.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:36 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:36.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:36 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:36.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:36 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:36.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:36 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:36.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:36 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:37.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:36 smithi023 ceph-mon[31954]: from='client.15420 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:37.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:36 smithi023 ceph-mon[31954]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:23:37.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:36 smithi023 ceph-mon[31954]: from='client.15462 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:37.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:36 smithi023 ceph-mon[31954]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:37.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:36 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:37.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:36 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:37.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:36 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:37.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:36 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:37.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:36 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:37.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:36 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:37.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:36 smithi098 ceph-mon[36087]: from='client.15420 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:37.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:36 smithi098 ceph-mon[36087]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:23:37.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:36 smithi098 ceph-mon[36087]: from='client.15462 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:37.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:36 smithi098 ceph-mon[36087]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:37.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:36 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:37.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:36 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:37.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:36 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:37.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:36 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:37.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:36 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:37.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:36 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:37.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:23:37.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:37.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: mgrmap e23: x(active, since 3s), standbys: y, z 2024-03-28T05:23:37.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:37.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:37.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:37.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:37.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:37.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:37.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:37.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:37.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:37.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:37.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:37.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:37.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:37.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:37.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:37.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:37.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:37.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:37.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:23:37.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:37 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:23:38.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:23:38.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:38.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: mgrmap e23: x(active, since 3s), standbys: y, z 2024-03-28T05:23:38.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:38.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:38.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:38.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:38.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:38.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:38.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:23:38.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:37 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:23:38.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:23:38.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:23:38.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: mgrmap e23: x(active, since 3s), standbys: y, z 2024-03-28T05:23:38.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:38.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:23:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:23:38.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:37 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:23:38.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: Adjusting osd_memory_target on smithi098 to 174.1M 2024-03-28T05:23:38.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: Unable to set osd_memory_target on smithi098 to 182633472: error parsing value: Value '182633472' is below minimum 939524096 2024-03-28T05:23:38.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: Adjusting osd_memory_target on smithi113 to 174.1M 2024-03-28T05:23:38.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: Unable to set osd_memory_target on smithi113 to 182619852: error parsing value: Value '182619852' is below minimum 939524096 2024-03-28T05:23:38.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: Adjusting osd_memory_target on smithi023 to 176.2M 2024-03-28T05:23:38.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: Unable to set osd_memory_target on smithi023 to 184790323: error parsing value: Value '184790323' is below minimum 939524096 2024-03-28T05:23:38.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-28T05:23:38.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-28T05:23:38.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: Updating smithi113:/etc/ceph/ceph.conf 2024-03-28T05:23:38.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: Updating smithi023:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:23:38.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: Updating smithi098:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:23:38.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: Updating smithi113:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:23:38.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:38.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:38.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:38.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:38.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:38.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:38.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:38.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:23:38.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:23:38.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:23:38.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:38 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:39.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: Adjusting osd_memory_target on smithi098 to 174.1M 2024-03-28T05:23:39.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: Unable to set osd_memory_target on smithi098 to 182633472: error parsing value: Value '182633472' is below minimum 939524096 2024-03-28T05:23:39.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: Adjusting osd_memory_target on smithi113 to 174.1M 2024-03-28T05:23:39.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: Unable to set osd_memory_target on smithi113 to 182619852: error parsing value: Value '182619852' is below minimum 939524096 2024-03-28T05:23:39.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: Adjusting osd_memory_target on smithi023 to 176.2M 2024-03-28T05:23:39.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: Unable to set osd_memory_target on smithi023 to 184790323: error parsing value: Value '184790323' is below minimum 939524096 2024-03-28T05:23:39.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-28T05:23:39.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-28T05:23:39.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: Updating smithi113:/etc/ceph/ceph.conf 2024-03-28T05:23:39.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: Updating smithi023:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:23:39.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: Updating smithi098:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:23:39.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: Updating smithi113:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:23:39.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:39.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:39.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:39.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:39.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:39.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:39.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:39.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:23:39.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:23:39.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:23:39.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:38 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:39.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: Adjusting osd_memory_target on smithi098 to 174.1M 2024-03-28T05:23:39.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: Unable to set osd_memory_target on smithi098 to 182633472: error parsing value: Value '182633472' is below minimum 939524096 2024-03-28T05:23:39.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: Adjusting osd_memory_target on smithi113 to 174.1M 2024-03-28T05:23:39.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: Unable to set osd_memory_target on smithi113 to 182619852: error parsing value: Value '182619852' is below minimum 939524096 2024-03-28T05:23:39.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: Adjusting osd_memory_target on smithi023 to 176.2M 2024-03-28T05:23:39.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: Unable to set osd_memory_target on smithi023 to 184790323: error parsing value: Value '184790323' is below minimum 939524096 2024-03-28T05:23:39.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-28T05:23:39.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-28T05:23:39.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: Updating smithi113:/etc/ceph/ceph.conf 2024-03-28T05:23:39.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: Updating smithi023:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:23:39.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: Updating smithi098:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:23:39.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: Updating smithi113:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T05:23:39.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:39.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:39.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:39.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:39.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:39.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:39.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:39.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:23:39.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:23:39.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:23:39.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:38 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:23:39.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:39 smithi113 ceph-mon[37164]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:23:39.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:39 smithi113 ceph-mon[37164]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-28T05:23:39.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:39 smithi113 ceph-mon[37164]: Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-03-28T05:23:39.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:39 smithi113 ceph-mon[37164]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-28T05:23:39.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:39 smithi113 ceph-mon[37164]: Updating smithi023:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T05:23:39.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:39 smithi113 ceph-mon[37164]: Updating smithi113:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T05:23:39.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:39 smithi113 ceph-mon[37164]: Updating smithi098:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T05:23:39.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:39 smithi113 ceph-mon[37164]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-03-28T05:23:40.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:39 smithi023 ceph-mon[31954]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:23:40.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:39 smithi023 ceph-mon[31954]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-28T05:23:40.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:39 smithi023 ceph-mon[31954]: Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-03-28T05:23:40.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:39 smithi023 ceph-mon[31954]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-28T05:23:40.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:39 smithi023 ceph-mon[31954]: Updating smithi023:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T05:23:40.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:39 smithi023 ceph-mon[31954]: Updating smithi113:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T05:23:40.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:39 smithi023 ceph-mon[31954]: Updating smithi098:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T05:23:40.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:39 smithi023 ceph-mon[31954]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-03-28T05:23:40.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:39 smithi098 ceph-mon[36087]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:23:40.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:39 smithi098 ceph-mon[36087]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-28T05:23:40.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:39 smithi098 ceph-mon[36087]: Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-03-28T05:23:40.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:39 smithi098 ceph-mon[36087]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-28T05:23:40.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:39 smithi098 ceph-mon[36087]: Updating smithi023:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T05:23:40.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:39 smithi098 ceph-mon[36087]: Updating smithi113:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T05:23:40.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:39 smithi098 ceph-mon[36087]: Updating smithi098:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T05:23:40.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:39 smithi098 ceph-mon[36087]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-03-28T05:23:41.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:41 smithi113 ceph-mon[37164]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:23:42.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:41 smithi023 ceph-mon[31954]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:23:42.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:41 smithi098 ceph-mon[36087]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:23:43.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:43 smithi113 ceph-mon[37164]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 132 KiB/s wr, 73 op/s 2024-03-28T05:23:44.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:43 smithi023 ceph-mon[31954]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 132 KiB/s wr, 73 op/s 2024-03-28T05:23:44.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:43 smithi098 ceph-mon[36087]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 132 KiB/s wr, 73 op/s 2024-03-28T05:23:45.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:45 smithi113 ceph-mon[37164]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 103 KiB/s wr, 57 op/s 2024-03-28T05:23:46.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:45 smithi023 ceph-mon[31954]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 103 KiB/s wr, 57 op/s 2024-03-28T05:23:46.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:45 smithi098 ceph-mon[36087]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 103 KiB/s wr, 57 op/s 2024-03-28T05:23:48.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:47 smithi023 ceph-mon[31954]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 84 KiB/s wr, 47 op/s 2024-03-28T05:23:48.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:47 smithi098 ceph-mon[36087]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 84 KiB/s wr, 47 op/s 2024-03-28T05:23:48.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:47 smithi113 ceph-mon[37164]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 84 KiB/s wr, 47 op/s 2024-03-28T05:23:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.413+0000 7f77db685640 1 -- 172.21.15.23:0/338486228 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77d40ffb40 msgr2=0x7f77d40fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.413+0000 7f77db685640 1 --2- 172.21.15.23:0/338486228 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77d40ffb40 0x7f77d40fffc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f77c4009930 tx=0x7f77c402bc30 comp rx=0 tx=0).stop 2024-03-28T05:23:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.414+0000 7f77db685640 1 -- 172.21.15.23:0/338486228 shutdown_connections 2024-03-28T05:23:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.414+0000 7f77db685640 1 --2- 172.21.15.23:0/338486228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77d4100500 0x7f77d410e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.414+0000 7f77db685640 1 --2- 172.21.15.23:0/338486228 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77d40ffb40 0x7f77d40fffc0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.414+0000 7f77db685640 1 --2- 172.21.15.23:0/338486228 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77d4107660 0x7f77d4107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.414+0000 7f77db685640 1 -- 172.21.15.23:0/338486228 >> 172.21.15.23:0/338486228 conn(0x7f77d4076ac0 msgr2=0x7f77d40fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.414+0000 7f77db685640 1 -- 172.21.15.23:0/338486228 shutdown_connections 2024-03-28T05:23:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.414+0000 7f77db685640 1 -- 172.21.15.23:0/338486228 wait complete. 2024-03-28T05:23:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.414+0000 7f77db685640 1 Processor -- start 2024-03-28T05:23:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.414+0000 7f77db685640 1 -- start start 2024-03-28T05:23:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.415+0000 7f77db685640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77d40ffb40 0x7f77d4197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.415+0000 7f77db685640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77d4100500 0x7f77d41983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.415+0000 7f77db685640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77d4107660 0x7f77d419c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.415+0000 7f77db685640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d4111a70 con 0x7f77d40ffb40 2024-03-28T05:23:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.415+0000 7f77db685640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d41118f0 con 0x7f77d4100500 2024-03-28T05:23:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.415+0000 7f77db685640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d4111bf0 con 0x7f77d4107660 2024-03-28T05:23:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.415+0000 7f77d9421640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77d40ffb40 0x7f77d4197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.415+0000 7f77d9421640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77d40ffb40 0x7f77d4197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43850/0 (socket says 172.21.15.23:43850) 2024-03-28T05:23:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.415+0000 7f77d9421640 1 -- 172.21.15.23:0/1263783108 learned_addr learned my addr 172.21.15.23:0/1263783108 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.415+0000 7f77d9c22640 1 --2- 172.21.15.23:0/1263783108 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77d4107660 0x7f77d419c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.415+0000 7f77d8c20640 1 --2- 172.21.15.23:0/1263783108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77d4100500 0x7f77d41983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.415+0000 7f77d9421640 1 -- 172.21.15.23:0/1263783108 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77d4107660 msgr2=0x7f77d419c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.415+0000 7f77d9421640 1 --2- 172.21.15.23:0/1263783108 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77d4107660 0x7f77d419c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.415+0000 7f77d9421640 1 -- 172.21.15.23:0/1263783108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77d4100500 msgr2=0x7f77d41983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.415+0000 7f77d9421640 1 --2- 172.21.15.23:0/1263783108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77d4100500 0x7f77d41983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.415+0000 7f77d9421640 1 -- 172.21.15.23:0/1263783108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d419cea0 con 0x7f77d40ffb40 2024-03-28T05:23:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.415+0000 7f77d8c20640 1 --2- 172.21.15.23:0/1263783108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77d4100500 0x7f77d41983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:23:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.415+0000 7f77d9421640 1 --2- 172.21.15.23:0/1263783108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77d40ffb40 0x7f77d4197eb0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f77d000e9f0 tx=0x7f77d000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.415+0000 7f77ca7fc640 1 -- 172.21.15.23:0/1263783108 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d000cd80 con 0x7f77d40ffb40 2024-03-28T05:23:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.415+0000 7f77ca7fc640 1 -- 172.21.15.23:0/1263783108 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77d0005500 con 0x7f77d40ffb40 2024-03-28T05:23:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.415+0000 7f77db685640 1 -- 172.21.15.23:0/1263783108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d419d190 con 0x7f77d40ffb40 2024-03-28T05:23:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.415+0000 7f77ca7fc640 1 -- 172.21.15.23:0/1263783108 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d0010690 con 0x7f77d40ffb40 2024-03-28T05:23:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.416+0000 7f77db685640 1 -- 172.21.15.23:0/1263783108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d41a4a20 con 0x7f77d40ffb40 2024-03-28T05:23:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.418+0000 7f77ca7fc640 1 -- 172.21.15.23:0/1263783108 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 24) v1 ==== 137466+0+0 (secure 0 0 0) 0x7f77d00046c0 con 0x7f77d40ffb40 2024-03-28T05:23:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.418+0000 7f77de276740 1 Processor -- start 2024-03-28T05:23:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.418+0000 7f77de276740 1 -- start start 2024-03-28T05:23:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.419+0000 7f77de276740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca89710 0x562d9ca89af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.419+0000 7f77ca7fc640 1 --2- 172.21.15.23:0/1263783108 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f77c00a2ab0 0x7f77c00a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.419+0000 7f77de276740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca8a030 0x562d9ca928c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.419+0000 7f77de276740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562d9ca92e00 0x562d9ca951f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.419+0000 7f77de276740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d9c9edb10 con 0x562d9ca8a030 2024-03-28T05:23:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.419+0000 7f77de276740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d9c996900 con 0x562d9ca92e00 2024-03-28T05:23:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.419+0000 7f77de276740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d9c9e9bb0 con 0x562d9ca89710 2024-03-28T05:23:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.419+0000 7f77d8c20640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca8a030 0x562d9ca928c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.419+0000 7f77d9421640 1 --2- 172.21.15.23:0/1263783108 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f77c00a2ab0 0x7f77c00a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.419+0000 7f77d9c22640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca89710 0x562d9ca89af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.419+0000 7f77d8c20640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca8a030 0x562d9ca928c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43860/0 (socket says 172.21.15.23:43860) 2024-03-28T05:23:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.419+0000 7f77d9c22640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca89710 0x562d9ca89af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:49164/0 (socket says 172.21.15.23:49164) 2024-03-28T05:23:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.419+0000 7f77d8c20640 1 -- 172.21.15.23:0/146335954 learned_addr learned my addr 172.21.15.23:0/146335954 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.419+0000 7f77ca7fc640 1 -- 172.21.15.23:0/1263783108 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f77d00cf7d0 con 0x7f77d40ffb40 2024-03-28T05:23:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.419+0000 7f77d9c22640 1 --2- 172.21.15.23:0/146335954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562d9ca92e00 0x562d9ca951f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.419+0000 7f77d8c20640 1 -- 172.21.15.23:0/146335954 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca89710 msgr2=0x562d9ca89af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:23:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.419+0000 7f77d8c20640 1 --2- 172.21.15.23:0/146335954 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca89710 0x562d9ca89af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.419+0000 7f77d8c20640 1 -- 172.21.15.23:0/146335954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562d9ca92e00 msgr2=0x562d9ca951f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:23:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.419+0000 7f77d8c20640 1 --2- 172.21.15.23:0/146335954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562d9ca92e00 0x562d9ca951f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.419+0000 7f77d8c20640 1 -- 172.21.15.23:0/146335954 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562d9c702210 con 0x562d9ca8a030 2024-03-28T05:23:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.419+0000 7f77d9c22640 1 --2- 172.21.15.23:0/146335954 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca89710 0x562d9ca89af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:23:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.419+0000 7f77d9421640 1 --2- 172.21.15.23:0/1263783108 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f77c00a2ab0 0x7f77c00a4f70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f77d00ccb30 tx=0x7f77d0086040 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.420+0000 7f77d8c20640 1 --2- 172.21.15.23:0/146335954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca8a030 0x562d9ca928c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f77c4000c00 tx=0x7f77c4005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cef2688bddbab1f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.420+0000 7f77bf7fe640 1 -- 172.21.15.23:0/146335954 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c402ca80 con 0x562d9ca8a030 2024-03-28T05:23:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.420+0000 7f77bf7fe640 1 -- 172.21.15.23:0/146335954 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77c402cc20 con 0x562d9ca8a030 2024-03-28T05:23:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.420+0000 7f77bf7fe640 1 -- 172.21.15.23:0/146335954 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c40046c0 con 0x562d9ca8a030 2024-03-28T05:23:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.420+0000 7f77de276740 1 -- 172.21.15.23:0/146335954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca8a030 msgr2=0x562d9ca928c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.420+0000 7f77de276740 1 --2- 172.21.15.23:0/146335954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca8a030 0x562d9ca928c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f77c4000c00 tx=0x7f77c4005df0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.420+0000 7f77de276740 1 -- 172.21.15.23:0/146335954 shutdown_connections 2024-03-28T05:23:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.420+0000 7f77de276740 1 --2- 172.21.15.23:0/146335954 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562d9ca92e00 0x562d9ca951f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.420+0000 7f77de276740 1 --2- 172.21.15.23:0/146335954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca8a030 0x562d9ca928c0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.420+0000 7f77de276740 1 --2- 172.21.15.23:0/146335954 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca89710 0x562d9ca89af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.421+0000 7f77de276740 1 -- 172.21.15.23:0/146335954 >> 172.21.15.23:0/146335954 conn(0x562d9ca84030 msgr2=0x562d9ca84410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.421+0000 7f77de276740 1 -- 172.21.15.23:0/146335954 shutdown_connections 2024-03-28T05:23:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.421+0000 7f77de276740 1 -- 172.21.15.23:0/146335954 wait complete. 2024-03-28T05:23:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.421+0000 7f77de276740 1 -- start start 2024-03-28T05:23:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.421+0000 7f77de276740 1 Processor -- start 2024-03-28T05:23:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.421+0000 7f77de276740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562d9ca89710 0x562d9ca8fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.421+0000 7f77de276740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca8a030 0x562d9ca90360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.421+0000 7f77de276740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca92e00 0x562d9ca908a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.421+0000 7f77de276740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d9ca959d0 con 0x562d9ca92e00 2024-03-28T05:23:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.421+0000 7f77de276740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d9c9a9d50 con 0x562d9ca89710 2024-03-28T05:23:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.421+0000 7f77de276740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d9ca95b50 con 0x562d9ca8a030 2024-03-28T05:23:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.421+0000 7f77de276740 10 client.0 _ll_register_callbacks cb 0x562d9c9a9720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:23:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.422+0000 7f77d9c22640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca92e00 0x562d9ca908a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.422+0000 7f77d8c20640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca8a030 0x562d9ca90360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.422+0000 7f77d9c22640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca92e00 0x562d9ca908a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43876/0 (socket says 172.21.15.23:43876) 2024-03-28T05:23:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.422+0000 7f77d9c22640 1 -- 172.21.15.23:0/2442237878 learned_addr learned my addr 172.21.15.23:0/2442237878 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.422+0000 7f77d9c22640 1 --2- 172.21.15.23:0/2442237878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562d9ca89710 0x562d9ca8fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.422+0000 7f77d9c22640 1 -- 172.21.15.23:0/2442237878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca8a030 msgr2=0x562d9ca90360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:23:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.422+0000 7f77d9c22640 1 --2- 172.21.15.23:0/2442237878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca8a030 0x562d9ca90360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.422+0000 7f77d9c22640 1 -- 172.21.15.23:0/2442237878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562d9ca89710 msgr2=0x562d9ca8fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:23:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.422+0000 7f77d9c22640 1 --2- 172.21.15.23:0/2442237878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562d9ca89710 0x562d9ca8fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.422+0000 7f77d9c22640 1 -- 172.21.15.23:0/2442237878 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562d9c7ce620 con 0x562d9ca92e00 2024-03-28T05:23:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.422+0000 7f77d8c20640 1 --2- 172.21.15.23:0/2442237878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca8a030 0x562d9ca90360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:23:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.422+0000 7f77d9c22640 1 --2- 172.21.15.23:0/2442237878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca92e00 0x562d9ca908a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f77cc015f80 tx=0x7f77cc019040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21ce2942c4b59ab2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.422+0000 7f77bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:23:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.422+0000 7f77bcff9640 1 -- 172.21.15.23:0/2442237878 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77cc0150d0 con 0x562d9ca92e00 2024-03-28T05:23:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.422+0000 7f77bcff9640 1 -- 172.21.15.23:0/2442237878 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f77cc004b20 con 0x562d9ca92e00 2024-03-28T05:23:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.422+0000 7f77de276740 1 -- 172.21.15.23:0/2442237878 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f77c402cc20 con 0x562d9ca92e00 2024-03-28T05:23:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.422+0000 7f77bcff9640 1 -- 172.21.15.23:0/2442237878 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77cc027050 con 0x562d9ca92e00 2024-03-28T05:23:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.422+0000 7f77de276740 10 client.15492 fetch_fsmap learned FSMap version 20 2024-03-28T05:23:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.422+0000 7f77de276740 1 -- 172.21.15.23:0/2442237878 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x562d9caa1c50 con 0x562d9ca92e00 2024-03-28T05:23:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.423+0000 7f77bcff9640 1 -- 172.21.15.23:0/2442237878 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77c402cc20 con 0x562d9ca92e00 2024-03-28T05:23:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.423+0000 7f77bcff9640 1 -- 172.21.15.23:0/2442237878 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f77cc00f040 con 0x562d9ca92e00 2024-03-28T05:23:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.423+0000 7f77de276740 10 client.15492 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:23:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.423+0000 7f77de276740 10 client.15492 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:23:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.423+0000 7f77de276740 20 client.15492 populate_metadata read hostname 'smithi023' 2024-03-28T05:23:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.423+0000 7f77de276740 1 --2- 172.21.15.23:0/2442237878 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x562d9caa3110 0x562d9caa5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.423+0000 7f77de276740 4 client.15492 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:23:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.423+0000 7f77de276740 1 -- 172.21.15.23:0/2442237878 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562d9c9fb050 con 0x562d9caa3110 2024-03-28T05:23:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.423+0000 7f77d8c20640 1 --2- 172.21.15.23:0/2442237878 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x562d9caa3110 0x562d9caa5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.423+0000 7f77d8c20640 1 --2- 172.21.15.23:0/2442237878 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x562d9caa3110 0x562d9caa5500 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9517a9e23500d22 server_cookie=335732fcf64cec18 in_seq=0 out_seq=0 2024-03-28T05:23:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.423+0000 7f77bcff9640 10 client.15492 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:23:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.424+0000 7f77bcff9640 1 -- 172.21.15.23:0/2442237878 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562d9c9fb050 con 0x562d9caa3110 2024-03-28T05:23:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.424+0000 7f77bcff9640 10 client.15492 handle_command_reply: tid=0 2024-03-28T05:23:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.424+0000 7f77de276740 1 client.15492 shutdown 2024-03-28T05:23:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.424+0000 7f77de276740 1 -- 172.21.15.23:0/2442237878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca92e00 msgr2=0x562d9ca908a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.424+0000 7f77de276740 1 --2- 172.21.15.23:0/2442237878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca92e00 0x562d9ca908a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f77cc015f80 tx=0x7f77cc019040 comp rx=0 tx=0).stop 2024-03-28T05:23:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.424+0000 7f77de276740 1 -- 172.21.15.23:0/2442237878 shutdown_connections 2024-03-28T05:23:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.424+0000 7f77de276740 1 --2- 172.21.15.23:0/2442237878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca92e00 0x562d9ca908a0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.424+0000 7f77de276740 1 --2- 172.21.15.23:0/2442237878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca8a030 0x562d9ca90360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.424+0000 7f77de276740 1 --2- 172.21.15.23:0/2442237878 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x562d9caa3110 0x562d9caa5500 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.424+0000 7f77de276740 1 --2- 172.21.15.23:0/2442237878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562d9ca89710 0x562d9ca8fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.424+0000 7f77bcff9640 0 client.15492 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:23:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.424+0000 7f77de276740 1 -- 172.21.15.23:0/2442237878 >> 172.21.15.23:0/2442237878 conn(0x562d9ca84030 msgr2=0x562d9ca85d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.424+0000 7f77de276740 1 -- 172.21.15.23:0/2442237878 shutdown_connections 2024-03-28T05:23:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.424+0000 7f77de276740 1 -- 172.21.15.23:0/2442237878 wait complete. 2024-03-28T05:23:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.425+0000 7f77de276740 20 client.15492 trim_cache size 0 max 16384 2024-03-28T05:23:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.438+0000 7f77de276740 1 Processor -- start 2024-03-28T05:23:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.438+0000 7f77de276740 1 -- start start 2024-03-28T05:23:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.439+0000 7f77de276740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca89710 0x562d9ca92550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.439+0000 7f77de276740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca8a030 0x562d9ca904f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.439+0000 7f77de276740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562d9ca92e00 0x562d9ca9e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.439+0000 7f77de276740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d41aca20 con 0x562d9ca8a030 2024-03-28T05:23:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.439+0000 7f77de276740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d41aefe0 con 0x562d9ca92e00 2024-03-28T05:23:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.439+0000 7f77de276740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d41ad020 con 0x562d9ca89710 2024-03-28T05:23:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.439+0000 7f77d8c20640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca8a030 0x562d9ca904f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.439+0000 7f77d8c20640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca8a030 0x562d9ca904f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43880/0 (socket says 172.21.15.23:43880) 2024-03-28T05:23:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.439+0000 7f77d8c20640 1 -- 172.21.15.23:0/224690658 learned_addr learned my addr 172.21.15.23:0/224690658 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.439+0000 7f77d9c22640 1 --2- 172.21.15.23:0/224690658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca89710 0x562d9ca92550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.439+0000 7f77d8c20640 1 -- 172.21.15.23:0/224690658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca89710 msgr2=0x562d9ca92550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:23:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.439+0000 7f77d8c20640 1 --2- 172.21.15.23:0/224690658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca89710 0x562d9ca92550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.439+0000 7f77d8c20640 1 -- 172.21.15.23:0/224690658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562d9ca92e00 msgr2=0x562d9ca9e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:23:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.439+0000 7f77d9c22640 1 --2- 172.21.15.23:0/224690658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562d9ca92e00 0x562d9ca9e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.439+0000 7f77d8c20640 1 --2- 172.21.15.23:0/224690658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562d9ca92e00 0x562d9ca9e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.439+0000 7f77d8c20640 1 -- 172.21.15.23:0/224690658 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562d9caa5870 con 0x562d9ca8a030 2024-03-28T05:23:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.439+0000 7f77d9c22640 1 --2- 172.21.15.23:0/224690658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562d9ca92e00 0x562d9ca9e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:23:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.439+0000 7f77d8c20640 1 --2- 172.21.15.23:0/224690658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca8a030 0x562d9ca904f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x562d9ca891c0 tx=0x7f77c402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bcd2a1985da9ca2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.439+0000 7f77beffd640 1 -- 172.21.15.23:0/224690658 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c402cb00 con 0x562d9ca8a030 2024-03-28T05:23:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.440+0000 7f77beffd640 1 -- 172.21.15.23:0/224690658 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77c402c650 con 0x562d9ca8a030 2024-03-28T05:23:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.440+0000 7f77beffd640 1 -- 172.21.15.23:0/224690658 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c4053cd0 con 0x562d9ca8a030 2024-03-28T05:23:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.440+0000 7f77de276740 1 -- 172.21.15.23:0/224690658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca8a030 msgr2=0x562d9ca904f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.440+0000 7f77de276740 1 --2- 172.21.15.23:0/224690658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca8a030 0x562d9ca904f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x562d9ca891c0 tx=0x7f77c402bc30 comp rx=0 tx=0).stop 2024-03-28T05:23:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.440+0000 7f77de276740 1 -- 172.21.15.23:0/224690658 shutdown_connections 2024-03-28T05:23:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.440+0000 7f77de276740 1 --2- 172.21.15.23:0/224690658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562d9ca92e00 0x562d9ca9e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.440+0000 7f77de276740 1 --2- 172.21.15.23:0/224690658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca8a030 0x562d9ca904f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.440+0000 7f77de276740 1 --2- 172.21.15.23:0/224690658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca89710 0x562d9ca92550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.440+0000 7f77de276740 1 -- 172.21.15.23:0/224690658 >> 172.21.15.23:0/224690658 conn(0x562d9ca84030 msgr2=0x562d9ca8fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.440+0000 7f77de276740 1 -- 172.21.15.23:0/224690658 shutdown_connections 2024-03-28T05:23:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.440+0000 7f77de276740 1 -- 172.21.15.23:0/224690658 wait complete. 2024-03-28T05:23:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.440+0000 7f77de276740 1 -- start start 2024-03-28T05:23:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.441+0000 7f77de276740 1 Processor -- start 2024-03-28T05:23:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.441+0000 7f77de276740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562d9ca89710 0x562d9ca9a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.441+0000 7f77de276740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca8a030 0x562d9ca8d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.441+0000 7f77de276740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca92e00 0x562d9ca8d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.441+0000 7f77de276740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d41ad660 con 0x562d9ca92e00 2024-03-28T05:23:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.441+0000 7f77de276740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d41ae980 con 0x562d9ca89710 2024-03-28T05:23:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.441+0000 7f77de276740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d41adcc0 con 0x562d9ca8a030 2024-03-28T05:23:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.441+0000 7f77de276740 10 client.0 _ll_register_callbacks cb 0x7f77cc015cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:23:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.441+0000 7f77d9c22640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca92e00 0x562d9ca8d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.441+0000 7f77d9c22640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca92e00 0x562d9ca8d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43886/0 (socket says 172.21.15.23:43886) 2024-03-28T05:23:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.441+0000 7f77d9c22640 1 -- 172.21.15.23:0/1119039509 learned_addr learned my addr 172.21.15.23:0/1119039509 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:23:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.441+0000 7f77d8c20640 1 --2- 172.21.15.23:0/1119039509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca8a030 0x562d9ca8d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.441+0000 7f77d9c22640 1 --2- 172.21.15.23:0/1119039509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562d9ca89710 0x562d9ca9a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.441+0000 7f77d8c20640 1 -- 172.21.15.23:0/1119039509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562d9ca89710 msgr2=0x562d9ca9a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:23:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.441+0000 7f77d8c20640 1 --2- 172.21.15.23:0/1119039509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562d9ca89710 0x562d9ca9a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.441+0000 7f77d8c20640 1 -- 172.21.15.23:0/1119039509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca92e00 msgr2=0x562d9ca8d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:23:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.441+0000 7f77d8c20640 1 --2- 172.21.15.23:0/1119039509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca92e00 0x562d9ca8d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.441+0000 7f77d8c20640 1 -- 172.21.15.23:0/1119039509 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562d9ca8da80 con 0x562d9ca8a030 2024-03-28T05:23:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.441+0000 7f77d9c22640 1 --2- 172.21.15.23:0/1119039509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca92e00 0x562d9ca8d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:23:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.442+0000 7f77d8c20640 1 --2- 172.21.15.23:0/1119039509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca8a030 0x562d9ca8d000 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x562d9ca95690 tx=0x7f77c4005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40e31c36f9175468 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:23:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.442+0000 7f77bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-28T05:23:49.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.442+0000 7f77bcff9640 1 -- 172.21.15.23:0/1119039509 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c4004440 con 0x562d9ca8a030 2024-03-28T05:23:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.442+0000 7f77de276740 1 -- 172.21.15.23:0/1119039509 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f77c402c650 con 0x562d9ca8a030 2024-03-28T05:23:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.442+0000 7f77bcff9640 1 -- 172.21.15.23:0/1119039509 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f77c40539b0 con 0x562d9ca8a030 2024-03-28T05:23:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.442+0000 7f77bcff9640 1 -- 172.21.15.23:0/1119039509 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77c4053bb0 con 0x562d9ca8a030 2024-03-28T05:23:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.442+0000 7f77de276740 10 client.25336 fetch_fsmap learned FSMap version 20 2024-03-28T05:23:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.442+0000 7f77de276740 1 -- 172.21.15.23:0/1119039509 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x562d9caff620 con 0x562d9ca8a030 2024-03-28T05:23:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.442+0000 7f77bcff9640 1 -- 172.21.15.23:0/1119039509 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77c402c650 con 0x562d9ca8a030 2024-03-28T05:23:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.442+0000 7f77bcff9640 1 -- 172.21.15.23:0/1119039509 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f77c4050070 con 0x562d9ca8a030 2024-03-28T05:23:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.442+0000 7f77de276740 10 client.25336 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:23:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.442+0000 7f77de276740 10 client.25336 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:23:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.442+0000 7f77de276740 20 client.25336 populate_metadata read hostname 'smithi023' 2024-03-28T05:23:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.442+0000 7f77de276740 1 --2- 172.21.15.23:0/1119039509 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x562d9caff8b0 0x562d9cb01ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:23:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.442+0000 7f77de276740 4 client.25336 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:23:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.442+0000 7f77de276740 1 -- 172.21.15.23:0/1119039509 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562d9c9f53a0 con 0x562d9caff8b0 2024-03-28T05:23:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.443+0000 7f77d9c22640 1 --2- 172.21.15.23:0/1119039509 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x562d9caff8b0 0x562d9cb01ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:23:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.443+0000 7f77d9c22640 1 --2- 172.21.15.23:0/1119039509 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x562d9caff8b0 0x562d9cb01ca0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cf25d6503da5f99 server_cookie=b589f3869180ffc8 in_seq=0 out_seq=0 2024-03-28T05:23:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.443+0000 7f77bcff9640 10 client.25336 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:23:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.454+0000 7f77bcff9640 1 -- 172.21.15.23:0/1119039509 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562d9c9f53a0 con 0x562d9caff8b0 2024-03-28T05:23:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.455+0000 7f77bcff9640 10 client.25336 handle_command_reply: tid=0 2024-03-28T05:23:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.455+0000 7f77de276740 1 client.25336 shutdown 2024-03-28T05:23:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.455+0000 7f77de276740 1 -- 172.21.15.23:0/1119039509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca8a030 msgr2=0x562d9ca8d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.455+0000 7f77de276740 1 --2- 172.21.15.23:0/1119039509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca8a030 0x562d9ca8d000 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x562d9ca95690 tx=0x7f77c4005850 comp rx=0 tx=0).stop 2024-03-28T05:23:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.455+0000 7f77de276740 1 -- 172.21.15.23:0/1119039509 shutdown_connections 2024-03-28T05:23:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.455+0000 7f77de276740 1 --2- 172.21.15.23:0/1119039509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562d9ca92e00 0x562d9ca8d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.455+0000 7f77de276740 1 --2- 172.21.15.23:0/1119039509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562d9ca8a030 0x562d9ca8d000 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.455+0000 7f77de276740 1 --2- 172.21.15.23:0/1119039509 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x562d9caff8b0 0x562d9cb01ca0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.455+0000 7f77de276740 1 --2- 172.21.15.23:0/1119039509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562d9ca89710 0x562d9ca9a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.455+0000 7f77bcff9640 0 client.25336 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:23:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.455+0000 7f77de276740 1 -- 172.21.15.23:0/1119039509 >> 172.21.15.23:0/1119039509 conn(0x562d9ca84030 msgr2=0x562d9ca90670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.455+0000 7f77de276740 1 -- 172.21.15.23:0/1119039509 shutdown_connections 2024-03-28T05:23:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.455+0000 7f77de276740 1 -- 172.21.15.23:0/1119039509 wait complete. 2024-03-28T05:23:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.455+0000 7f77de276740 20 client.25336 trim_cache size 0 max 16384 2024-03-28T05:23:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:23:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:23:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:23:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:23:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.456+0000 7f77db685640 1 -- 172.21.15.23:0/1263783108 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f77c00a2ab0 msgr2=0x7f77c00a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.456+0000 7f77db685640 1 --2- 172.21.15.23:0/1263783108 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f77c00a2ab0 0x7f77c00a4f70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f77d00ccb30 tx=0x7f77d0086040 comp rx=0 tx=0).stop 2024-03-28T05:23:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.456+0000 7f77db685640 1 -- 172.21.15.23:0/1263783108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77d40ffb40 msgr2=0x7f77d4197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:23:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.456+0000 7f77db685640 1 --2- 172.21.15.23:0/1263783108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77d40ffb40 0x7f77d4197eb0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f77d000e9f0 tx=0x7f77d000edf0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.456+0000 7f77db685640 1 -- 172.21.15.23:0/1263783108 shutdown_connections 2024-03-28T05:23:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.456+0000 7f77db685640 1 --2- 172.21.15.23:0/1263783108 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77d4107660 0x7f77d419c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.456+0000 7f77db685640 1 --2- 172.21.15.23:0/1263783108 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f77c00a2ab0 0x7f77c00a4f70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.456+0000 7f77db685640 1 --2- 172.21.15.23:0/1263783108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f77d4100500 0x7f77d41983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.456+0000 7f77db685640 1 --2- 172.21.15.23:0/1263783108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77d40ffb40 0x7f77d4197eb0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:23:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.456+0000 7f77db685640 1 -- 172.21.15.23:0/1263783108 >> 172.21.15.23:0/1263783108 conn(0x7f77d4076ac0 msgr2=0x7f77d4104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:23:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.456+0000 7f77db685640 1 -- 172.21.15.23:0/1263783108 shutdown_connections 2024-03-28T05:23:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:23:49.456+0000 7f77db685640 1 -- 172.21.15.23:0/1263783108 wait complete. 2024-03-28T05:23:50.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:49 smithi023 ceph-mon[31954]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 43 op/s 2024-03-28T05:23:50.044 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:49 smithi113 ceph-mon[37164]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 43 op/s 2024-03-28T05:23:50.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:49 smithi098 ceph-mon[36087]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 43 op/s 2024-03-28T05:23:52.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:51 smithi023 ceph-mon[31954]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-03-28T05:23:52.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:51 smithi098 ceph-mon[36087]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-03-28T05:23:52.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:51 smithi113 ceph-mon[37164]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-03-28T05:23:54.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:53 smithi023 ceph-mon[31954]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 78 KiB/s wr, 49 op/s 2024-03-28T05:23:54.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:53 smithi098 ceph-mon[36087]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 78 KiB/s wr, 49 op/s 2024-03-28T05:23:54.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:53 smithi113 ceph-mon[37164]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 78 KiB/s wr, 49 op/s 2024-03-28T05:23:56.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:55 smithi023 ceph-mon[31954]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.6 KiB/s wr, 6 op/s 2024-03-28T05:23:56.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:55 smithi098 ceph-mon[36087]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.6 KiB/s wr, 6 op/s 2024-03-28T05:23:56.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:55 smithi113 ceph-mon[37164]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.6 KiB/s wr, 6 op/s 2024-03-28T05:23:58.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:57 smithi023 ceph-mon[31954]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-28T05:23:58.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:57 smithi098 ceph-mon[36087]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-28T05:23:58.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:57 smithi113 ceph-mon[37164]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-28T05:24:00.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:23:59 smithi023 ceph-mon[31954]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.6 KiB/s wr, 6 op/s 2024-03-28T05:24:00.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:23:59 smithi098 ceph-mon[36087]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.6 KiB/s wr, 6 op/s 2024-03-28T05:24:00.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:23:59 smithi113 ceph-mon[37164]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.6 KiB/s wr, 6 op/s 2024-03-28T05:24:00.781 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:24:00 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:24:00.451+0000 7fb0cb717700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T05:24:00.781 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:24:00 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:24:00.451+0000 7fb0cb717700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T05:24:00.781 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:24:00 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:24:00.451+0000 7fb0cb717700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T05:24:00.781 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:24:00 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:24:00.451+0000 7fb0cb717700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T05:24:00.781 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:24:00 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:24:00.451+0000 7fb0cb717700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T05:24:02.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:01 smithi023 ceph-mon[31954]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.6 KiB/s wr, 6 op/s 2024-03-28T05:24:02.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:01 smithi098 ceph-mon[36087]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.6 KiB/s wr, 6 op/s 2024-03-28T05:24:02.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:01 smithi113 ceph-mon[37164]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.6 KiB/s wr, 6 op/s 2024-03-28T05:24:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.554+0000 7f124be7e640 1 -- 172.21.15.23:0/3923335711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12440696a0 msgr2=0x7f124410e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.554+0000 7f124be7e640 1 --2- 172.21.15.23:0/3923335711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12440696a0 0x7f124410e540 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f1234009cd0 tx=0x7f123402c0f0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.554+0000 7f124be7e640 1 -- 172.21.15.23:0/3923335711 shutdown_connections 2024-03-28T05:24:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.554+0000 7f124be7e640 1 --2- 172.21.15.23:0/3923335711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12440696a0 0x7f124410e540 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.554+0000 7f124be7e640 1 --2- 172.21.15.23:0/3923335711 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1244068d20 0x7f1244069160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.556 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.554+0000 7f124be7e640 1 --2- 172.21.15.23:0/3923335711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1244107680 0x7f1244107a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.554+0000 7f124be7e640 1 -- 172.21.15.23:0/3923335711 >> 172.21.15.23:0/3923335711 conn(0x7f12440773f0 msgr2=0x7f1244077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.554+0000 7f124be7e640 1 -- 172.21.15.23:0/3923335711 shutdown_connections 2024-03-28T05:24:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.554+0000 7f124be7e640 1 -- 172.21.15.23:0/3923335711 wait complete. 2024-03-28T05:24:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.555+0000 7f124be7e640 1 Processor -- start 2024-03-28T05:24:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.555+0000 7f124be7e640 1 -- start start 2024-03-28T05:24:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.555+0000 7f124be7e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1244068d20 0x7f1244197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.555+0000 7f124be7e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12440696a0 0x7f12441984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.555+0000 7f124be7e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1244107680 0x7f124419c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.555+0000 7f124be7e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1244111a80 con 0x7f1244107680 2024-03-28T05:24:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.555+0000 7f124be7e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1244111900 con 0x7f12440696a0 2024-03-28T05:24:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.555+0000 7f124be7e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1244111c00 con 0x7f1244068d20 2024-03-28T05:24:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.555+0000 7f124a41b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1244107680 0x7f124419c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.555+0000 7f1249c1a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1244068d20 0x7f1244197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.555+0000 7f1249c1a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1244068d20 0x7f1244197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:56136/0 (socket says 172.21.15.23:56136) 2024-03-28T05:24:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.555+0000 7f124a41b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1244107680 0x7f124419c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49214/0 (socket says 172.21.15.23:49214) 2024-03-28T05:24:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.555+0000 7f1249c1a640 1 -- 172.21.15.23:0/3136631950 learned_addr learned my addr 172.21.15.23:0/3136631950 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.555+0000 7f1249419640 1 --2- 172.21.15.23:0/3136631950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12440696a0 0x7f12441984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.555+0000 7f1249c1a640 1 -- 172.21.15.23:0/3136631950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12440696a0 msgr2=0x7f12441984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.555+0000 7f1249c1a640 1 --2- 172.21.15.23:0/3136631950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12440696a0 0x7f12441984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.555+0000 7f1249c1a640 1 -- 172.21.15.23:0/3136631950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1244107680 msgr2=0x7f124419c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.555+0000 7f1249c1a640 1 --2- 172.21.15.23:0/3136631950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1244107680 0x7f124419c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.555+0000 7f1249c1a640 1 -- 172.21.15.23:0/3136631950 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f124419cf20 con 0x7f1244068d20 2024-03-28T05:24:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.555+0000 7f124a41b640 1 --2- 172.21.15.23:0/3136631950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1244107680 0x7f124419c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:24:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.556+0000 7f1249c1a640 1 --2- 172.21.15.23:0/3136631950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1244068d20 0x7f1244197f80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f123c00e9f0 tx=0x7f123c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.556+0000 7f1242ffd640 1 -- 172.21.15.23:0/3136631950 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c00cd80 con 0x7f1244068d20 2024-03-28T05:24:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.556+0000 7f1242ffd640 1 -- 172.21.15.23:0/3136631950 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f123c010910 con 0x7f1244068d20 2024-03-28T05:24:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.556+0000 7f124be7e640 1 -- 172.21.15.23:0/3136631950 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f124419d210 con 0x7f1244068d20 2024-03-28T05:24:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.556+0000 7f1242ffd640 1 -- 172.21.15.23:0/3136631950 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c018990 con 0x7f1244068d20 2024-03-28T05:24:02.558 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.556+0000 7f124be7e640 1 -- 172.21.15.23:0/3136631950 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12441007f0 con 0x7f1244068d20 2024-03-28T05:24:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.558+0000 7f1242ffd640 1 -- 172.21.15.23:0/3136631950 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 24) v1 ==== 137466+0+0 (secure 0 0 0) 0x7f123c010ac0 con 0x7f1244068d20 2024-03-28T05:24:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.559+0000 7f124ea62740 1 Processor -- start 2024-03-28T05:24:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.559+0000 7f124ea62740 1 -- start start 2024-03-28T05:24:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.559+0000 7f1242ffd640 1 --2- 172.21.15.23:0/3136631950 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f122c0a2ab0 0x7f122c0a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.559+0000 7f124ea62740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564e6f348710 0x564e6f348af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.559+0000 7f124ea62740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f349030 0x564e6f3518c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.559+0000 7f124ea62740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e6f351e00 0x564e6f3541f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.559+0000 7f124ea62740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6f2acb10 con 0x564e6f349030 2024-03-28T05:24:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.559+0000 7f124ea62740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6f255900 con 0x564e6f351e00 2024-03-28T05:24:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.559+0000 7f124ea62740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6f2a8bb0 con 0x564e6f348710 2024-03-28T05:24:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.559+0000 7f124a41b640 1 --2- 172.21.15.23:0/3136631950 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f122c0a2ab0 0x7f122c0a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.559+0000 7f1249419640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f349030 0x564e6f3518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.559+0000 7f1249419640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f349030 0x564e6f3518c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49224/0 (socket says 172.21.15.23:49224) 2024-03-28T05:24:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.559+0000 7f1249419640 1 -- 172.21.15.23:0/2661355104 learned_addr learned my addr 172.21.15.23:0/2661355104 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.559+0000 7f1249c1a640 1 --2- 172.21.15.23:0/2661355104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564e6f348710 0x564e6f348af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.559+0000 7f1242ffd640 1 -- 172.21.15.23:0/3136631950 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f123c0cf6b0 con 0x7f1244068d20 2024-03-28T05:24:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.559+0000 7f124a41b640 1 --2- 172.21.15.23:0/2661355104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e6f351e00 0x564e6f3541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.560+0000 7f1249c1a640 1 -- 172.21.15.23:0/2661355104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e6f351e00 msgr2=0x564e6f3541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.560+0000 7f1249c1a640 1 --2- 172.21.15.23:0/2661355104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e6f351e00 0x564e6f3541f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.560+0000 7f1249c1a640 1 -- 172.21.15.23:0/2661355104 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f349030 msgr2=0x564e6f3518c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.560+0000 7f1249c1a640 1 --2- 172.21.15.23:0/2661355104 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f349030 0x564e6f3518c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.560+0000 7f1249c1a640 1 -- 172.21.15.23:0/2661355104 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564e6efc1210 con 0x564e6f348710 2024-03-28T05:24:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.560+0000 7f1249419640 1 --2- 172.21.15.23:0/2661355104 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f349030 0x564e6f3518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:24:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.560+0000 7f124a41b640 1 --2- 172.21.15.23:0/3136631950 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f122c0a2ab0 0x7f122c0a4f70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1234005cc0 tx=0x7f123402cec0 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.560+0000 7f1249c1a640 1 --2- 172.21.15.23:0/2661355104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564e6f348710 0x564e6f348af0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f123c0cc950 tx=0x7f123c09b460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd7dcbc145a40c0f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.560+0000 7f122bfff640 1 -- 172.21.15.23:0/2661355104 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c026040 con 0x564e6f348710 2024-03-28T05:24:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.560+0000 7f122bfff640 1 -- 172.21.15.23:0/2661355104 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f123c0adf80 con 0x564e6f348710 2024-03-28T05:24:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.560+0000 7f122bfff640 1 -- 172.21.15.23:0/2661355104 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c0860d0 con 0x564e6f348710 2024-03-28T05:24:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.561+0000 7f124ea62740 1 -- 172.21.15.23:0/2661355104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564e6f348710 msgr2=0x564e6f348af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.561+0000 7f124ea62740 1 --2- 172.21.15.23:0/2661355104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564e6f348710 0x564e6f348af0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f123c0cc950 tx=0x7f123c09b460 comp rx=0 tx=0).stop 2024-03-28T05:24:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.561+0000 7f124ea62740 1 -- 172.21.15.23:0/2661355104 shutdown_connections 2024-03-28T05:24:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.561+0000 7f124ea62740 1 --2- 172.21.15.23:0/2661355104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e6f351e00 0x564e6f3541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.561+0000 7f124ea62740 1 --2- 172.21.15.23:0/2661355104 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f349030 0x564e6f3518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.561+0000 7f124ea62740 1 --2- 172.21.15.23:0/2661355104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564e6f348710 0x564e6f348af0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.561+0000 7f124ea62740 1 -- 172.21.15.23:0/2661355104 >> 172.21.15.23:0/2661355104 conn(0x564e6f343030 msgr2=0x564e6f343410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.561+0000 7f124ea62740 1 -- 172.21.15.23:0/2661355104 shutdown_connections 2024-03-28T05:24:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.561+0000 7f124ea62740 1 -- 172.21.15.23:0/2661355104 wait complete. 2024-03-28T05:24:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.561+0000 7f124ea62740 1 -- start start 2024-03-28T05:24:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.561+0000 7f124ea62740 1 Processor -- start 2024-03-28T05:24:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.562+0000 7f124ea62740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f348710 0x564e6f34ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.562+0000 7f124ea62740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e6f349030 0x564e6f34f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.562+0000 7f124ea62740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564e6f351e00 0x564e6f34f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.562+0000 7f124ea62740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6f354a30 con 0x564e6f348710 2024-03-28T05:24:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.562+0000 7f124ea62740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6f268d50 con 0x564e6f349030 2024-03-28T05:24:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.562+0000 7f124ea62740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6f354bb0 con 0x564e6f351e00 2024-03-28T05:24:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.562+0000 7f124ea62740 10 client.0 _ll_register_callbacks cb 0x564e6f268720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:24:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.562+0000 7f1249c1a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f348710 0x564e6f34ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.562+0000 7f1249c1a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f348710 0x564e6f34ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49238/0 (socket says 172.21.15.23:49238) 2024-03-28T05:24:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.562+0000 7f1249c1a640 1 -- 172.21.15.23:0/2724552048 learned_addr learned my addr 172.21.15.23:0/2724552048 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.562+0000 7f124a41b640 1 --2- 172.21.15.23:0/2724552048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564e6f351e00 0x564e6f34f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.562+0000 7f1249419640 1 --2- 172.21.15.23:0/2724552048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e6f349030 0x564e6f34f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.562+0000 7f1249c1a640 1 -- 172.21.15.23:0/2724552048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564e6f351e00 msgr2=0x564e6f34f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.562+0000 7f1249c1a640 1 --2- 172.21.15.23:0/2724552048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564e6f351e00 0x564e6f34f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.562+0000 7f1249c1a640 1 -- 172.21.15.23:0/2724552048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e6f349030 msgr2=0x564e6f34f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.562+0000 7f1249c1a640 1 --2- 172.21.15.23:0/2724552048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e6f349030 0x564e6f34f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.562+0000 7f1249c1a640 1 -- 172.21.15.23:0/2724552048 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564e6f08d620 con 0x564e6f348710 2024-03-28T05:24:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.562+0000 7f1249419640 1 --2- 172.21.15.23:0/2724552048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e6f349030 0x564e6f34f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:24:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.562+0000 7f1249c1a640 1 --2- 172.21.15.23:0/2724552048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f348710 0x564e6f34ee90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f123c09b7a0 tx=0x7f123c018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=457054620da7d827 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.562+0000 7f12297fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:24:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.562+0000 7f12297fa640 1 -- 172.21.15.23:0/2724552048 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c09f900 con 0x564e6f348710 2024-03-28T05:24:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.562+0000 7f12297fa640 1 -- 172.21.15.23:0/2724552048 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f123c0afb90 con 0x564e6f348710 2024-03-28T05:24:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.563+0000 7f124ea62740 1 -- 172.21.15.23:0/2724552048 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f123c0adf80 con 0x564e6f348710 2024-03-28T05:24:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.563+0000 7f12297fa640 1 -- 172.21.15.23:0/2724552048 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f123c0cf260 con 0x564e6f348710 2024-03-28T05:24:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.563+0000 7f124ea62740 10 client.15516 fetch_fsmap learned FSMap version 20 2024-03-28T05:24:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.563+0000 7f124ea62740 1 -- 172.21.15.23:0/2724552048 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x564e6f360cf0 con 0x564e6f348710 2024-03-28T05:24:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.563+0000 7f12297fa640 1 -- 172.21.15.23:0/2724552048 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f123c0adf80 con 0x564e6f348710 2024-03-28T05:24:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.563+0000 7f12297fa640 1 -- 172.21.15.23:0/2724552048 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f123c0a8060 con 0x564e6f348710 2024-03-28T05:24:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.563+0000 7f124ea62740 10 client.15516 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:24:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.563+0000 7f124ea62740 10 client.15516 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:24:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.563+0000 7f124ea62740 20 client.15516 populate_metadata read hostname 'smithi023' 2024-03-28T05:24:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.563+0000 7f124ea62740 1 --2- 172.21.15.23:0/2724552048 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x564e6f362040 0x564e6f364430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.563+0000 7f124ea62740 4 client.15516 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:24:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.563+0000 7f124ea62740 1 -- 172.21.15.23:0/2724552048 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564e6f2ba050 con 0x564e6f362040 2024-03-28T05:24:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.563+0000 7f1249419640 1 --2- 172.21.15.23:0/2724552048 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x564e6f362040 0x564e6f364430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.564+0000 7f1249419640 1 --2- 172.21.15.23:0/2724552048 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x564e6f362040 0x564e6f364430 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98f9ab797aed2614 server_cookie=dbfbeb1500212535 in_seq=0 out_seq=0 2024-03-28T05:24:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.564+0000 7f12297fa640 10 client.15516 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:24:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.564+0000 7f12297fa640 1 -- 172.21.15.23:0/2724552048 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564e6f2ba050 con 0x564e6f362040 2024-03-28T05:24:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.564+0000 7f12297fa640 10 client.15516 handle_command_reply: tid=0 2024-03-28T05:24:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.564+0000 7f124ea62740 1 client.15516 shutdown 2024-03-28T05:24:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.564+0000 7f124ea62740 1 -- 172.21.15.23:0/2724552048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f348710 msgr2=0x564e6f34ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.564+0000 7f124ea62740 1 --2- 172.21.15.23:0/2724552048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f348710 0x564e6f34ee90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f123c09b7a0 tx=0x7f123c018420 comp rx=0 tx=0).stop 2024-03-28T05:24:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.565+0000 7f124ea62740 1 -- 172.21.15.23:0/2724552048 shutdown_connections 2024-03-28T05:24:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.565+0000 7f124ea62740 1 --2- 172.21.15.23:0/2724552048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564e6f351e00 0x564e6f34f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.565+0000 7f124ea62740 1 --2- 172.21.15.23:0/2724552048 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x564e6f362040 0x564e6f364430 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.565+0000 7f124ea62740 1 --2- 172.21.15.23:0/2724552048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e6f349030 0x564e6f34f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.565+0000 7f12297fa640 0 client.15516 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:24:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.565+0000 7f124ea62740 1 --2- 172.21.15.23:0/2724552048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f348710 0x564e6f34ee90 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.565+0000 7f124ea62740 1 -- 172.21.15.23:0/2724552048 >> 172.21.15.23:0/2724552048 conn(0x564e6f343030 msgr2=0x564e6f344d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.565+0000 7f124ea62740 1 -- 172.21.15.23:0/2724552048 shutdown_connections 2024-03-28T05:24:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.565+0000 7f124ea62740 1 -- 172.21.15.23:0/2724552048 wait complete. 2024-03-28T05:24:02.571 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.565+0000 7f124ea62740 20 client.15516 trim_cache size 0 max 16384 2024-03-28T05:24:02.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.579+0000 7f124ea62740 1 Processor -- start 2024-03-28T05:24:02.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.579+0000 7f124ea62740 1 -- start start 2024-03-28T05:24:02.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.579+0000 7f124ea62740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564e6f348710 0x564e6f34d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:02.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.579+0000 7f124ea62740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e6f349030 0x564e6f34dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:02.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.579+0000 7f124ea62740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f351e00 0x564e6f34c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:02.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.579+0000 7f124ea62740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12441aa930 con 0x564e6f351e00 2024-03-28T05:24:02.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.579+0000 7f124ea62740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12441acef0 con 0x564e6f349030 2024-03-28T05:24:02.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.579+0000 7f124ea62740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12441aaf30 con 0x564e6f348710 2024-03-28T05:24:02.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.579+0000 7f124a41b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f351e00 0x564e6f34c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:02.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.579+0000 7f124a41b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f351e00 0x564e6f34c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49252/0 (socket says 172.21.15.23:49252) 2024-03-28T05:24:02.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.579+0000 7f124a41b640 1 -- 172.21.15.23:0/3719651714 learned_addr learned my addr 172.21.15.23:0/3719651714 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:02.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.579+0000 7f1249c1a640 1 --2- 172.21.15.23:0/3719651714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564e6f348710 0x564e6f34d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:02.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.579+0000 7f1249419640 1 --2- 172.21.15.23:0/3719651714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e6f349030 0x564e6f34dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:02.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.579+0000 7f124a41b640 1 -- 172.21.15.23:0/3719651714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564e6f348710 msgr2=0x564e6f34d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:02.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.579+0000 7f124a41b640 1 --2- 172.21.15.23:0/3719651714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564e6f348710 0x564e6f34d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.579+0000 7f124a41b640 1 -- 172.21.15.23:0/3719651714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e6f349030 msgr2=0x564e6f34dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:02.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.579+0000 7f124a41b640 1 --2- 172.21.15.23:0/3719651714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e6f349030 0x564e6f34dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.579+0000 7f124a41b640 1 -- 172.21.15.23:0/3719651714 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564e6f354ab0 con 0x564e6f351e00 2024-03-28T05:24:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.579+0000 7f1249c1a640 1 --2- 172.21.15.23:0/3719651714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564e6f348710 0x564e6f34d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:24:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.579+0000 7f1249419640 1 --2- 172.21.15.23:0/3719651714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e6f349030 0x564e6f34dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:24:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.579+0000 7f124a41b640 1 --2- 172.21.15.23:0/3719651714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f351e00 0x564e6f34c0f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1234009cd0 tx=0x7f123402e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e3fb2f556fec2bd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.580+0000 7f122b7fe640 1 -- 172.21.15.23:0/3719651714 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1234039040 con 0x564e6f351e00 2024-03-28T05:24:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.580+0000 7f122b7fe640 1 -- 172.21.15.23:0/3719651714 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1234053070 con 0x564e6f351e00 2024-03-28T05:24:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.580+0000 7f122b7fe640 1 -- 172.21.15.23:0/3719651714 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12340042a0 con 0x564e6f351e00 2024-03-28T05:24:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.580+0000 7f124ea62740 1 -- 172.21.15.23:0/3719651714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f351e00 msgr2=0x564e6f34c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.580+0000 7f124ea62740 1 --2- 172.21.15.23:0/3719651714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f351e00 0x564e6f34c0f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1234009cd0 tx=0x7f123402e050 comp rx=0 tx=0).stop 2024-03-28T05:24:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.580+0000 7f124ea62740 1 -- 172.21.15.23:0/3719651714 shutdown_connections 2024-03-28T05:24:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.580+0000 7f124ea62740 1 --2- 172.21.15.23:0/3719651714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f351e00 0x564e6f34c0f0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.580+0000 7f124ea62740 1 --2- 172.21.15.23:0/3719651714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e6f349030 0x564e6f34dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.580+0000 7f124ea62740 1 --2- 172.21.15.23:0/3719651714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564e6f348710 0x564e6f34d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.580+0000 7f124ea62740 1 -- 172.21.15.23:0/3719651714 >> 172.21.15.23:0/3719651714 conn(0x564e6f343030 msgr2=0x564e6f343410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.580+0000 7f124ea62740 1 -- 172.21.15.23:0/3719651714 shutdown_connections 2024-03-28T05:24:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.580+0000 7f124ea62740 1 -- 172.21.15.23:0/3719651714 wait complete. 2024-03-28T05:24:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.581+0000 7f124ea62740 1 -- start start 2024-03-28T05:24:02.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.581+0000 7f124ea62740 1 Processor -- start 2024-03-28T05:24:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.581+0000 7f124ea62740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564e6f348710 0x564e6f367260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.581+0000 7f124ea62740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e6f349030 0x564e6f3645f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.581+0000 7f124ea62740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f351e00 0x564e6f364b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.581+0000 7f124ea62740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12441ab570 con 0x564e6f351e00 2024-03-28T05:24:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.581+0000 7f124ea62740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12441ac890 con 0x564e6f349030 2024-03-28T05:24:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.581+0000 7f124ea62740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12441abbd0 con 0x564e6f348710 2024-03-28T05:24:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.581+0000 7f124ea62740 10 client.0 _ll_register_callbacks cb 0x7f123c09b630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:24:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.581+0000 7f124a41b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f351e00 0x564e6f364b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.581+0000 7f124a41b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f351e00 0x564e6f364b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49254/0 (socket says 172.21.15.23:49254) 2024-03-28T05:24:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.581+0000 7f124a41b640 1 -- 172.21.15.23:0/545942667 learned_addr learned my addr 172.21.15.23:0/545942667 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.581+0000 7f1249c1a640 1 --2- 172.21.15.23:0/545942667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564e6f348710 0x564e6f367260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.581+0000 7f1249419640 1 --2- 172.21.15.23:0/545942667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e6f349030 0x564e6f3645f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.582+0000 7f124a41b640 1 -- 172.21.15.23:0/545942667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564e6f348710 msgr2=0x564e6f367260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.582+0000 7f124a41b640 1 --2- 172.21.15.23:0/545942667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564e6f348710 0x564e6f367260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.582+0000 7f124a41b640 1 -- 172.21.15.23:0/545942667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e6f349030 msgr2=0x564e6f3645f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.582+0000 7f124a41b640 1 --2- 172.21.15.23:0/545942667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e6f349030 0x564e6f3645f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.582+0000 7f124a41b640 1 -- 172.21.15.23:0/545942667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564e6f347c20 con 0x564e6f351e00 2024-03-28T05:24:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.582+0000 7f124a41b640 1 --2- 172.21.15.23:0/545942667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f351e00 0x564e6f364b50 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f123402c700 tx=0x7f1234005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=789a8f92f4277d72 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.582+0000 7f12297fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:24:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.582+0000 7f12297fa640 1 -- 172.21.15.23:0/545942667 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123405d020 con 0x564e6f351e00 2024-03-28T05:24:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.582+0000 7f12297fa640 1 -- 172.21.15.23:0/545942667 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f1234003970 con 0x564e6f351e00 2024-03-28T05:24:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.582+0000 7f124ea62740 1 -- 172.21.15.23:0/545942667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1234053070 con 0x564e6f351e00 2024-03-28T05:24:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.582+0000 7f12297fa640 1 -- 172.21.15.23:0/545942667 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1234068050 con 0x564e6f351e00 2024-03-28T05:24:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.582+0000 7f124ea62740 10 client.15522 fetch_fsmap learned FSMap version 20 2024-03-28T05:24:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.582+0000 7f124ea62740 1 -- 172.21.15.23:0/545942667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x564e6f3bd430 con 0x564e6f351e00 2024-03-28T05:24:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.582+0000 7f12297fa640 1 -- 172.21.15.23:0/545942667 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1234053070 con 0x564e6f351e00 2024-03-28T05:24:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.583+0000 7f12297fa640 1 -- 172.21.15.23:0/545942667 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f123403f070 con 0x564e6f351e00 2024-03-28T05:24:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.583+0000 7f124ea62740 10 client.15522 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:24:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.583+0000 7f124ea62740 10 client.15522 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:24:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.583+0000 7f124ea62740 20 client.15522 populate_metadata read hostname 'smithi023' 2024-03-28T05:24:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.583+0000 7f124ea62740 1 --2- 172.21.15.23:0/545942667 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x564e6f3bdad0 0x564e6f3bfef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.583+0000 7f124ea62740 4 client.15522 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:24:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.583+0000 7f124ea62740 1 -- 172.21.15.23:0/545942667 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564e6f2b43a0 con 0x564e6f3bdad0 2024-03-28T05:24:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.583+0000 7f1249c1a640 1 --2- 172.21.15.23:0/545942667 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x564e6f3bdad0 0x564e6f3bfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.583+0000 7f1249c1a640 1 --2- 172.21.15.23:0/545942667 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x564e6f3bdad0 0x564e6f3bfef0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adfbd7be2a355693 server_cookie=f04e5c911008bd8b in_seq=0 out_seq=0 2024-03-28T05:24:02.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.583+0000 7f12297fa640 10 client.15522 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:24:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.590+0000 7f12297fa640 1 -- 172.21.15.23:0/545942667 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564e6f2b43a0 con 0x564e6f3bdad0 2024-03-28T05:24:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.590+0000 7f12297fa640 10 client.15522 handle_command_reply: tid=0 2024-03-28T05:24:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.590+0000 7f124ea62740 1 client.15522 shutdown 2024-03-28T05:24:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.590+0000 7f124ea62740 1 -- 172.21.15.23:0/545942667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f351e00 msgr2=0x564e6f364b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.590+0000 7f124ea62740 1 --2- 172.21.15.23:0/545942667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f351e00 0x564e6f364b50 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f123402c700 tx=0x7f1234005bd0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.590+0000 7f124ea62740 1 -- 172.21.15.23:0/545942667 shutdown_connections 2024-03-28T05:24:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.590+0000 7f124ea62740 1 --2- 172.21.15.23:0/545942667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e6f351e00 0x564e6f364b50 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.590+0000 7f124ea62740 1 --2- 172.21.15.23:0/545942667 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x564e6f3bdad0 0x564e6f3bfef0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.590+0000 7f124ea62740 1 --2- 172.21.15.23:0/545942667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e6f349030 0x564e6f3645f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.590+0000 7f124ea62740 1 --2- 172.21.15.23:0/545942667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564e6f348710 0x564e6f367260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.590+0000 7f12297fa640 0 client.15522 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:24:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.590+0000 7f124ea62740 1 -- 172.21.15.23:0/545942667 >> 172.21.15.23:0/545942667 conn(0x564e6f343030 msgr2=0x564e6f34d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.590+0000 7f124ea62740 1 -- 172.21.15.23:0/545942667 shutdown_connections 2024-03-28T05:24:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.590+0000 7f124ea62740 1 -- 172.21.15.23:0/545942667 wait complete. 2024-03-28T05:24:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.591+0000 7f124ea62740 20 client.15522 trim_cache size 0 max 16384 2024-03-28T05:24:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:24:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:24:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:24:02.591 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:24:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.591+0000 7f124be7e640 1 -- 172.21.15.23:0/3136631950 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f122c0a2ab0 msgr2=0x7f122c0a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.591+0000 7f124be7e640 1 --2- 172.21.15.23:0/3136631950 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f122c0a2ab0 0x7f122c0a4f70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1234005cc0 tx=0x7f123402cec0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.591+0000 7f124be7e640 1 -- 172.21.15.23:0/3136631950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1244068d20 msgr2=0x7f1244197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.591+0000 7f124be7e640 1 --2- 172.21.15.23:0/3136631950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1244068d20 0x7f1244197f80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f123c00e9f0 tx=0x7f123c00eeb0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.591+0000 7f124be7e640 1 -- 172.21.15.23:0/3136631950 shutdown_connections 2024-03-28T05:24:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.591+0000 7f124be7e640 1 --2- 172.21.15.23:0/3136631950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1244107680 0x7f124419c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.591+0000 7f124be7e640 1 --2- 172.21.15.23:0/3136631950 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f122c0a2ab0 0x7f122c0a4f70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.591+0000 7f124be7e640 1 --2- 172.21.15.23:0/3136631950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12440696a0 0x7f12441984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.591+0000 7f124be7e640 1 --2- 172.21.15.23:0/3136631950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1244068d20 0x7f1244197f80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.591+0000 7f124be7e640 1 -- 172.21.15.23:0/3136631950 >> 172.21.15.23:0/3136631950 conn(0x7f12440773f0 msgr2=0x7f12440fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.591+0000 7f124be7e640 1 -- 172.21.15.23:0/3136631950 shutdown_connections 2024-03-28T05:24:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:02.591+0000 7f124be7e640 1 -- 172.21.15.23:0/3136631950 wait complete. 2024-03-28T05:24:04.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:03 smithi098 ceph-mon[36087]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-28T05:24:04.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:03 smithi098 ceph-mon[36087]: mgrmap e25: x(active, since 28s), standbys: y, z 2024-03-28T05:24:04.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:03 smithi113 ceph-mon[37164]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-28T05:24:04.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:03 smithi113 ceph-mon[37164]: mgrmap e25: x(active, since 28s), standbys: y, z 2024-03-28T05:24:04.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:03 smithi023 ceph-mon[31954]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-28T05:24:04.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:03 smithi023 ceph-mon[31954]: mgrmap e25: x(active, since 28s), standbys: y, z 2024-03-28T05:24:06.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:05 smithi098 ceph-mon[36087]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:24:06.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:05 smithi113 ceph-mon[37164]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:24:06.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:05 smithi023 ceph-mon[31954]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:24:08.083 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:07 smithi023 ceph-mon[31954]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 19 op/s 2024-03-28T05:24:08.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:07 smithi098 ceph-mon[36087]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 19 op/s 2024-03-28T05:24:08.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:07 smithi113 ceph-mon[37164]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 19 op/s 2024-03-28T05:24:10.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:09 smithi098 ceph-mon[36087]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 18 op/s 2024-03-28T05:24:10.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:09 smithi113 ceph-mon[37164]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 18 op/s 2024-03-28T05:24:10.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:09 smithi023 ceph-mon[31954]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 18 op/s 2024-03-28T05:24:12.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:11 smithi098 ceph-mon[36087]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 18 op/s 2024-03-28T05:24:12.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:11 smithi113 ceph-mon[37164]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 18 op/s 2024-03-28T05:24:12.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:11 smithi023 ceph-mon[31954]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 24 KiB/s wr, 18 op/s 2024-03-28T05:24:14.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:13 smithi098 ceph-mon[36087]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 19 op/s 2024-03-28T05:24:14.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:13 smithi113 ceph-mon[37164]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 19 op/s 2024-03-28T05:24:14.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:13 smithi023 ceph-mon[31954]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 19 op/s 2024-03-28T05:24:15.683 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.682+0000 7fc53837e640 1 -- 172.21.15.23:0/4089089886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5301007f0 msgr2=0x7fc530100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:15.683 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.682+0000 7fc53837e640 1 --2- 172.21.15.23:0/4089089886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5301007f0 0x7fc530100c70 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc520009900 tx=0x7fc52002bc30 comp rx=0 tx=0).stop 2024-03-28T05:24:15.684 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.682+0000 7fc53837e640 1 -- 172.21.15.23:0/4089089886 shutdown_connections 2024-03-28T05:24:15.684 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.682+0000 7fc53837e640 1 --2- 172.21.15.23:0/4089089886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5301011b0 0x7fc53010d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.684 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.682+0000 7fc53837e640 1 --2- 172.21.15.23:0/4089089886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5301007f0 0x7fc530100c70 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.684 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.682+0000 7fc53837e640 1 --2- 172.21.15.23:0/4089089886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc530105540 0x7fc530105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.684 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.682+0000 7fc53837e640 1 -- 172.21.15.23:0/4089089886 >> 172.21.15.23:0/4089089886 conn(0x7fc5300773f0 msgr2=0x7fc530077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:15.684 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.682+0000 7fc53837e640 1 -- 172.21.15.23:0/4089089886 shutdown_connections 2024-03-28T05:24:15.684 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.682+0000 7fc53837e640 1 -- 172.21.15.23:0/4089089886 wait complete. 2024-03-28T05:24:15.684 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.683+0000 7fc53837e640 1 Processor -- start 2024-03-28T05:24:15.684 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.683+0000 7fc53837e640 1 -- start start 2024-03-28T05:24:15.684 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.683+0000 7fc53837e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5301007f0 0x7fc530197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:15.684 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.683+0000 7fc53837e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5301011b0 0x7fc5301984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:15.684 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.683+0000 7fc53837e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc530105540 0x7fc53019c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:15.684 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.683+0000 7fc53837e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc530110a60 con 0x7fc5301007f0 2024-03-28T05:24:15.684 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.683+0000 7fc53837e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5301108e0 con 0x7fc530105540 2024-03-28T05:24:15.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.683+0000 7fc53837e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc530110be0 con 0x7fc5301011b0 2024-03-28T05:24:15.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.683+0000 7fc53611a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5301007f0 0x7fc530197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:15.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.683+0000 7fc53611a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5301007f0 0x7fc530197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41896/0 (socket says 172.21.15.23:41896) 2024-03-28T05:24:15.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.683+0000 7fc53611a640 1 -- 172.21.15.23:0/1515686525 learned_addr learned my addr 172.21.15.23:0/1515686525 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:15.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.683+0000 7fc535919640 1 --2- 172.21.15.23:0/1515686525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5301011b0 0x7fc5301984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:15.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.683+0000 7fc53691b640 1 --2- 172.21.15.23:0/1515686525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc530105540 0x7fc53019c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:15.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.684+0000 7fc53611a640 1 -- 172.21.15.23:0/1515686525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5301011b0 msgr2=0x7fc5301984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:15.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.684+0000 7fc53611a640 1 --2- 172.21.15.23:0/1515686525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5301011b0 0x7fc5301984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.684+0000 7fc53611a640 1 -- 172.21.15.23:0/1515686525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc530105540 msgr2=0x7fc53019c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:15.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.684+0000 7fc53611a640 1 --2- 172.21.15.23:0/1515686525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc530105540 0x7fc53019c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.684+0000 7fc53611a640 1 -- 172.21.15.23:0/1515686525 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc53019cf40 con 0x7fc5301007f0 2024-03-28T05:24:15.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.684+0000 7fc535919640 1 --2- 172.21.15.23:0/1515686525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5301011b0 0x7fc5301984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:24:15.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.684+0000 7fc53611a640 1 --2- 172.21.15.23:0/1515686525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5301007f0 0x7fc530197fa0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00e9f0 tx=0x7fc52c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:15.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.684+0000 7fc5277fe640 1 -- 172.21.15.23:0/1515686525 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c00cd80 con 0x7fc5301007f0 2024-03-28T05:24:15.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.684+0000 7fc5277fe640 1 -- 172.21.15.23:0/1515686525 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc52c005500 con 0x7fc5301007f0 2024-03-28T05:24:15.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.684+0000 7fc53837e640 1 -- 172.21.15.23:0/1515686525 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc53019d230 con 0x7fc5301007f0 2024-03-28T05:24:15.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.684+0000 7fc5277fe640 1 -- 172.21.15.23:0/1515686525 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c010690 con 0x7fc5301007f0 2024-03-28T05:24:15.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.684+0000 7fc53837e640 1 -- 172.21.15.23:0/1515686525 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5301a4ac0 con 0x7fc5301007f0 2024-03-28T05:24:15.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.687+0000 7fc53af6a740 1 Processor -- start 2024-03-28T05:24:15.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.687+0000 7fc53af6a740 1 -- start start 2024-03-28T05:24:15.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.687+0000 7fc53af6a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6c710 0x55af46f6caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:15.692 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.687+0000 7fc53af6a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af46f6d030 0x55af46f758c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.687+0000 7fc53af6a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af46f75e00 0x55af46f781f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.687+0000 7fc53af6a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af46ed0b10 con 0x55af46f6d030 2024-03-28T05:24:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.687+0000 7fc53af6a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af46e79900 con 0x55af46f75e00 2024-03-28T05:24:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.687+0000 7fc53af6a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af46eccbb0 con 0x55af46f6c710 2024-03-28T05:24:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.687+0000 7fc53691b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6c710 0x55af46f6caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.687+0000 7fc53691b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6c710 0x55af46f6caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:51760/0 (socket says 172.21.15.23:51760) 2024-03-28T05:24:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.687+0000 7fc53691b640 1 -- 172.21.15.23:0/348711 learned_addr learned my addr 172.21.15.23:0/348711 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.687+0000 7fc5277fe640 1 -- 172.21.15.23:0/1515686525 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7fc52c0046c0 con 0x7fc5301007f0 2024-03-28T05:24:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.687+0000 7fc535919640 1 --2- 172.21.15.23:0/348711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af46f75e00 0x55af46f781f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.688+0000 7fc53691b640 1 -- 172.21.15.23:0/348711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af46f75e00 msgr2=0x55af46f781f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.688+0000 7fc53691b640 1 --2- 172.21.15.23:0/348711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af46f75e00 0x55af46f781f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.688+0000 7fc53691b640 1 -- 172.21.15.23:0/348711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af46f6d030 msgr2=0x55af46f758c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.688+0000 7fc53691b640 1 --2- 172.21.15.23:0/348711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af46f6d030 0x55af46f758c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.688+0000 7fc53691b640 1 -- 172.21.15.23:0/348711 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af46be5210 con 0x55af46f6c710 2024-03-28T05:24:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.688+0000 7fc53691b640 1 --2- 172.21.15.23:0/348711 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6c710 0x55af46f6caf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc528009fc0 tx=0x7fc5280118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1bd1cb826603f49 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.688+0000 7fc5277fe640 1 --2- 172.21.15.23:0/1515686525 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fc51c0a2bb0 0x7fc51c0a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.688+0000 7fc524bfe640 1 -- 172.21.15.23:0/348711 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc528004d20 con 0x55af46f6c710 2024-03-28T05:24:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.688+0000 7fc524bfe640 1 -- 172.21.15.23:0/348711 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc528004050 con 0x55af46f6c710 2024-03-28T05:24:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.688+0000 7fc524bfe640 1 -- 172.21.15.23:0/348711 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc528005240 con 0x55af46f6c710 2024-03-28T05:24:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.688+0000 7fc53611a640 1 --2- 172.21.15.23:0/1515686525 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fc51c0a2bb0 0x7fc51c0a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.688+0000 7fc5277fe640 1 -- 172.21.15.23:0/1515686525 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fc52c0cf820 con 0x7fc5301007f0 2024-03-28T05:24:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.689+0000 7fc53af6a740 1 -- 172.21.15.23:0/348711 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6c710 msgr2=0x55af46f6caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.689+0000 7fc53af6a740 1 --2- 172.21.15.23:0/348711 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6c710 0x55af46f6caf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc528009fc0 tx=0x7fc5280118d0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.689+0000 7fc53611a640 1 --2- 172.21.15.23:0/1515686525 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fc51c0a2bb0 0x7fc51c0a5070 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc52c0ccac0 tx=0x7fc52c004a40 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.689+0000 7fc53af6a740 1 -- 172.21.15.23:0/348711 shutdown_connections 2024-03-28T05:24:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.689+0000 7fc53af6a740 1 --2- 172.21.15.23:0/348711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af46f75e00 0x55af46f781f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.689+0000 7fc53af6a740 1 --2- 172.21.15.23:0/348711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af46f6d030 0x55af46f758c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.689+0000 7fc53af6a740 1 --2- 172.21.15.23:0/348711 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6c710 0x55af46f6caf0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.689+0000 7fc53af6a740 1 -- 172.21.15.23:0/348711 >> 172.21.15.23:0/348711 conn(0x55af46f67030 msgr2=0x55af46f67410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.689+0000 7fc53af6a740 1 -- 172.21.15.23:0/348711 shutdown_connections 2024-03-28T05:24:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.689+0000 7fc53af6a740 1 -- 172.21.15.23:0/348711 wait complete. 2024-03-28T05:24:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.689+0000 7fc53af6a740 1 -- start start 2024-03-28T05:24:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.689+0000 7fc53af6a740 1 Processor -- start 2024-03-28T05:24:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.690+0000 7fc53af6a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af46f6c710 0x55af46f72e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.690+0000 7fc53af6a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6d030 0x55af46f73350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.690+0000 7fc53af6a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af46f75e00 0x55af46f73890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.690+0000 7fc53af6a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af46f78a30 con 0x55af46f75e00 2024-03-28T05:24:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.690+0000 7fc53af6a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af46e8cd50 con 0x55af46f6c710 2024-03-28T05:24:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.690+0000 7fc53af6a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af46f78bb0 con 0x55af46f6d030 2024-03-28T05:24:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.690+0000 7fc53af6a740 10 client.0 _ll_register_callbacks cb 0x55af46e8c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:24:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.690+0000 7fc53691b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af46f75e00 0x55af46f73890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.690+0000 7fc53691b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af46f75e00 0x55af46f73890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41914/0 (socket says 172.21.15.23:41914) 2024-03-28T05:24:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.690+0000 7fc53691b640 1 -- 172.21.15.23:0/759942975 learned_addr learned my addr 172.21.15.23:0/759942975 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.690+0000 7fc535919640 1 --2- 172.21.15.23:0/759942975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6d030 0x55af46f73350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:15.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.690+0000 7fc53691b640 1 --2- 172.21.15.23:0/759942975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af46f6c710 0x55af46f72e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:15.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.690+0000 7fc535919640 1 -- 172.21.15.23:0/759942975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af46f6c710 msgr2=0x55af46f72e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:15.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.690+0000 7fc535919640 1 --2- 172.21.15.23:0/759942975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af46f6c710 0x55af46f72e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.690+0000 7fc535919640 1 -- 172.21.15.23:0/759942975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af46f75e00 msgr2=0x55af46f73890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:15.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.690+0000 7fc535919640 1 --2- 172.21.15.23:0/759942975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af46f75e00 0x55af46f73890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.690+0000 7fc535919640 1 -- 172.21.15.23:0/759942975 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55af46cb1620 con 0x55af46f6d030 2024-03-28T05:24:15.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.690+0000 7fc535919640 1 --2- 172.21.15.23:0/759942975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6d030 0x55af46f73350 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc520036040 tx=0x7fc520004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5218355a1e4fe21 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:15.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.690+0000 7fc519ffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-28T05:24:15.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.691+0000 7fc519ffb640 1 -- 172.21.15.23:0/759942975 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52002cab0 con 0x55af46f6d030 2024-03-28T05:24:15.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.691+0000 7fc519ffb640 1 -- 172.21.15.23:0/759942975 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fc52002cc50 con 0x55af46f6d030 2024-03-28T05:24:15.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.691+0000 7fc53af6a740 1 -- 172.21.15.23:0/759942975 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc528004050 con 0x55af46f6d030 2024-03-28T05:24:15.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.691+0000 7fc519ffb640 1 -- 172.21.15.23:0/759942975 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc520004f20 con 0x55af46f6d030 2024-03-28T05:24:15.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.691+0000 7fc53af6a740 10 client.25372 fetch_fsmap learned FSMap version 20 2024-03-28T05:24:15.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.691+0000 7fc53af6a740 1 -- 172.21.15.23:0/759942975 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55af46f84cc0 con 0x55af46f6d030 2024-03-28T05:24:15.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.691+0000 7fc519ffb640 1 -- 172.21.15.23:0/759942975 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc528004050 con 0x55af46f6d030 2024-03-28T05:24:15.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.691+0000 7fc519ffb640 1 -- 172.21.15.23:0/759942975 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fc520046070 con 0x55af46f6d030 2024-03-28T05:24:15.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.691+0000 7fc53af6a740 10 client.25372 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:24:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.691+0000 7fc53af6a740 10 client.25372 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:24:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.691+0000 7fc53af6a740 20 client.25372 populate_metadata read hostname 'smithi023' 2024-03-28T05:24:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.691+0000 7fc53af6a740 1 --2- 172.21.15.23:0/759942975 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55af46f86180 0x55af46f88570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.691+0000 7fc53af6a740 4 client.25372 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:24:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.691+0000 7fc53af6a740 1 -- 172.21.15.23:0/759942975 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55af46ede050 con 0x55af46f86180 2024-03-28T05:24:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.692+0000 7fc53691b640 1 --2- 172.21.15.23:0/759942975 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55af46f86180 0x55af46f88570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.692+0000 7fc53691b640 1 --2- 172.21.15.23:0/759942975 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55af46f86180 0x55af46f88570 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11dc25aed0ebc58f server_cookie=edcab38676797e6e in_seq=0 out_seq=0 2024-03-28T05:24:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.692+0000 7fc519ffb640 10 client.25372 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:24:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.692+0000 7fc519ffb640 1 -- 172.21.15.23:0/759942975 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55af46ede050 con 0x55af46f86180 2024-03-28T05:24:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.692+0000 7fc519ffb640 10 client.25372 handle_command_reply: tid=0 2024-03-28T05:24:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.692+0000 7fc53af6a740 1 client.25372 shutdown 2024-03-28T05:24:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.693+0000 7fc53af6a740 1 -- 172.21.15.23:0/759942975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6d030 msgr2=0x55af46f73350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.693+0000 7fc53af6a740 1 --2- 172.21.15.23:0/759942975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6d030 0x55af46f73350 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc520036040 tx=0x7fc520004270 comp rx=0 tx=0).stop 2024-03-28T05:24:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.693+0000 7fc53af6a740 1 -- 172.21.15.23:0/759942975 shutdown_connections 2024-03-28T05:24:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.693+0000 7fc53af6a740 1 --2- 172.21.15.23:0/759942975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af46f75e00 0x55af46f73890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.693+0000 7fc53af6a740 1 --2- 172.21.15.23:0/759942975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6d030 0x55af46f73350 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.693+0000 7fc53af6a740 1 --2- 172.21.15.23:0/759942975 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55af46f86180 0x55af46f88570 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.693+0000 7fc53af6a740 1 --2- 172.21.15.23:0/759942975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af46f6c710 0x55af46f72e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.693+0000 7fc519ffb640 0 client.25372 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:24:15.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.693+0000 7fc53af6a740 1 -- 172.21.15.23:0/759942975 >> 172.21.15.23:0/759942975 conn(0x55af46f67030 msgr2=0x55af46f68d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:15.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.693+0000 7fc53af6a740 1 -- 172.21.15.23:0/759942975 shutdown_connections 2024-03-28T05:24:15.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.693+0000 7fc53af6a740 1 -- 172.21.15.23:0/759942975 wait complete. 2024-03-28T05:24:15.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.693+0000 7fc53af6a740 20 client.25372 trim_cache size 0 max 16384 2024-03-28T05:24:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.721+0000 7fc53af6a740 1 Processor -- start 2024-03-28T05:24:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.721+0000 7fc53af6a740 1 -- start start 2024-03-28T05:24:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.721+0000 7fc53af6a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6c710 0x55af46f75540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.721+0000 7fc53af6a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af46f6d030 0x55af46f65ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.721+0000 7fc53af6a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af46f75e00 0x55af46f812c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.721+0000 7fc53af6a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5301ac980 con 0x55af46f6d030 2024-03-28T05:24:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.721+0000 7fc53af6a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5301aef40 con 0x55af46f75e00 2024-03-28T05:24:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.721+0000 7fc53af6a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5301acf80 con 0x55af46f6c710 2024-03-28T05:24:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.721+0000 7fc535919640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af46f6d030 0x55af46f65ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.721+0000 7fc53691b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6c710 0x55af46f75540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.721+0000 7fc53691b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6c710 0x55af46f75540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:51772/0 (socket says 172.21.15.23:51772) 2024-03-28T05:24:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.721+0000 7fc535919640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af46f6d030 0x55af46f65ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41930/0 (socket says 172.21.15.23:41930) 2024-03-28T05:24:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.721+0000 7fc53691b640 1 -- 172.21.15.23:0/350187018 learned_addr learned my addr 172.21.15.23:0/350187018 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.721+0000 7fc53691b640 1 --2- 172.21.15.23:0/350187018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af46f75e00 0x55af46f812c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.721+0000 7fc53691b640 1 -- 172.21.15.23:0/350187018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af46f75e00 msgr2=0x55af46f812c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.721+0000 7fc53691b640 1 --2- 172.21.15.23:0/350187018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af46f75e00 0x55af46f812c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.721+0000 7fc53691b640 1 -- 172.21.15.23:0/350187018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af46f6d030 msgr2=0x55af46f65ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.721+0000 7fc53691b640 1 --2- 172.21.15.23:0/350187018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af46f6d030 0x55af46f65ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.721+0000 7fc53691b640 1 -- 172.21.15.23:0/350187018 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af46f88bf0 con 0x55af46f6c710 2024-03-28T05:24:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.722+0000 7fc53691b640 1 --2- 172.21.15.23:0/350187018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6c710 0x55af46f75540 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc528009c00 tx=0x7fc528009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=441b0130816b75fa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.722+0000 7fc51bfff640 1 -- 172.21.15.23:0/350187018 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52802dac0 con 0x55af46f6c710 2024-03-28T05:24:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.722+0000 7fc51bfff640 1 -- 172.21.15.23:0/350187018 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc528003e30 con 0x55af46f6c710 2024-03-28T05:24:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.722+0000 7fc51bfff640 1 -- 172.21.15.23:0/350187018 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52800f840 con 0x55af46f6c710 2024-03-28T05:24:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.722+0000 7fc53af6a740 1 -- 172.21.15.23:0/350187018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6c710 msgr2=0x55af46f75540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.722+0000 7fc53af6a740 1 --2- 172.21.15.23:0/350187018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6c710 0x55af46f75540 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc528009c00 tx=0x7fc528009500 comp rx=0 tx=0).stop 2024-03-28T05:24:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.722+0000 7fc53af6a740 1 -- 172.21.15.23:0/350187018 shutdown_connections 2024-03-28T05:24:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.722+0000 7fc53af6a740 1 --2- 172.21.15.23:0/350187018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af46f75e00 0x55af46f812c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.722+0000 7fc53af6a740 1 --2- 172.21.15.23:0/350187018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af46f6d030 0x55af46f65ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.722+0000 7fc53af6a740 1 --2- 172.21.15.23:0/350187018 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6c710 0x55af46f75540 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.722+0000 7fc53af6a740 1 -- 172.21.15.23:0/350187018 >> 172.21.15.23:0/350187018 conn(0x55af46f67030 msgr2=0x55af46f65790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.722+0000 7fc53af6a740 1 -- 172.21.15.23:0/350187018 shutdown_connections 2024-03-28T05:24:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.722+0000 7fc53af6a740 1 -- 172.21.15.23:0/350187018 wait complete. 2024-03-28T05:24:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.723+0000 7fc53af6a740 1 -- start start 2024-03-28T05:24:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.723+0000 7fc53af6a740 1 Processor -- start 2024-03-28T05:24:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.723+0000 7fc53af6a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6c710 0x55af46f79da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.723+0000 7fc53af6a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af46f6d030 0x55af46f7a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.723+0000 7fc53af6a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af46f75e00 0x55af46fd9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.723+0000 7fc53af6a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5301ad5c0 con 0x55af46f75e00 2024-03-28T05:24:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.723+0000 7fc53af6a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5301ae8e0 con 0x55af46f6d030 2024-03-28T05:24:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.723+0000 7fc53af6a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5301adc20 con 0x55af46f6c710 2024-03-28T05:24:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.723+0000 7fc53af6a740 10 client.0 _ll_register_callbacks cb 0x7fc52002c5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:24:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.723+0000 7fc53691b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6c710 0x55af46f79da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.723+0000 7fc53691b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6c710 0x55af46f79da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:51782/0 (socket says 172.21.15.23:51782) 2024-03-28T05:24:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.723+0000 7fc53691b640 1 -- 172.21.15.23:0/2261723890 learned_addr learned my addr 172.21.15.23:0/2261723890 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.723+0000 7fc53691b640 1 --2- 172.21.15.23:0/2261723890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af46f75e00 0x55af46fd9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.723+0000 7fc535919640 1 --2- 172.21.15.23:0/2261723890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af46f6d030 0x55af46f7a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.723+0000 7fc53691b640 1 -- 172.21.15.23:0/2261723890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af46f6d030 msgr2=0x55af46f7a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.723+0000 7fc53691b640 1 --2- 172.21.15.23:0/2261723890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af46f6d030 0x55af46f7a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.724+0000 7fc53691b640 1 -- 172.21.15.23:0/2261723890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af46f75e00 msgr2=0x55af46fd9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:15.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.724+0000 7fc53691b640 1 --2- 172.21.15.23:0/2261723890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af46f75e00 0x55af46fd9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.724+0000 7fc53691b640 1 -- 172.21.15.23:0/2261723890 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55af46fda1e0 con 0x55af46f6c710 2024-03-28T05:24:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.724+0000 7fc53691b640 1 --2- 172.21.15.23:0/2261723890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6c710 0x55af46f79da0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc528002230 tx=0x7fc52800f4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97b40e41fd5a4c3b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.724+0000 7fc519ffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-28T05:24:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.724+0000 7fc519ffb640 1 -- 172.21.15.23:0/2261723890 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52802d8c0 con 0x55af46f6c710 2024-03-28T05:24:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.724+0000 7fc53af6a740 1 -- 172.21.15.23:0/2261723890 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc528003e30 con 0x55af46f6c710 2024-03-28T05:24:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.724+0000 7fc519ffb640 1 -- 172.21.15.23:0/2261723890 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fc528002030 con 0x55af46f6c710 2024-03-28T05:24:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.724+0000 7fc519ffb640 1 -- 172.21.15.23:0/2261723890 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc52802da60 con 0x55af46f6c710 2024-03-28T05:24:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.724+0000 7fc53af6a740 10 client.25378 fetch_fsmap learned FSMap version 20 2024-03-28T05:24:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.724+0000 7fc53af6a740 1 -- 172.21.15.23:0/2261723890 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55af46fe2550 con 0x55af46f6c710 2024-03-28T05:24:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.724+0000 7fc519ffb640 1 -- 172.21.15.23:0/2261723890 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc528003e30 con 0x55af46f6c710 2024-03-28T05:24:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.725+0000 7fc519ffb640 1 -- 172.21.15.23:0/2261723890 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fc528025040 con 0x55af46f6c710 2024-03-28T05:24:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.725+0000 7fc53af6a740 10 client.25378 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:24:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.725+0000 7fc53af6a740 10 client.25378 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:24:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.725+0000 7fc53af6a740 20 client.25378 populate_metadata read hostname 'smithi023' 2024-03-28T05:24:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.725+0000 7fc53af6a740 1 --2- 172.21.15.23:0/2261723890 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55af46fe28d0 0x55af46fe4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.725+0000 7fc53af6a740 4 client.25378 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:24:15.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.725+0000 7fc53af6a740 1 -- 172.21.15.23:0/2261723890 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55af46ed83a0 con 0x55af46fe28d0 2024-03-28T05:24:15.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.725+0000 7fc535919640 1 --2- 172.21.15.23:0/2261723890 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55af46fe28d0 0x55af46fe4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:15.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.725+0000 7fc519ffb640 10 client.25378 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:24:15.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.725+0000 7fc535919640 1 --2- 172.21.15.23:0/2261723890 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55af46fe28d0 0x55af46fe4cc0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1006d3da2097326 server_cookie=b3c44cf7ea23ed46 in_seq=0 out_seq=0 2024-03-28T05:24:15.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.732+0000 7fc519ffb640 1 -- 172.21.15.23:0/2261723890 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55af46ed83a0 con 0x55af46fe28d0 2024-03-28T05:24:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.732+0000 7fc519ffb640 10 client.25378 handle_command_reply: tid=0 2024-03-28T05:24:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.732+0000 7fc53af6a740 1 client.25378 shutdown 2024-03-28T05:24:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.732+0000 7fc53af6a740 1 -- 172.21.15.23:0/2261723890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6c710 msgr2=0x55af46f79da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.732+0000 7fc53af6a740 1 --2- 172.21.15.23:0/2261723890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6c710 0x55af46f79da0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc528002230 tx=0x7fc52800f4e0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.732+0000 7fc53af6a740 1 -- 172.21.15.23:0/2261723890 shutdown_connections 2024-03-28T05:24:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.732+0000 7fc53af6a740 1 --2- 172.21.15.23:0/2261723890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55af46f75e00 0x55af46fd9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.732+0000 7fc53af6a740 1 --2- 172.21.15.23:0/2261723890 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55af46fe28d0 0x55af46fe4cc0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.732+0000 7fc53af6a740 1 --2- 172.21.15.23:0/2261723890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af46f6d030 0x55af46f7a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.732+0000 7fc53af6a740 1 --2- 172.21.15.23:0/2261723890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55af46f6c710 0x55af46f79da0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.732+0000 7fc519ffb640 0 client.25378 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:24:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.732+0000 7fc53af6a740 1 -- 172.21.15.23:0/2261723890 >> 172.21.15.23:0/2261723890 conn(0x55af46f67030 msgr2=0x55af46f66020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.732+0000 7fc53af6a740 1 -- 172.21.15.23:0/2261723890 shutdown_connections 2024-03-28T05:24:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.732+0000 7fc53af6a740 1 -- 172.21.15.23:0/2261723890 wait complete. 2024-03-28T05:24:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.733+0000 7fc53af6a740 20 client.25378 trim_cache size 0 max 16384 2024-03-28T05:24:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:24:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:24:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:24:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:24:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.733+0000 7fc53837e640 1 -- 172.21.15.23:0/1515686525 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fc51c0a2bb0 msgr2=0x7fc51c0a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.733+0000 7fc53837e640 1 --2- 172.21.15.23:0/1515686525 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fc51c0a2bb0 0x7fc51c0a5070 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc52c0ccac0 tx=0x7fc52c004a40 comp rx=0 tx=0).stop 2024-03-28T05:24:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.733+0000 7fc53837e640 1 -- 172.21.15.23:0/1515686525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5301007f0 msgr2=0x7fc530197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.733+0000 7fc53837e640 1 --2- 172.21.15.23:0/1515686525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5301007f0 0x7fc530197fa0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00e9f0 tx=0x7fc52c00edf0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.733+0000 7fc53837e640 1 -- 172.21.15.23:0/1515686525 shutdown_connections 2024-03-28T05:24:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.733+0000 7fc53837e640 1 --2- 172.21.15.23:0/1515686525 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fc51c0a2bb0 0x7fc51c0a5070 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.733+0000 7fc53837e640 1 --2- 172.21.15.23:0/1515686525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc530105540 0x7fc53019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.733+0000 7fc53837e640 1 --2- 172.21.15.23:0/1515686525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc5301011b0 0x7fc5301984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.733+0000 7fc53837e640 1 --2- 172.21.15.23:0/1515686525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5301007f0 0x7fc530197fa0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.733+0000 7fc53837e640 1 -- 172.21.15.23:0/1515686525 >> 172.21.15.23:0/1515686525 conn(0x7fc5300773f0 msgr2=0x7fc530068fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.733+0000 7fc53837e640 1 -- 172.21.15.23:0/1515686525 shutdown_connections 2024-03-28T05:24:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:15.733+0000 7fc53837e640 1 -- 172.21.15.23:0/1515686525 wait complete. 2024-03-28T05:24:16.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:15 smithi098 ceph-mon[36087]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 KiB/s wr, 18 op/s 2024-03-28T05:24:16.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:15 smithi113 ceph-mon[37164]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 KiB/s wr, 18 op/s 2024-03-28T05:24:16.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:15 smithi023 ceph-mon[31954]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 25 KiB/s wr, 18 op/s 2024-03-28T05:24:18.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:17 smithi098 ceph-mon[36087]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 19 op/s 2024-03-28T05:24:18.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:17 smithi113 ceph-mon[37164]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 19 op/s 2024-03-28T05:24:18.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:17 smithi023 ceph-mon[31954]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 19 op/s 2024-03-28T05:24:20.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:19 smithi098 ceph-mon[36087]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2024-03-28T05:24:20.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:19 smithi113 ceph-mon[37164]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2024-03-28T05:24:20.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:19 smithi023 ceph-mon[31954]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2024-03-28T05:24:22.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:21 smithi098 ceph-mon[36087]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.3 KiB/s wr, 4 op/s 2024-03-28T05:24:22.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:21 smithi113 ceph-mon[37164]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.3 KiB/s wr, 4 op/s 2024-03-28T05:24:22.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:21 smithi023 ceph-mon[31954]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.3 KiB/s wr, 4 op/s 2024-03-28T05:24:24.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:23 smithi098 ceph-mon[36087]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.4 KiB/s wr, 4 op/s 2024-03-28T05:24:24.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:23 smithi113 ceph-mon[37164]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.4 KiB/s wr, 4 op/s 2024-03-28T05:24:24.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:23 smithi023 ceph-mon[31954]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.4 KiB/s wr, 4 op/s 2024-03-28T05:24:26.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:25 smithi113 ceph-mon[37164]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-28T05:24:26.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:25 smithi023 ceph-mon[31954]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-28T05:24:26.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:25 smithi098 ceph-mon[36087]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-28T05:24:28.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:27 smithi113 ceph-mon[37164]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-28T05:24:28.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:27 smithi023 ceph-mon[31954]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-28T05:24:28.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:27 smithi098 ceph-mon[36087]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-28T05:24:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.858+0000 7f2565cea640 1 -- 172.21.15.23:0/1397410175 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2560101650 msgr2=0x7f2560101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.858+0000 7f2565cea640 1 --2- 172.21.15.23:0/1397410175 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2560101650 0x7f2560101ad0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2554009930 tx=0x7f255402bc30 comp rx=0 tx=0).stop 2024-03-28T05:24:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.858+0000 7f2565cea640 1 -- 172.21.15.23:0/1397410175 shutdown_connections 2024-03-28T05:24:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.858+0000 7f2565cea640 1 --2- 172.21.15.23:0/1397410175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2560102010 0x7f256010e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.858+0000 7f2565cea640 1 --2- 172.21.15.23:0/1397410175 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2560101650 0x7f2560101ad0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.858+0000 7f2565cea640 1 --2- 172.21.15.23:0/1397410175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2560107660 0x7f2560107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.858+0000 7f2565cea640 1 -- 172.21.15.23:0/1397410175 >> 172.21.15.23:0/1397410175 conn(0x7f25600fd120 msgr2=0x7f25600ff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.858+0000 7f2565cea640 1 -- 172.21.15.23:0/1397410175 shutdown_connections 2024-03-28T05:24:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.858+0000 7f2565cea640 1 -- 172.21.15.23:0/1397410175 wait complete. 2024-03-28T05:24:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.859+0000 7f2565cea640 1 Processor -- start 2024-03-28T05:24:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.859+0000 7f2565cea640 1 -- start start 2024-03-28T05:24:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.859+0000 7f2565cea640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2560101650 0x7f2560197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.859+0000 7f2565cea640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2560102010 0x7f2560198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.859+0000 7f2565cea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2560107660 0x7f256019c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.859+0000 7f255effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2560102010 0x7f2560198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.859+0000 7f2565cea640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2560111a70 con 0x7f2560102010 2024-03-28T05:24:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.859+0000 7f2565cea640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25601118f0 con 0x7f2560107660 2024-03-28T05:24:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.859+0000 7f255effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2560102010 0x7f2560198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59944/0 (socket says 172.21.15.23:59944) 2024-03-28T05:24:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.859+0000 7f2565cea640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2560111bf0 con 0x7f2560101650 2024-03-28T05:24:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.859+0000 7f255effd640 1 -- 172.21.15.23:0/494469949 learned_addr learned my addr 172.21.15.23:0/494469949 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.859+0000 7f255f7fe640 1 --2- 172.21.15.23:0/494469949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2560101650 0x7f2560197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.860+0000 7f255ffff640 1 --2- 172.21.15.23:0/494469949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2560107660 0x7f256019c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.860+0000 7f255effd640 1 -- 172.21.15.23:0/494469949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2560101650 msgr2=0x7f2560197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.860+0000 7f255effd640 1 --2- 172.21.15.23:0/494469949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2560101650 0x7f2560197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.860+0000 7f255effd640 1 -- 172.21.15.23:0/494469949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2560107660 msgr2=0x7f256019c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.860+0000 7f255effd640 1 --2- 172.21.15.23:0/494469949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2560107660 0x7f256019c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.860+0000 7f255effd640 1 -- 172.21.15.23:0/494469949 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f256019cf00 con 0x7f2560102010 2024-03-28T05:24:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.860+0000 7f255ffff640 1 --2- 172.21.15.23:0/494469949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2560107660 0x7f256019c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:24:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.860+0000 7f255effd640 1 --2- 172.21.15.23:0/494469949 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2560102010 0x7f2560198450 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2554009230 tx=0x7f255402cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.860+0000 7f255cff9640 1 -- 172.21.15.23:0/494469949 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2554041070 con 0x7f2560102010 2024-03-28T05:24:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.860+0000 7f2565cea640 1 -- 172.21.15.23:0/494469949 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f256019d190 con 0x7f2560102010 2024-03-28T05:24:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.860+0000 7f255cff9640 1 -- 172.21.15.23:0/494469949 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2554004450 con 0x7f2560102010 2024-03-28T05:24:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.860+0000 7f255cff9640 1 -- 172.21.15.23:0/494469949 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2554034440 con 0x7f2560102010 2024-03-28T05:24:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.861+0000 7f2565cea640 1 -- 172.21.15.23:0/494469949 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25601a4a80 con 0x7f2560102010 2024-03-28T05:24:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.862+0000 7f255cff9640 1 -- 172.21.15.23:0/494469949 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f2554044070 con 0x7f2560102010 2024-03-28T05:24:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.863+0000 7f255cff9640 1 --2- 172.21.15.23:0/494469949 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f25480a2b30 0x7f25480a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.863+0000 7f255f7fe640 1 --2- 172.21.15.23:0/494469949 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f25480a2b30 0x7f25480a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.863+0000 7f255cff9640 1 -- 172.21.15.23:0/494469949 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f25540f0e90 con 0x7f2560102010 2024-03-28T05:24:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.864+0000 7f255f7fe640 1 --2- 172.21.15.23:0/494469949 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f25480a2b30 0x7f25480a4ff0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f255000bb60 tx=0x7f255000b390 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.864+0000 7f25689b2740 1 Processor -- start 2024-03-28T05:24:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.864+0000 7f25689b2740 1 -- start start 2024-03-28T05:24:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.864+0000 7f25689b2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c953710 0x56362c953af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.864+0000 7f25689b2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56362c954030 0x56362c95c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.865+0000 7f25689b2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56362c95ce00 0x56362c95f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.865+0000 7f25689b2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56362c8b7b10 con 0x56362c953710 2024-03-28T05:24:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.865+0000 7f255f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c953710 0x56362c953af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.865+0000 7f255f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c953710 0x56362c953af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59954/0 (socket says 172.21.15.23:59954) 2024-03-28T05:24:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.865+0000 7f255f7fe640 1 -- 172.21.15.23:0/2940268540 learned_addr learned my addr 172.21.15.23:0/2940268540 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.865+0000 7f25689b2740 1 -- 172.21.15.23:0/2940268540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x56362c860900 con 0x56362c95ce00 2024-03-28T05:24:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.865+0000 7f25689b2740 1 -- 172.21.15.23:0/2940268540 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x56362c8b3bb0 con 0x56362c954030 2024-03-28T05:24:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.865+0000 7f255effd640 1 --2- 172.21.15.23:0/2940268540 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56362c954030 0x56362c95c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.865+0000 7f255ffff640 1 --2- 172.21.15.23:0/2940268540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56362c95ce00 0x56362c95f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.865+0000 7f255f7fe640 1 -- 172.21.15.23:0/2940268540 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56362c954030 msgr2=0x56362c95c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.865+0000 7f255f7fe640 1 --2- 172.21.15.23:0/2940268540 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56362c954030 0x56362c95c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.865+0000 7f255f7fe640 1 -- 172.21.15.23:0/2940268540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56362c95ce00 msgr2=0x56362c95f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.865+0000 7f255f7fe640 1 --2- 172.21.15.23:0/2940268540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56362c95ce00 0x56362c95f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.865+0000 7f255f7fe640 1 -- 172.21.15.23:0/2940268540 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56362c5cc210 con 0x56362c953710 2024-03-28T05:24:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.865+0000 7f255ffff640 1 --2- 172.21.15.23:0/2940268540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56362c95ce00 0x56362c95f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:24:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.865+0000 7f255effd640 1 --2- 172.21.15.23:0/2940268540 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56362c954030 0x56362c95c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:24:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.865+0000 7f255f7fe640 1 --2- 172.21.15.23:0/2940268540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c953710 0x56362c953af0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2550010fa0 tx=0x7f2550013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f7641ff3bf64b49 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.865+0000 7f2545ffb640 1 -- 172.21.15.23:0/2940268540 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255001e620 con 0x56362c953710 2024-03-28T05:24:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.865+0000 7f2545ffb640 1 -- 172.21.15.23:0/2940268540 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f255001eca0 con 0x56362c953710 2024-03-28T05:24:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.865+0000 7f2545ffb640 1 -- 172.21.15.23:0/2940268540 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255001cc80 con 0x56362c953710 2024-03-28T05:24:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.866+0000 7f25689b2740 1 -- 172.21.15.23:0/2940268540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c953710 msgr2=0x56362c953af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.866+0000 7f25689b2740 1 --2- 172.21.15.23:0/2940268540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c953710 0x56362c953af0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2550010fa0 tx=0x7f2550013000 comp rx=0 tx=0).stop 2024-03-28T05:24:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.866+0000 7f25689b2740 1 -- 172.21.15.23:0/2940268540 shutdown_connections 2024-03-28T05:24:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.866+0000 7f25689b2740 1 --2- 172.21.15.23:0/2940268540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56362c95ce00 0x56362c95f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.866+0000 7f25689b2740 1 --2- 172.21.15.23:0/2940268540 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56362c954030 0x56362c95c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.866+0000 7f25689b2740 1 --2- 172.21.15.23:0/2940268540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c953710 0x56362c953af0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.866+0000 7f25689b2740 1 -- 172.21.15.23:0/2940268540 >> 172.21.15.23:0/2940268540 conn(0x56362c94e030 msgr2=0x56362c94e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.866+0000 7f25689b2740 1 -- 172.21.15.23:0/2940268540 shutdown_connections 2024-03-28T05:24:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.866+0000 7f25689b2740 1 -- 172.21.15.23:0/2940268540 wait complete. 2024-03-28T05:24:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.866+0000 7f25689b2740 1 -- start start 2024-03-28T05:24:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.867+0000 7f25689b2740 1 Processor -- start 2024-03-28T05:24:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.867+0000 7f25689b2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c953710 0x56362c966e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.867+0000 7f25689b2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56362c954030 0x56362c967340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.867+0000 7f25689b2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56362c95ce00 0x56362c961210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.867+0000 7f25689b2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56362c95fa30 con 0x56362c953710 2024-03-28T05:24:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.867+0000 7f25689b2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x56362c873d50 con 0x56362c954030 2024-03-28T05:24:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.867+0000 7f25689b2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x56362c95fbb0 con 0x56362c95ce00 2024-03-28T05:24:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.867+0000 7f25689b2740 10 client.0 _ll_register_callbacks cb 0x56362c873720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:24:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.867+0000 7f255f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c953710 0x56362c966e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.867+0000 7f255f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c953710 0x56362c966e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59964/0 (socket says 172.21.15.23:59964) 2024-03-28T05:24:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.867+0000 7f255f7fe640 1 -- 172.21.15.23:0/2822230768 learned_addr learned my addr 172.21.15.23:0/2822230768 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.867+0000 7f255ffff640 1 --2- 172.21.15.23:0/2822230768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56362c95ce00 0x56362c961210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.867+0000 7f255effd640 1 --2- 172.21.15.23:0/2822230768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56362c954030 0x56362c967340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.867+0000 7f255f7fe640 1 -- 172.21.15.23:0/2822230768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56362c95ce00 msgr2=0x56362c961210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.867+0000 7f255f7fe640 1 --2- 172.21.15.23:0/2822230768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56362c95ce00 0x56362c961210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.867+0000 7f255f7fe640 1 -- 172.21.15.23:0/2822230768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56362c954030 msgr2=0x56362c967340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.867+0000 7f255f7fe640 1 --2- 172.21.15.23:0/2822230768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56362c954030 0x56362c967340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.867+0000 7f255f7fe640 1 -- 172.21.15.23:0/2822230768 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56362c698620 con 0x56362c953710 2024-03-28T05:24:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.868+0000 7f255f7fe640 1 --2- 172.21.15.23:0/2822230768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c953710 0x56362c966e00 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2550010860 tx=0x7f255001c5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d709aea49d2e161d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.868+0000 7f253effd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:24:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.868+0000 7f253effd640 1 -- 172.21.15.23:0/2822230768 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2550036060 con 0x56362c953710 2024-03-28T05:24:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.868+0000 7f25689b2740 1 -- 172.21.15.23:0/2822230768 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f255001eca0 con 0x56362c953710 2024-03-28T05:24:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.868+0000 7f253effd640 1 -- 172.21.15.23:0/2822230768 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f255001cbd0 con 0x56362c953710 2024-03-28T05:24:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.868+0000 7f253effd640 1 -- 172.21.15.23:0/2822230768 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f255003d050 con 0x56362c953710 2024-03-28T05:24:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.868+0000 7f25689b2740 10 client.15549 fetch_fsmap learned FSMap version 20 2024-03-28T05:24:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.868+0000 7f25689b2740 1 -- 172.21.15.23:0/2822230768 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x56362c96fff0 con 0x56362c953710 2024-03-28T05:24:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.868+0000 7f253effd640 1 -- 172.21.15.23:0/2822230768 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f255001eca0 con 0x56362c953710 2024-03-28T05:24:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.868+0000 7f253effd640 1 -- 172.21.15.23:0/2822230768 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f2550026790 con 0x56362c953710 2024-03-28T05:24:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.868+0000 7f25689b2740 10 client.15549 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:24:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.868+0000 7f25689b2740 10 client.15549 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:24:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.868+0000 7f25689b2740 20 client.15549 populate_metadata read hostname 'smithi023' 2024-03-28T05:24:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.869+0000 7f25689b2740 1 --2- 172.21.15.23:0/2822230768 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x56362c971400 0x56362c9737f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.869+0000 7f25689b2740 4 client.15549 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:24:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.869+0000 7f25689b2740 1 -- 172.21.15.23:0/2822230768 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56362c8c5050 con 0x56362c971400 2024-03-28T05:24:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.869+0000 7f255effd640 1 --2- 172.21.15.23:0/2822230768 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x56362c971400 0x56362c9737f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.869+0000 7f255effd640 1 --2- 172.21.15.23:0/2822230768 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x56362c971400 0x56362c9737f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b15cf5f956c089be server_cookie=d517c0b9c0804ed6 in_seq=0 out_seq=0 2024-03-28T05:24:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.869+0000 7f253effd640 10 client.15549 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:24:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.870+0000 7f253effd640 1 -- 172.21.15.23:0/2822230768 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56362c8c5050 con 0x56362c971400 2024-03-28T05:24:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.870+0000 7f253effd640 10 client.15549 handle_command_reply: tid=0 2024-03-28T05:24:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.870+0000 7f25689b2740 1 client.15549 shutdown 2024-03-28T05:24:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.870+0000 7f25689b2740 1 -- 172.21.15.23:0/2822230768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c953710 msgr2=0x56362c966e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.870+0000 7f25689b2740 1 --2- 172.21.15.23:0/2822230768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c953710 0x56362c966e00 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2550010860 tx=0x7f255001c5e0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.870+0000 7f25689b2740 1 -- 172.21.15.23:0/2822230768 shutdown_connections 2024-03-28T05:24:28.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.870+0000 7f25689b2740 1 --2- 172.21.15.23:0/2822230768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56362c95ce00 0x56362c961210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.870+0000 7f25689b2740 1 --2- 172.21.15.23:0/2822230768 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x56362c971400 0x56362c9737f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.870+0000 7f253effd640 0 client.15549 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:24:28.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.870+0000 7f25689b2740 1 --2- 172.21.15.23:0/2822230768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56362c954030 0x56362c967340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.870+0000 7f25689b2740 1 --2- 172.21.15.23:0/2822230768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c953710 0x56362c966e00 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.870+0000 7f25689b2740 1 -- 172.21.15.23:0/2822230768 >> 172.21.15.23:0/2822230768 conn(0x56362c94e030 msgr2=0x56362c94fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:28.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.870+0000 7f25689b2740 1 -- 172.21.15.23:0/2822230768 shutdown_connections 2024-03-28T05:24:28.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.870+0000 7f25689b2740 1 -- 172.21.15.23:0/2822230768 wait complete. 2024-03-28T05:24:28.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.870+0000 7f25689b2740 20 client.15549 trim_cache size 0 max 16384 2024-03-28T05:24:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.885+0000 7f25689b2740 1 Processor -- start 2024-03-28T05:24:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.885+0000 7f25689b2740 1 -- start start 2024-03-28T05:24:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.885+0000 7f25689b2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c953710 0x56362c967060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.885+0000 7f25689b2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56362c954030 0x56362c978820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.885+0000 7f25689b2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56362c95ce00 0x56362c95b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.885+0000 7f25689b2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25601aa890 con 0x56362c953710 2024-03-28T05:24:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.885+0000 7f25689b2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25601ace50 con 0x56362c95ce00 2024-03-28T05:24:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.885+0000 7f25689b2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25601aae90 con 0x56362c954030 2024-03-28T05:24:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.885+0000 7f255f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c953710 0x56362c967060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.885+0000 7f255f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c953710 0x56362c967060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59974/0 (socket says 172.21.15.23:59974) 2024-03-28T05:24:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.885+0000 7f255f7fe640 1 -- 172.21.15.23:0/3738671928 learned_addr learned my addr 172.21.15.23:0/3738671928 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.885+0000 7f255effd640 1 --2- 172.21.15.23:0/3738671928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56362c954030 0x56362c978820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.885+0000 7f255ffff640 1 --2- 172.21.15.23:0/3738671928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56362c95ce00 0x56362c95b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.885+0000 7f255f7fe640 1 -- 172.21.15.23:0/3738671928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56362c954030 msgr2=0x56362c978820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.885+0000 7f255f7fe640 1 --2- 172.21.15.23:0/3738671928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56362c954030 0x56362c978820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.885+0000 7f255f7fe640 1 -- 172.21.15.23:0/3738671928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56362c95ce00 msgr2=0x56362c95b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.885+0000 7f255f7fe640 1 --2- 172.21.15.23:0/3738671928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56362c95ce00 0x56362c95b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.885+0000 7f255f7fe640 1 -- 172.21.15.23:0/3738671928 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56362c95f1f0 con 0x56362c953710 2024-03-28T05:24:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.886+0000 7f255f7fe640 1 --2- 172.21.15.23:0/3738671928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c953710 0x56362c967060 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f255001c8d0 tx=0x7f2550026f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b6da1b987a696 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.886+0000 7f25457fa640 1 -- 172.21.15.23:0/3738671928 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2550036030 con 0x56362c953710 2024-03-28T05:24:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.886+0000 7f25457fa640 1 -- 172.21.15.23:0/3738671928 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f255001ee50 con 0x56362c953710 2024-03-28T05:24:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.886+0000 7f25457fa640 1 -- 172.21.15.23:0/3738671928 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2550026400 con 0x56362c953710 2024-03-28T05:24:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.886+0000 7f25689b2740 1 -- 172.21.15.23:0/3738671928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c953710 msgr2=0x56362c967060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.886+0000 7f25689b2740 1 --2- 172.21.15.23:0/3738671928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c953710 0x56362c967060 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f255001c8d0 tx=0x7f2550026f50 comp rx=0 tx=0).stop 2024-03-28T05:24:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.887+0000 7f25689b2740 1 -- 172.21.15.23:0/3738671928 shutdown_connections 2024-03-28T05:24:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.887+0000 7f25689b2740 1 --2- 172.21.15.23:0/3738671928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56362c95ce00 0x56362c95b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.887+0000 7f25689b2740 1 --2- 172.21.15.23:0/3738671928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56362c954030 0x56362c978820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.887+0000 7f25689b2740 1 --2- 172.21.15.23:0/3738671928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c953710 0x56362c967060 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.887+0000 7f25689b2740 1 -- 172.21.15.23:0/3738671928 >> 172.21.15.23:0/3738671928 conn(0x56362c94e030 msgr2=0x56362c966950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.887+0000 7f25689b2740 1 -- 172.21.15.23:0/3738671928 shutdown_connections 2024-03-28T05:24:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.887+0000 7f25689b2740 1 -- 172.21.15.23:0/3738671928 wait complete. 2024-03-28T05:24:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.887+0000 7f25689b2740 1 -- start start 2024-03-28T05:24:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.887+0000 7f25689b2740 1 Processor -- start 2024-03-28T05:24:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.887+0000 7f25689b2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56362c953710 0x56362c96d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.887+0000 7f25689b2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56362c954030 0x56362c96d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.887+0000 7f25689b2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c95ce00 0x56362c96b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.887+0000 7f25689b2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25601ab4d0 con 0x56362c95ce00 2024-03-28T05:24:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.887+0000 7f25689b2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25601ac7f0 con 0x56362c953710 2024-03-28T05:24:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.887+0000 7f25689b2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25601abb30 con 0x56362c954030 2024-03-28T05:24:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.888+0000 7f25689b2740 10 client.0 _ll_register_callbacks cb 0x7f25500056d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:24:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.888+0000 7f255ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c95ce00 0x56362c96b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.888+0000 7f255ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c95ce00 0x56362c96b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59984/0 (socket says 172.21.15.23:59984) 2024-03-28T05:24:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.888+0000 7f255ffff640 1 -- 172.21.15.23:0/3684104374 learned_addr learned my addr 172.21.15.23:0/3684104374 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.888+0000 7f255effd640 1 --2- 172.21.15.23:0/3684104374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56362c954030 0x56362c96d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.888+0000 7f255f7fe640 1 --2- 172.21.15.23:0/3684104374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56362c953710 0x56362c96d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.888+0000 7f255effd640 1 -- 172.21.15.23:0/3684104374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56362c953710 msgr2=0x56362c96d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.888+0000 7f255effd640 1 --2- 172.21.15.23:0/3684104374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56362c953710 0x56362c96d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.888+0000 7f255effd640 1 -- 172.21.15.23:0/3684104374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c95ce00 msgr2=0x56362c96b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.888+0000 7f255effd640 1 --2- 172.21.15.23:0/3684104374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c95ce00 0x56362c96b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.888+0000 7f255effd640 1 -- 172.21.15.23:0/3684104374 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56362c96bcd0 con 0x56362c954030 2024-03-28T05:24:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.888+0000 7f255effd640 1 --2- 172.21.15.23:0/3684104374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56362c954030 0x56362c96d5e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f25540ee1e0 tx=0x7f255402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd99d910c88fe5b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.888+0000 7f253effd640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-28T05:24:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.888+0000 7f253effd640 1 -- 172.21.15.23:0/3684104374 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255404b040 con 0x56362c954030 2024-03-28T05:24:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.888+0000 7f253effd640 1 -- 172.21.15.23:0/3684104374 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f25540036f0 con 0x56362c954030 2024-03-28T05:24:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.888+0000 7f25689b2740 1 -- 172.21.15.23:0/3684104374 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f255001ee50 con 0x56362c954030 2024-03-28T05:24:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.888+0000 7f253effd640 1 -- 172.21.15.23:0/3684104374 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f25540a5050 con 0x56362c954030 2024-03-28T05:24:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.888+0000 7f25689b2740 10 client.25396 fetch_fsmap learned FSMap version 20 2024-03-28T05:24:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.888+0000 7f25689b2740 1 -- 172.21.15.23:0/3684104374 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x56362c9cd410 con 0x56362c954030 2024-03-28T05:24:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.889+0000 7f253effd640 1 -- 172.21.15.23:0/3684104374 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f25540d8030 con 0x56362c954030 2024-03-28T05:24:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.889+0000 7f253effd640 1 -- 172.21.15.23:0/3684104374 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f25540c5330 con 0x56362c954030 2024-03-28T05:24:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.889+0000 7f25689b2740 10 client.25396 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:24:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.889+0000 7f25689b2740 10 client.25396 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:24:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.889+0000 7f25689b2740 20 client.25396 populate_metadata read hostname 'smithi023' 2024-03-28T05:24:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.889+0000 7f25689b2740 1 --2- 172.21.15.23:0/3684104374 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x56362c971400 0x56362c9cfb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.889+0000 7f25689b2740 4 client.25396 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:24:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.889+0000 7f25689b2740 1 -- 172.21.15.23:0/3684104374 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x7f255001ee50 con 0x56362c971400 2024-03-28T05:24:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.889+0000 7f255f7fe640 1 --2- 172.21.15.23:0/3684104374 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x56362c971400 0x56362c9cfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.889+0000 7f255f7fe640 1 --2- 172.21.15.23:0/3684104374 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x56362c971400 0x56362c9cfb80 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33fd0384fff0cc12 server_cookie=e02f4e2702ae445e in_seq=0 out_seq=0 2024-03-28T05:24:28.893 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.889+0000 7f253effd640 10 client.25396 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:24:28.894 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.893+0000 7f253effd640 1 -- 172.21.15.23:0/3684104374 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x7f255001ee50 con 0x56362c971400 2024-03-28T05:24:28.894 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.893+0000 7f253effd640 10 client.25396 handle_command_reply: tid=0 2024-03-28T05:24:28.894 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.893+0000 7f25689b2740 1 client.25396 shutdown 2024-03-28T05:24:28.894 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.893+0000 7f25689b2740 1 -- 172.21.15.23:0/3684104374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56362c954030 msgr2=0x56362c96d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:28.894 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.893+0000 7f25689b2740 1 --2- 172.21.15.23:0/3684104374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56362c954030 0x56362c96d5e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f25540ee1e0 tx=0x7f255402bc30 comp rx=0 tx=0).stop 2024-03-28T05:24:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.893+0000 7f25689b2740 1 -- 172.21.15.23:0/3684104374 shutdown_connections 2024-03-28T05:24:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.893+0000 7f25689b2740 1 --2- 172.21.15.23:0/3684104374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56362c95ce00 0x56362c96b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.893+0000 7f25689b2740 1 --2- 172.21.15.23:0/3684104374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56362c954030 0x56362c96d5e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.893+0000 7f25689b2740 1 --2- 172.21.15.23:0/3684104374 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x56362c971400 0x56362c9cfb80 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.893+0000 7f25689b2740 1 --2- 172.21.15.23:0/3684104374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56362c953710 0x56362c96d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.893+0000 7f253effd640 0 client.25396 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:24:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.893+0000 7f25689b2740 1 -- 172.21.15.23:0/3684104374 >> 172.21.15.23:0/3684104374 conn(0x56362c94e030 msgr2=0x56362c961180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.893+0000 7f25689b2740 1 -- 172.21.15.23:0/3684104374 shutdown_connections 2024-03-28T05:24:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.893+0000 7f25689b2740 1 -- 172.21.15.23:0/3684104374 wait complete. 2024-03-28T05:24:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.894+0000 7f25689b2740 20 client.25396 trim_cache size 0 max 16384 2024-03-28T05:24:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:24:28.896 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:24:28.896 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:24:28.896 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:24:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.894+0000 7f2565cea640 1 -- 172.21.15.23:0/494469949 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f25480a2b30 msgr2=0x7f25480a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.894+0000 7f2565cea640 1 --2- 172.21.15.23:0/494469949 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f25480a2b30 0x7f25480a4ff0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f255000bb60 tx=0x7f255000b390 comp rx=0 tx=0).stop 2024-03-28T05:24:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.894+0000 7f2565cea640 1 -- 172.21.15.23:0/494469949 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2560102010 msgr2=0x7f2560198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.894+0000 7f2565cea640 1 --2- 172.21.15.23:0/494469949 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2560102010 0x7f2560198450 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2554009230 tx=0x7f255402cc70 comp rx=0 tx=0).stop 2024-03-28T05:24:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.894+0000 7f2565cea640 1 -- 172.21.15.23:0/494469949 shutdown_connections 2024-03-28T05:24:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.894+0000 7f2565cea640 1 --2- 172.21.15.23:0/494469949 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f25480a2b30 0x7f25480a4ff0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.894+0000 7f2565cea640 1 --2- 172.21.15.23:0/494469949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2560107660 0x7f256019c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.894+0000 7f2565cea640 1 --2- 172.21.15.23:0/494469949 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2560102010 0x7f2560198450 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.894+0000 7f2565cea640 1 --2- 172.21.15.23:0/494469949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2560101650 0x7f2560197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.894+0000 7f2565cea640 1 -- 172.21.15.23:0/494469949 >> 172.21.15.23:0/494469949 conn(0x7f25600fd120 msgr2=0x7f25600fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.895+0000 7f2565cea640 1 -- 172.21.15.23:0/494469949 shutdown_connections 2024-03-28T05:24:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:28.895+0000 7f2565cea640 1 -- 172.21.15.23:0/494469949 wait complete. 2024-03-28T05:24:30.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:29 smithi113 ceph-mon[37164]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-28T05:24:30.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:29 smithi023 ceph-mon[31954]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-28T05:24:30.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:29 smithi098 ceph-mon[36087]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-28T05:24:31.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:31 smithi113 ceph-mon[37164]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-28T05:24:32.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:31 smithi023 ceph-mon[31954]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-28T05:24:32.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:31 smithi098 ceph-mon[36087]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-28T05:24:34.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:33 smithi113 ceph-mon[37164]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 426 B/s wr, 2 op/s 2024-03-28T05:24:34.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:33 smithi023 ceph-mon[31954]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 426 B/s wr, 2 op/s 2024-03-28T05:24:34.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:33 smithi098 ceph-mon[36087]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 426 B/s wr, 2 op/s 2024-03-28T05:24:36.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:35 smithi113 ceph-mon[37164]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:36.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:35 smithi023 ceph-mon[31954]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:36.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:35 smithi098 ceph-mon[36087]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:38.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:37 smithi113 ceph-mon[37164]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:38.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:37 smithi023 ceph-mon[31954]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:38.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:37 smithi098 ceph-mon[36087]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:39.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:38 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:24:39.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:38 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:24:39.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:38 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:24:40.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:39 smithi113 ceph-mon[37164]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:40.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:39 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:24:40.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:39 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:24:40.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:39 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:24:40.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:39 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:24:40.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:39 smithi023 ceph-mon[31954]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:40.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:39 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:24:40.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:39 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:24:40.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:39 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:24:40.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:39 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:24:40.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:39 smithi098 ceph-mon[36087]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:40.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:39 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:24:40.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:39 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:24:40.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:39 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:24:40.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:39 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:24:41.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:40 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:24:41.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:40 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:24:41.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:40 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:24:41.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:40 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:24:41.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:40 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:24:41.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:40 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:24:41.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:40 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:24:41.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:40 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:24:41.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:40 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:24:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.031+0000 7f9bcccf8640 1 -- 172.21.15.23:0/554744959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bc81007f0 msgr2=0x7f9bc8100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.031+0000 7f9bcccf8640 1 --2- 172.21.15.23:0/554744959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bc81007f0 0x7f9bc8100c70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc009900 tx=0x7f9bbc02bc30 comp rx=0 tx=0).stop 2024-03-28T05:24:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.032+0000 7f9bcccf8640 1 -- 172.21.15.23:0/554744959 shutdown_connections 2024-03-28T05:24:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.032+0000 7f9bcccf8640 1 --2- 172.21.15.23:0/554744959 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bc81011b0 0x7f9bc810d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.032+0000 7f9bcccf8640 1 --2- 172.21.15.23:0/554744959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bc81007f0 0x7f9bc8100c70 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.032+0000 7f9bcccf8640 1 --2- 172.21.15.23:0/554744959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bc8105540 0x7f9bc8105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.032+0000 7f9bcccf8640 1 -- 172.21.15.23:0/554744959 >> 172.21.15.23:0/554744959 conn(0x7f9bc80773f0 msgr2=0x7f9bc8077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.032+0000 7f9bcccf8640 1 -- 172.21.15.23:0/554744959 shutdown_connections 2024-03-28T05:24:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.032+0000 7f9bcccf8640 1 -- 172.21.15.23:0/554744959 wait complete. 2024-03-28T05:24:42.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.033+0000 7f9bcccf8640 1 Processor -- start 2024-03-28T05:24:42.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.033+0000 7f9bcccf8640 1 -- start start 2024-03-28T05:24:42.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.034+0000 7f9bcccf8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bc81007f0 0x7f9bc8196e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:42.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.034+0000 7f9bcccf8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bc81011b0 0x7f9bc8197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:42.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.034+0000 7f9bcccf8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bc8105540 0x7f9bc819b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:42.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.034+0000 7f9bcccf8640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc8110a20 con 0x7f9bc81011b0 2024-03-28T05:24:42.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.034+0000 7f9bc659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bc81007f0 0x7f9bc8196e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:42.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.034+0000 7f9bc659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bc81007f0 0x7f9bc8196e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.23:53598/0 (socket says 172.21.15.23:53598) 2024-03-28T05:24:42.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.034+0000 7f9bc659c640 1 -- 172.21.15.23:0/3561378957 learned_addr learned my addr 172.21.15.23:0/3561378957 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.034+0000 7f9bc6d9d640 1 --2- 172.21.15.23:0/3561378957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bc8105540 0x7f9bc819b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.034+0000 7f9bc5d9b640 1 --2- 172.21.15.23:0/3561378957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bc81011b0 0x7f9bc8197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.034+0000 7f9bcccf8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc81108a0 con 0x7f9bc81007f0 2024-03-28T05:24:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.034+0000 7f9bcccf8640 1 -- 172.21.15.23:0/3561378957 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc8110ba0 con 0x7f9bc8105540 2024-03-28T05:24:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.034+0000 7f9bc6d9d640 1 -- 172.21.15.23:0/3561378957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bc81007f0 msgr2=0x7f9bc8196e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.034+0000 7f9bc6d9d640 1 --2- 172.21.15.23:0/3561378957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bc81007f0 0x7f9bc8196e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.034+0000 7f9bc6d9d640 1 -- 172.21.15.23:0/3561378957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bc81011b0 msgr2=0x7f9bc8197380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.034+0000 7f9bc6d9d640 1 --2- 172.21.15.23:0/3561378957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bc81011b0 0x7f9bc8197380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.034+0000 7f9bc6d9d640 1 -- 172.21.15.23:0/3561378957 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc819bde0 con 0x7f9bc8105540 2024-03-28T05:24:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.034+0000 7f9bc659c640 1 --2- 172.21.15.23:0/3561378957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bc81007f0 0x7f9bc8196e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:24:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.035+0000 7f9bc6d9d640 1 --2- 172.21.15.23:0/3561378957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bc8105540 0x7f9bc819b6b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0005fd0 tx=0x7f9bc000ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.035+0000 7f9bb77fe640 1 -- 172.21.15.23:0/3561378957 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc000cd80 con 0x7f9bc8105540 2024-03-28T05:24:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.035+0000 7f9bb77fe640 1 -- 172.21.15.23:0/3561378957 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9bc00137d0 con 0x7f9bc8105540 2024-03-28T05:24:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.035+0000 7f9bcccf8640 1 -- 172.21.15.23:0/3561378957 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc819c0d0 con 0x7f9bc8105540 2024-03-28T05:24:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.035+0000 7f9bb77fe640 1 -- 172.21.15.23:0/3561378957 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc00055a0 con 0x7f9bc8105540 2024-03-28T05:24:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.035+0000 7f9bcccf8640 1 -- 172.21.15.23:0/3561378957 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc8068d20 con 0x7f9bc8105540 2024-03-28T05:24:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.038+0000 7f9bb77fe640 1 -- 172.21.15.23:0/3561378957 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f9bc0005860 con 0x7f9bc8105540 2024-03-28T05:24:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.038+0000 7f9bcf9c2740 1 Processor -- start 2024-03-28T05:24:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.039+0000 7f9bcf9c2740 1 -- start start 2024-03-28T05:24:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.039+0000 7f9bcf9c2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d36b710 0x55a63d36baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.039+0000 7f9bb77fe640 1 --2- 172.21.15.23:0/3561378957 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f9bb00a2bb0 0x7f9bb00a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.039+0000 7f9bcf9c2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36c030 0x55a63d3748c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.039+0000 7f9bcf9c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a63d374e00 0x55a63d3771f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.039+0000 7f9bcf9c2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a63d2cfb10 con 0x55a63d36b710 2024-03-28T05:24:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.039+0000 7f9bcf9c2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a63d278900 con 0x55a63d374e00 2024-03-28T05:24:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.039+0000 7f9bcf9c2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a63d2cbbb0 con 0x55a63d36c030 2024-03-28T05:24:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.039+0000 7f9bb77fe640 1 -- 172.21.15.23:0/3561378957 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f9bc00d39e0 con 0x7f9bc8105540 2024-03-28T05:24:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.039+0000 7f9bc659c640 1 --2- 172.21.15.23:0/3561378957 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f9bb00a2bb0 0x7f9bb00a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.039+0000 7f9bc6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a63d374e00 0x55a63d3771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.039+0000 7f9bc5d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d36b710 0x55a63d36baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.039+0000 7f9bc5d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d36b710 0x55a63d36baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47740/0 (socket says 172.21.15.23:47740) 2024-03-28T05:24:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.039+0000 7f9bc5d9b640 1 -- 172.21.15.23:0/1472301777 learned_addr learned my addr 172.21.15.23:0/1472301777 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.040+0000 7f9bc6d9d640 1 -- 172.21.15.23:0/1472301777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36c030 msgr2=0x55a63d3748c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-28T05:24:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.040+0000 7f9bc6d9d640 1 --2- 172.21.15.23:0/1472301777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36c030 0x55a63d3748c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.040+0000 7f9bc6d9d640 1 -- 172.21.15.23:0/1472301777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d36b710 msgr2=0x55a63d36baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.040+0000 7f9bc6d9d640 1 --2- 172.21.15.23:0/1472301777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d36b710 0x55a63d36baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.040+0000 7f9bc6d9d640 1 -- 172.21.15.23:0/1472301777 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a63cfe4210 con 0x55a63d374e00 2024-03-28T05:24:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.040+0000 7f9bc659c640 1 --2- 172.21.15.23:0/3561378957 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f9bb00a2bb0 0x7f9bb00a5070 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8011550 tx=0x7f9bb8005f90 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.040+0000 7f9bc5d9b640 1 --2- 172.21.15.23:0/1472301777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d36b710 0x55a63d36baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:24:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.040+0000 7f9bc6d9d640 1 --2- 172.21.15.23:0/1472301777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a63d374e00 0x55a63d3771f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9bc00d0d30 tx=0x7f9bc009f770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a66cca27c6f0dad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.040+0000 7f9bb4ff9640 1 -- 172.21.15.23:0/1472301777 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0013cf0 con 0x55a63d374e00 2024-03-28T05:24:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.040+0000 7f9bb4ff9640 1 -- 172.21.15.23:0/1472301777 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9bc00a0270 con 0x55a63d374e00 2024-03-28T05:24:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.040+0000 7f9bb4ff9640 1 -- 172.21.15.23:0/1472301777 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc009c1e0 con 0x55a63d374e00 2024-03-28T05:24:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.041+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/1472301777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a63d374e00 msgr2=0x55a63d3771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.041+0000 7f9bcf9c2740 1 --2- 172.21.15.23:0/1472301777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a63d374e00 0x55a63d3771f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9bc00d0d30 tx=0x7f9bc009f770 comp rx=0 tx=0).stop 2024-03-28T05:24:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.041+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/1472301777 shutdown_connections 2024-03-28T05:24:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.041+0000 7f9bcf9c2740 1 --2- 172.21.15.23:0/1472301777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a63d374e00 0x55a63d3771f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.042 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.041+0000 7f9bcf9c2740 1 --2- 172.21.15.23:0/1472301777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36c030 0x55a63d3748c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.042 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.041+0000 7f9bcf9c2740 1 --2- 172.21.15.23:0/1472301777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d36b710 0x55a63d36baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.042 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.041+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/1472301777 >> 172.21.15.23:0/1472301777 conn(0x55a63d366030 msgr2=0x55a63d366410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:42.042 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.042+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/1472301777 shutdown_connections 2024-03-28T05:24:42.042 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.042+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/1472301777 wait complete. 2024-03-28T05:24:42.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.042+0000 7f9bcf9c2740 1 -- start start 2024-03-28T05:24:42.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.042+0000 7f9bcf9c2740 1 Processor -- start 2024-03-28T05:24:42.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.042+0000 7f9bcf9c2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36b710 0x55a63d371f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:42.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.042+0000 7f9bcf9c2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d36c030 0x55a63d372490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:42.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.042+0000 7f9bcf9c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a63d374e00 0x55a63d3729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:42.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.042+0000 7f9bcf9c2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a63d3779d0 con 0x55a63d36c030 2024-03-28T05:24:42.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.042+0000 7f9bcf9c2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a63d28bd50 con 0x55a63d374e00 2024-03-28T05:24:42.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.042+0000 7f9bcf9c2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a63d377b50 con 0x55a63d36b710 2024-03-28T05:24:42.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.043+0000 7f9bcf9c2740 10 client.0 _ll_register_callbacks cb 0x55a63d28b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:24:42.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.043+0000 7f9bc5d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d36c030 0x55a63d372490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:42.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.043+0000 7f9bc5d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d36c030 0x55a63d372490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47752/0 (socket says 172.21.15.23:47752) 2024-03-28T05:24:42.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.043+0000 7f9bc5d9b640 1 -- 172.21.15.23:0/1955447718 learned_addr learned my addr 172.21.15.23:0/1955447718 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.043+0000 7f9bc659c640 1 --2- 172.21.15.23:0/1955447718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36b710 0x55a63d371f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.043+0000 7f9bc6d9d640 1 --2- 172.21.15.23:0/1955447718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a63d374e00 0x55a63d3729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.043+0000 7f9bc5d9b640 1 -- 172.21.15.23:0/1955447718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36b710 msgr2=0x55a63d371f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.043+0000 7f9bc5d9b640 1 --2- 172.21.15.23:0/1955447718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36b710 0x55a63d371f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.043+0000 7f9bc5d9b640 1 -- 172.21.15.23:0/1955447718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a63d374e00 msgr2=0x55a63d3729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.043+0000 7f9bc5d9b640 1 --2- 172.21.15.23:0/1955447718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a63d374e00 0x55a63d3729d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.043+0000 7f9bc5d9b640 1 -- 172.21.15.23:0/1955447718 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a63d0b0620 con 0x55a63d36c030 2024-03-28T05:24:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.043+0000 7f9bc659c640 1 --2- 172.21.15.23:0/1955447718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36b710 0x55a63d371f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:24:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.043+0000 7f9bc6d9d640 1 --2- 172.21.15.23:0/1955447718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a63d374e00 0x55a63d3729d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:24:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.043+0000 7f9bc5d9b640 1 --2- 172.21.15.23:0/1955447718 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d36c030 0x55a63d372490 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc009230 tx=0x7f9bbc02c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76d3e3c6c70c2ec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.043+0000 7f9badffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:24:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.044+0000 7f9badffb640 1 -- 172.21.15.23:0/1955447718 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc049060 con 0x55a63d36c030 2024-03-28T05:24:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.044+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/1955447718 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9bc00a0270 con 0x55a63d36c030 2024-03-28T05:24:42.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.044+0000 7f9badffb640 1 -- 172.21.15.23:0/1955447718 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f9bbc02ccc0 con 0x55a63d36c030 2024-03-28T05:24:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.044+0000 7f9badffb640 1 -- 172.21.15.23:0/1955447718 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9bbc050050 con 0x55a63d36c030 2024-03-28T05:24:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.044+0000 7f9bcf9c2740 10 client.15567 fetch_fsmap learned FSMap version 20 2024-03-28T05:24:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.044+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/1955447718 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55a63d383d60 con 0x55a63d36c030 2024-03-28T05:24:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.044+0000 7f9badffb640 1 -- 172.21.15.23:0/1955447718 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9bc00a0270 con 0x55a63d36c030 2024-03-28T05:24:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.044+0000 7f9badffb640 1 -- 172.21.15.23:0/1955447718 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f9bbc0035f0 con 0x55a63d36c030 2024-03-28T05:24:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.044+0000 7f9bcf9c2740 10 client.15567 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:24:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.044+0000 7f9bcf9c2740 10 client.15567 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:24:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.044+0000 7f9bcf9c2740 20 client.15567 populate_metadata read hostname 'smithi023' 2024-03-28T05:24:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.044+0000 7f9bcf9c2740 1 --2- 172.21.15.23:0/1955447718 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a63d385220 0x55a63d387610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.044+0000 7f9bcf9c2740 4 client.15567 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:24:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.044+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/1955447718 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a63d2dd050 con 0x55a63d385220 2024-03-28T05:24:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.045+0000 7f9bc659c640 1 --2- 172.21.15.23:0/1955447718 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a63d385220 0x55a63d387610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.045+0000 7f9badffb640 10 client.15567 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:24:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.045+0000 7f9bc659c640 1 --2- 172.21.15.23:0/1955447718 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a63d385220 0x55a63d387610 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=606012138506df3d server_cookie=bd3d6b7d21c86f9 in_seq=0 out_seq=0 2024-03-28T05:24:42.049 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.046+0000 7f9badffb640 1 -- 172.21.15.23:0/1955447718 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a63d2dd050 con 0x55a63d385220 2024-03-28T05:24:42.049 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.046+0000 7f9badffb640 10 client.15567 handle_command_reply: tid=0 2024-03-28T05:24:42.049 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.046+0000 7f9bcf9c2740 1 client.15567 shutdown 2024-03-28T05:24:42.049 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.046+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/1955447718 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d36c030 msgr2=0x55a63d372490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:42.049 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.046+0000 7f9bcf9c2740 1 --2- 172.21.15.23:0/1955447718 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d36c030 0x55a63d372490 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc009230 tx=0x7f9bbc02c800 comp rx=0 tx=0).stop 2024-03-28T05:24:42.049 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.046+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/1955447718 shutdown_connections 2024-03-28T05:24:42.049 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.046+0000 7f9bcf9c2740 1 --2- 172.21.15.23:0/1955447718 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a63d385220 0x55a63d387610 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.049 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.046+0000 7f9bcf9c2740 1 --2- 172.21.15.23:0/1955447718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a63d374e00 0x55a63d3729d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.049 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.046+0000 7f9badffb640 0 client.15567 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:24:42.049 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.046+0000 7f9bcf9c2740 1 --2- 172.21.15.23:0/1955447718 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d36c030 0x55a63d372490 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.049 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.046+0000 7f9bcf9c2740 1 --2- 172.21.15.23:0/1955447718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36b710 0x55a63d371f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.049 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.046+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/1955447718 >> 172.21.15.23:0/1955447718 conn(0x55a63d366030 msgr2=0x55a63d367d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:42.049 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.046+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/1955447718 shutdown_connections 2024-03-28T05:24:42.049 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.046+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/1955447718 wait complete. 2024-03-28T05:24:42.049 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.046+0000 7f9bcf9c2740 20 client.15567 trim_cache size 0 max 16384 2024-03-28T05:24:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.060+0000 7f9bcf9c2740 1 Processor -- start 2024-03-28T05:24:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.060+0000 7f9bcf9c2740 1 -- start start 2024-03-28T05:24:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.061+0000 7f9bcf9c2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36b710 0x55a63d374680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.061+0000 7f9bcf9c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a63d36c030 0x55a63d367b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.061+0000 7f9bcf9c2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d374e00 0x55a63d3727f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.061+0000 7f9bcf9c2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc81aa740 con 0x55a63d374e00 2024-03-28T05:24:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.061+0000 7f9bcf9c2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc81acd00 con 0x55a63d36c030 2024-03-28T05:24:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.061+0000 7f9bcf9c2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc81aad40 con 0x55a63d36b710 2024-03-28T05:24:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.061+0000 7f9bc659c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36b710 0x55a63d374680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.061+0000 7f9bc659c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36b710 0x55a63d374680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:38346/0 (socket says 172.21.15.23:38346) 2024-03-28T05:24:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.061+0000 7f9bc659c640 1 -- 172.21.15.23:0/1196698272 learned_addr learned my addr 172.21.15.23:0/1196698272 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.061+0000 7f9bc6d9d640 1 --2- 172.21.15.23:0/1196698272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d374e00 0x55a63d3727f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:42.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.061+0000 7f9bc5d9b640 1 --2- 172.21.15.23:0/1196698272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a63d36c030 0x55a63d367b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.061+0000 7f9bc659c640 1 -- 172.21.15.23:0/1196698272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a63d36c030 msgr2=0x55a63d367b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.061+0000 7f9bc659c640 1 --2- 172.21.15.23:0/1196698272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a63d36c030 0x55a63d367b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.061+0000 7f9bc659c640 1 -- 172.21.15.23:0/1196698272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d374e00 msgr2=0x55a63d3727f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.061+0000 7f9bc659c640 1 --2- 172.21.15.23:0/1196698272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d374e00 0x55a63d3727f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.061+0000 7f9bc659c640 1 -- 172.21.15.23:0/1196698272 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a63d377560 con 0x55a63d36b710 2024-03-28T05:24:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.062+0000 7f9bc659c640 1 --2- 172.21.15.23:0/1196698272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36b710 0x55a63d374680 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9bb800f720 tx=0x7f9bb8018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84130a5bebd023f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.062+0000 7f9baffff640 1 -- 172.21.15.23:0/1196698272 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8028d90 con 0x55a63d36b710 2024-03-28T05:24:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.062+0000 7f9baffff640 1 -- 172.21.15.23:0/1196698272 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9bb802c030 con 0x55a63d36b710 2024-03-28T05:24:42.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.062+0000 7f9baffff640 1 -- 172.21.15.23:0/1196698272 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb800e5f0 con 0x55a63d36b710 2024-03-28T05:24:42.065 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.063+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/1196698272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36b710 msgr2=0x55a63d374680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:42.065 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.063+0000 7f9bcf9c2740 1 --2- 172.21.15.23:0/1196698272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36b710 0x55a63d374680 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9bb800f720 tx=0x7f9bb8018000 comp rx=0 tx=0).stop 2024-03-28T05:24:42.065 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.063+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/1196698272 shutdown_connections 2024-03-28T05:24:42.065 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.063+0000 7f9bcf9c2740 1 --2- 172.21.15.23:0/1196698272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d374e00 0x55a63d3727f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.065 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.063+0000 7f9bcf9c2740 1 --2- 172.21.15.23:0/1196698272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a63d36c030 0x55a63d367b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.065 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.063+0000 7f9bcf9c2740 1 --2- 172.21.15.23:0/1196698272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36b710 0x55a63d374680 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.065 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.063+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/1196698272 >> 172.21.15.23:0/1196698272 conn(0x55a63d366030 msgr2=0x55a63d367430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:42.065 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.063+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/1196698272 shutdown_connections 2024-03-28T05:24:42.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.063+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/1196698272 wait complete. 2024-03-28T05:24:42.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.063+0000 7f9bcf9c2740 1 -- start start 2024-03-28T05:24:42.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.063+0000 7f9bcf9c2740 1 Processor -- start 2024-03-28T05:24:42.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.064+0000 7f9bcf9c2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d36b710 0x55a63d378570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:42.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.064+0000 7f9bcf9c2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36c030 0x55a63d378ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:42.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.064+0000 7f9bcf9c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a63d374e00 0x55a63d3df9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:42.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.064+0000 7f9bcf9c2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc81ab380 con 0x55a63d36b710 2024-03-28T05:24:42.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.064+0000 7f9bc659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d36b710 0x55a63d378570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:42.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.064+0000 7f9bcf9c2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc81ac6a0 con 0x55a63d374e00 2024-03-28T05:24:42.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.064+0000 7f9bcf9c2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc81ab9e0 con 0x55a63d36c030 2024-03-28T05:24:42.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.064+0000 7f9bc5d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36c030 0x55a63d378ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:42.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.064+0000 7f9bc659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d36b710 0x55a63d378570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47774/0 (socket says 172.21.15.23:47774) 2024-03-28T05:24:42.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.064+0000 7f9bc659c640 1 -- 172.21.15.23:0/2304552681 learned_addr learned my addr 172.21.15.23:0/2304552681 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:42.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.064+0000 7f9bcf9c2740 10 client.0 _ll_register_callbacks cb 0x7f9bbc0044b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:24:42.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.064+0000 7f9bc6d9d640 1 --2- 172.21.15.23:0/2304552681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a63d374e00 0x55a63d3df9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.064+0000 7f9bc5d9b640 1 -- 172.21.15.23:0/2304552681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a63d374e00 msgr2=0x55a63d3df9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.064+0000 7f9bc5d9b640 1 --2- 172.21.15.23:0/2304552681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a63d374e00 0x55a63d3df9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.064+0000 7f9bc5d9b640 1 -- 172.21.15.23:0/2304552681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d36b710 msgr2=0x55a63d378570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.064+0000 7f9bc5d9b640 1 --2- 172.21.15.23:0/2304552681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d36b710 0x55a63d378570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.064+0000 7f9bc5d9b640 1 -- 172.21.15.23:0/2304552681 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a63d3dff20 con 0x55a63d36c030 2024-03-28T05:24:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.064+0000 7f9bc659c640 1 --2- 172.21.15.23:0/2304552681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d36b710 0x55a63d378570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:24:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.064+0000 7f9bc5d9b640 1 --2- 172.21.15.23:0/2304552681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36c030 0x55a63d378ab0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc009900 tx=0x7f9bbc003d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6063a6c15b24a1e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.064+0000 7f9badffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-28T05:24:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.065+0000 7f9badffb640 1 -- 172.21.15.23:0/2304552681 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc058070 con 0x55a63d36c030 2024-03-28T05:24:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.065+0000 7f9badffb640 1 -- 172.21.15.23:0/2304552681 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f9bbc02ccc0 con 0x55a63d36c030 2024-03-28T05:24:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.065+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/2304552681 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9bb802c030 con 0x55a63d36c030 2024-03-28T05:24:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.065+0000 7f9badffb640 1 -- 172.21.15.23:0/2304552681 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9bbc049040 con 0x55a63d36c030 2024-03-28T05:24:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.065+0000 7f9bcf9c2740 10 client.25408 fetch_fsmap learned FSMap version 20 2024-03-28T05:24:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.065+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/2304552681 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55a63d3e1040 con 0x55a63d36c030 2024-03-28T05:24:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.065+0000 7f9badffb640 1 -- 172.21.15.23:0/2304552681 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9bb802c030 con 0x55a63d36c030 2024-03-28T05:24:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.066+0000 7f9badffb640 1 -- 172.21.15.23:0/2304552681 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f9bbc05b020 con 0x55a63d36c030 2024-03-28T05:24:42.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.066+0000 7f9bcf9c2740 10 client.25408 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:24:42.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.066+0000 7f9bcf9c2740 10 client.25408 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:24:42.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.066+0000 7f9bcf9c2740 20 client.25408 populate_metadata read hostname 'smithi023' 2024-03-28T05:24:42.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.066+0000 7f9bcf9c2740 1 --2- 172.21.15.23:0/2304552681 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a63d3e1700 0x55a63d3e3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:42.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.066+0000 7f9bcf9c2740 4 client.25408 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:24:42.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.066+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/2304552681 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a63d2d73a0 con 0x55a63d3e1700 2024-03-28T05:24:42.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.066+0000 7f9bc659c640 1 --2- 172.21.15.23:0/2304552681 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a63d3e1700 0x55a63d3e3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:42.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.067+0000 7f9badffb640 10 client.25408 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:24:42.068 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.067+0000 7f9bc659c640 1 --2- 172.21.15.23:0/2304552681 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a63d3e1700 0x55a63d3e3b20 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4da3d19545d311b server_cookie=d1a8523dbcc0e778 in_seq=0 out_seq=0 2024-03-28T05:24:42.069 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.070+0000 7f9badffb640 1 -- 172.21.15.23:0/2304552681 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a63d2d73a0 con 0x55a63d3e1700 2024-03-28T05:24:42.069 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.070+0000 7f9badffb640 10 client.25408 handle_command_reply: tid=0 2024-03-28T05:24:42.069 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.070+0000 7f9bcf9c2740 1 client.25408 shutdown 2024-03-28T05:24:42.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.070+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/2304552681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36c030 msgr2=0x55a63d378ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:42.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.070+0000 7f9bcf9c2740 1 --2- 172.21.15.23:0/2304552681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36c030 0x55a63d378ab0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc009900 tx=0x7f9bbc003d90 comp rx=0 tx=0).stop 2024-03-28T05:24:42.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.070+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/2304552681 shutdown_connections 2024-03-28T05:24:42.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.070+0000 7f9bcf9c2740 1 --2- 172.21.15.23:0/2304552681 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a63d3e1700 0x55a63d3e3b20 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.070+0000 7f9badffb640 0 client.25408 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:24:42.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.070+0000 7f9bcf9c2740 1 --2- 172.21.15.23:0/2304552681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a63d374e00 0x55a63d3df9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.070+0000 7f9bcf9c2740 1 --2- 172.21.15.23:0/2304552681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a63d36c030 0x55a63d378ab0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.070+0000 7f9bcf9c2740 1 --2- 172.21.15.23:0/2304552681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a63d36b710 0x55a63d378570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.071+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/2304552681 >> 172.21.15.23:0/2304552681 conn(0x55a63d366030 msgr2=0x55a63d365020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:42.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.071+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/2304552681 shutdown_connections 2024-03-28T05:24:42.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.071+0000 7f9bcf9c2740 1 -- 172.21.15.23:0/2304552681 wait complete. 2024-03-28T05:24:42.070 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.071+0000 7f9bcf9c2740 20 client.25408 trim_cache size 0 max 16384 2024-03-28T05:24:42.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:24:42.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:24:42.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:24:42.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:24:42.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.071+0000 7f9bcccf8640 1 -- 172.21.15.23:0/3561378957 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f9bb00a2bb0 msgr2=0x7f9bb00a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:42.071 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.071+0000 7f9bcccf8640 1 --2- 172.21.15.23:0/3561378957 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f9bb00a2bb0 0x7f9bb00a5070 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8011550 tx=0x7f9bb8005f90 comp rx=0 tx=0).stop 2024-03-28T05:24:42.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.071+0000 7f9bcccf8640 1 -- 172.21.15.23:0/3561378957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bc8105540 msgr2=0x7f9bc819b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:42.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.071+0000 7f9bcccf8640 1 --2- 172.21.15.23:0/3561378957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bc8105540 0x7f9bc819b6b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0005fd0 tx=0x7f9bc000ee60 comp rx=0 tx=0).stop 2024-03-28T05:24:42.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.071+0000 7f9bcccf8640 1 -- 172.21.15.23:0/3561378957 shutdown_connections 2024-03-28T05:24:42.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.071+0000 7f9bcccf8640 1 --2- 172.21.15.23:0/3561378957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9bc8105540 0x7f9bc819b6b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.071+0000 7f9bcccf8640 1 --2- 172.21.15.23:0/3561378957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bc81011b0 0x7f9bc8197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.071+0000 7f9bcccf8640 1 --2- 172.21.15.23:0/3561378957 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f9bb00a2bb0 0x7f9bb00a5070 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.071+0000 7f9bcccf8640 1 --2- 172.21.15.23:0/3561378957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bc81007f0 0x7f9bc8196e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:42.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.071+0000 7f9bcccf8640 1 -- 172.21.15.23:0/3561378957 >> 172.21.15.23:0/3561378957 conn(0x7f9bc80773f0 msgr2=0x7f9bc80fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:42.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.071+0000 7f9bcccf8640 1 -- 172.21.15.23:0/3561378957 shutdown_connections 2024-03-28T05:24:42.072 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:42.072+0000 7f9bcccf8640 1 -- 172.21.15.23:0/3561378957 wait complete. 2024-03-28T05:24:42.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:41 smithi113 ceph-mon[37164]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:42.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:41 smithi023 ceph-mon[31954]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:42.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:41 smithi098 ceph-mon[36087]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:43.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:42 smithi113 ceph-mon[37164]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:24:43.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:42 smithi023 ceph-mon[31954]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:24:43.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:42 smithi098 ceph-mon[36087]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:24:45.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:45 smithi098 ceph-mon[36087]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:24:45.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:45 smithi113 ceph-mon[37164]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:24:45.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:45 smithi023 ceph-mon[31954]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:24:47.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:47 smithi098 ceph-mon[36087]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:47.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:47 smithi113 ceph-mon[37164]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:47.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:47 smithi023 ceph-mon[31954]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:49.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:49 smithi098 ceph-mon[36087]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:49.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:49 smithi113 ceph-mon[37164]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:49.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:49 smithi023 ceph-mon[31954]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:51.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:51 smithi098 ceph-mon[36087]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:51.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:51 smithi113 ceph-mon[37164]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:51.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:51 smithi023 ceph-mon[31954]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:53.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:53 smithi098 ceph-mon[36087]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:53.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:53 smithi113 ceph-mon[37164]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:53.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:53 smithi023 ceph-mon[31954]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:54.489 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 95% (11718/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-28T05:24:55.065 INFO:tasks.workunit.client.0.smithi023.stderr:Note: switching to '3be9a5c9bc793e11e2800a8c0c696e8b46742033'. 2024-03-28T05:24:55.065 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-28T05:24:55.065 INFO:tasks.workunit.client.0.smithi023.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-28T05:24:55.065 INFO:tasks.workunit.client.0.smithi023.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-28T05:24:55.065 INFO:tasks.workunit.client.0.smithi023.stderr:state without impacting any branches by switching back to a branch. 2024-03-28T05:24:55.065 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-28T05:24:55.065 INFO:tasks.workunit.client.0.smithi023.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-28T05:24:55.065 INFO:tasks.workunit.client.0.smithi023.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-28T05:24:55.065 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-28T05:24:55.065 INFO:tasks.workunit.client.0.smithi023.stderr: git switch -c 2024-03-28T05:24:55.065 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-28T05:24:55.065 INFO:tasks.workunit.client.0.smithi023.stderr:Or undo this operation with: 2024-03-28T05:24:55.066 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-28T05:24:55.066 INFO:tasks.workunit.client.0.smithi023.stderr: git switch - 2024-03-28T05:24:55.066 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-28T05:24:55.066 INFO:tasks.workunit.client.0.smithi023.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-28T05:24:55.066 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-28T05:24:55.066 INFO:tasks.workunit.client.0.smithi023.stderr:HEAD is now at 3be9a5c9bc7 dummy commit #1 2024-03-28T05:24:55.070 DEBUG:teuthology.orchestra.run.smithi023:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-28T05:24:55.130 INFO:tasks.workunit.client.0.smithi023.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-28T05:24:55.133 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T05:24:55.133 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-28T05:24:55.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.159+0000 7f410f3ab640 1 -- 172.21.15.23:0/68863489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081011b0 msgr2=0x7f410810d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:55.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.159+0000 7f410f3ab640 1 --2- 172.21.15.23:0/68863489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081011b0 0x7f410810d3d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f4100009ca0 tx=0x7f410002c0f0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.159+0000 7f410f3ab640 1 -- 172.21.15.23:0/68863489 shutdown_connections 2024-03-28T05:24:55.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.159+0000 7f410f3ab640 1 --2- 172.21.15.23:0/68863489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081011b0 0x7f410810d3d0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.159+0000 7f410f3ab640 1 --2- 172.21.15.23:0/68863489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41081007f0 0x7f4108100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.159+0000 7f410f3ab640 1 --2- 172.21.15.23:0/68863489 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4108105540 0x7f4108105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.159+0000 7f410f3ab640 1 -- 172.21.15.23:0/68863489 >> 172.21.15.23:0/68863489 conn(0x7f41080773f0 msgr2=0x7f4108077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:55.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.159+0000 7f410f3ab640 1 -- 172.21.15.23:0/68863489 shutdown_connections 2024-03-28T05:24:55.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.159+0000 7f410f3ab640 1 -- 172.21.15.23:0/68863489 wait complete. 2024-03-28T05:24:55.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.160+0000 7f410f3ab640 1 Processor -- start 2024-03-28T05:24:55.160 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.160+0000 7f410f3ab640 1 -- start start 2024-03-28T05:24:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.160+0000 7f410f3ab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41081007f0 0x7f4108196dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.160+0000 7f410f3ab640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081011b0 0x7f4108197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.160+0000 7f410f3ab640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4108105540 0x7f410819b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.160+0000 7f410f3ab640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4108110a60 con 0x7f41081011b0 2024-03-28T05:24:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.160+0000 7f410f3ab640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41081108e0 con 0x7f41081007f0 2024-03-28T05:24:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.160+0000 7f410f3ab640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4108110be0 con 0x7f4108105540 2024-03-28T05:24:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.160+0000 7f410c946640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081011b0 0x7f4108197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.160+0000 7f410c946640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081011b0 0x7f4108197300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50450/0 (socket says 172.21.15.23:50450) 2024-03-28T05:24:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.160+0000 7f410c946640 1 -- 172.21.15.23:0/2002858562 learned_addr learned my addr 172.21.15.23:0/2002858562 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.160+0000 7f410d147640 1 --2- 172.21.15.23:0/2002858562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41081007f0 0x7f4108196dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.160+0000 7f410d948640 1 --2- 172.21.15.23:0/2002858562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4108105540 0x7f410819b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.161+0000 7f410c946640 1 -- 172.21.15.23:0/2002858562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4108105540 msgr2=0x7f410819b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.161+0000 7f410c946640 1 --2- 172.21.15.23:0/2002858562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4108105540 0x7f410819b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.161+0000 7f410c946640 1 -- 172.21.15.23:0/2002858562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41081007f0 msgr2=0x7f4108196dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.161+0000 7f410c946640 1 --2- 172.21.15.23:0/2002858562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41081007f0 0x7f4108196dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.161+0000 7f410c946640 1 -- 172.21.15.23:0/2002858562 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410819bdb0 con 0x7f41081011b0 2024-03-28T05:24:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.161+0000 7f410c946640 1 --2- 172.21.15.23:0/2002858562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081011b0 0x7f4108197300 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f410400d950 tx=0x7f410400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.161+0000 7f40fe7fc640 1 -- 172.21.15.23:0/2002858562 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4104017070 con 0x7f41081011b0 2024-03-28T05:24:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.161+0000 7f410f3ab640 1 -- 172.21.15.23:0/2002858562 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f410819c0a0 con 0x7f41081011b0 2024-03-28T05:24:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.161+0000 7f40fe7fc640 1 -- 172.21.15.23:0/2002858562 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4104009c30 con 0x7f41081011b0 2024-03-28T05:24:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.161+0000 7f40fe7fc640 1 -- 172.21.15.23:0/2002858562 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410400f6b0 con 0x7f41081011b0 2024-03-28T05:24:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.161+0000 7f410f3ab640 1 -- 172.21.15.23:0/2002858562 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41081a3930 con 0x7f41081011b0 2024-03-28T05:24:55.168 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.163+0000 7f40fe7fc640 1 -- 172.21.15.23:0/2002858562 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f41040109d0 con 0x7f41081011b0 2024-03-28T05:24:55.168 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.163+0000 7f4111f8a740 1 Processor -- start 2024-03-28T05:24:55.168 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f4111f8a740 1 -- start start 2024-03-28T05:24:55.168 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f4111f8a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77b920 0x56284f77bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f4111f8a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f77c240 0x56284f784ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f4111f8a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56284f785010 0x56284f787400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f4111f8a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56284f6ddb10 con 0x56284f77b920 2024-03-28T05:24:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f4111f8a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x56284f686900 con 0x56284f785010 2024-03-28T05:24:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f4111f8a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x56284f6d9bb0 con 0x56284f77c240 2024-03-28T05:24:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f410d948640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77b920 0x56284f77bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f40fe7fc640 1 --2- 172.21.15.23:0/2002858562 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f40f40a2b30 0x7f40f40a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f410d948640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77b920 0x56284f77bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50452/0 (socket says 172.21.15.23:50452) 2024-03-28T05:24:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f410d948640 1 -- 172.21.15.23:0/817045274 learned_addr learned my addr 172.21.15.23:0/817045274 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f410d147640 1 --2- 172.21.15.23:0/817045274 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f77c240 0x56284f784ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f410d948640 1 --2- 172.21.15.23:0/2002858562 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f40f40a2b30 0x7f40f40a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f40fe7fc640 1 -- 172.21.15.23:0/2002858562 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f41040cf030 con 0x7f41081011b0 2024-03-28T05:24:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f410c946640 1 --2- 172.21.15.23:0/817045274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56284f785010 0x56284f787400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f410d948640 1 -- 172.21.15.23:0/817045274 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f77c240 msgr2=0x56284f784ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f410d948640 1 --2- 172.21.15.23:0/817045274 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f77c240 0x56284f784ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f410d948640 1 -- 172.21.15.23:0/817045274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56284f785010 msgr2=0x56284f787400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f410d948640 1 --2- 172.21.15.23:0/817045274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56284f785010 0x56284f787400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f410d948640 1 -- 172.21.15.23:0/817045274 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56284f3f2210 con 0x56284f77b920 2024-03-28T05:24:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f410c946640 1 --2- 172.21.15.23:0/817045274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56284f785010 0x56284f787400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:24:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f410d147640 1 --2- 172.21.15.23:0/817045274 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f77c240 0x56284f784ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:24:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.164+0000 7f410d948640 1 --2- 172.21.15.23:0/817045274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77b920 0x56284f77bd00 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f4100004af0 tx=0x7f4100032f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c16066093a177072 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.165+0000 7f40f37fe640 1 -- 172.21.15.23:0/817045274 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4100050070 con 0x56284f77b920 2024-03-28T05:24:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.165+0000 7f40f37fe640 1 -- 172.21.15.23:0/817045274 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4100032870 con 0x56284f77b920 2024-03-28T05:24:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.165+0000 7f40f37fe640 1 -- 172.21.15.23:0/817045274 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4100039040 con 0x56284f77b920 2024-03-28T05:24:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.165+0000 7f4111f8a740 1 -- 172.21.15.23:0/817045274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77b920 msgr2=0x56284f77bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.165+0000 7f410d948640 1 --2- 172.21.15.23:0/2002858562 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f40f40a2b30 0x7f40f40a4ff0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4100006010 tx=0x7f4100049000 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.165+0000 7f4111f8a740 1 --2- 172.21.15.23:0/817045274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77b920 0x56284f77bd00 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f4100004af0 tx=0x7f4100032f10 comp rx=0 tx=0).stop 2024-03-28T05:24:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.165+0000 7f4111f8a740 1 -- 172.21.15.23:0/817045274 shutdown_connections 2024-03-28T05:24:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.165+0000 7f4111f8a740 1 --2- 172.21.15.23:0/817045274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56284f785010 0x56284f787400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.165+0000 7f4111f8a740 1 --2- 172.21.15.23:0/817045274 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f77c240 0x56284f784ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.165+0000 7f4111f8a740 1 --2- 172.21.15.23:0/817045274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77b920 0x56284f77bd00 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.165+0000 7f4111f8a740 1 -- 172.21.15.23:0/817045274 >> 172.21.15.23:0/817045274 conn(0x56284f776240 msgr2=0x56284f776620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.165+0000 7f4111f8a740 1 -- 172.21.15.23:0/817045274 shutdown_connections 2024-03-28T05:24:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.166+0000 7f4111f8a740 1 -- 172.21.15.23:0/817045274 wait complete. 2024-03-28T05:24:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.166+0000 7f4111f8a740 1 -- start start 2024-03-28T05:24:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.166+0000 7f4111f8a740 1 Processor -- start 2024-03-28T05:24:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.166+0000 7f4111f8a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f77b920 0x56284f782090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.166+0000 7f4111f8a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77c240 0x56284f7825d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.166+0000 7f4111f8a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56284f785010 0x56284f782b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.166+0000 7f4111f8a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56284f787c40 con 0x56284f77c240 2024-03-28T05:24:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.166+0000 7f4111f8a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x56284f699d50 con 0x56284f785010 2024-03-28T05:24:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.166+0000 7f4111f8a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x56284f787dc0 con 0x56284f77b920 2024-03-28T05:24:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.166+0000 7f4111f8a740 10 client.0 _ll_register_callbacks cb 0x56284f699720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:24:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.166+0000 7f410c946640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77c240 0x56284f7825d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.166+0000 7f410c946640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77c240 0x56284f7825d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50460/0 (socket says 172.21.15.23:50460) 2024-03-28T05:24:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.166+0000 7f410d147640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f77b920 0x56284f782090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.166+0000 7f410c946640 1 -- 172.21.15.23:0/2570625246 learned_addr learned my addr 172.21.15.23:0/2570625246 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.167+0000 7f410d948640 1 --2- 172.21.15.23:0/2570625246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56284f785010 0x56284f782b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.167+0000 7f410c946640 1 -- 172.21.15.23:0/2570625246 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f77b920 msgr2=0x56284f782090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.167+0000 7f410c946640 1 --2- 172.21.15.23:0/2570625246 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f77b920 0x56284f782090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.167+0000 7f410c946640 1 -- 172.21.15.23:0/2570625246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56284f785010 msgr2=0x56284f782b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.167+0000 7f410c946640 1 --2- 172.21.15.23:0/2570625246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56284f785010 0x56284f782b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.167+0000 7f410c946640 1 -- 172.21.15.23:0/2570625246 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56284f4be620 con 0x56284f77c240 2024-03-28T05:24:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.167+0000 7f410d948640 1 --2- 172.21.15.23:0/2570625246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56284f785010 0x56284f782b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:24:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.167+0000 7f410c946640 1 --2- 172.21.15.23:0/2570625246 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77c240 0x56284f7825d0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f410409b0f0 tx=0x7f4104000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae6088d5cdf2e3c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.167+0000 7f40f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:24:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.167+0000 7f40f0ff9640 1 -- 172.21.15.23:0/2570625246 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41040b5020 con 0x56284f77c240 2024-03-28T05:24:55.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.167+0000 7f40f0ff9640 1 -- 172.21.15.23:0/2570625246 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f41040b9db0 con 0x56284f77c240 2024-03-28T05:24:55.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.167+0000 7f4111f8a740 1 -- 172.21.15.23:0/2570625246 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4100032870 con 0x56284f77c240 2024-03-28T05:24:55.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.167+0000 7f40f0ff9640 1 -- 172.21.15.23:0/2570625246 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4104020040 con 0x56284f77c240 2024-03-28T05:24:55.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.167+0000 7f4111f8a740 10 client.15585 fetch_fsmap learned FSMap version 20 2024-03-28T05:24:55.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.167+0000 7f4111f8a740 1 -- 172.21.15.23:0/2570625246 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x56284f793eb0 con 0x56284f77c240 2024-03-28T05:24:55.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.168+0000 7f40f0ff9640 1 -- 172.21.15.23:0/2570625246 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4100032870 con 0x56284f77c240 2024-03-28T05:24:55.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.168+0000 7f40f0ff9640 1 -- 172.21.15.23:0/2570625246 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f41040cc550 con 0x56284f77c240 2024-03-28T05:24:55.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.168+0000 7f4111f8a740 10 client.15585 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:24:55.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.168+0000 7f4111f8a740 10 client.15585 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:24:55.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.168+0000 7f4111f8a740 20 client.15585 populate_metadata read hostname 'smithi023' 2024-03-28T05:24:55.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.168+0000 7f4111f8a740 1 --2- 172.21.15.23:0/2570625246 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x56284f795370 0x56284f797760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:55.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.168+0000 7f4111f8a740 4 client.15585 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:24:55.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.168+0000 7f4111f8a740 1 -- 172.21.15.23:0/2570625246 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56284f6eb050 con 0x56284f795370 2024-03-28T05:24:55.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.168+0000 7f410d147640 1 --2- 172.21.15.23:0/2570625246 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x56284f795370 0x56284f797760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:55.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.168+0000 7f410d147640 1 --2- 172.21.15.23:0/2570625246 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x56284f795370 0x56284f797760 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2fe706bf1168803 server_cookie=5b42d63e94f58f6c in_seq=0 out_seq=0 2024-03-28T05:24:55.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.168+0000 7f40f0ff9640 10 client.15585 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:24:55.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.169+0000 7f40f0ff9640 1 -- 172.21.15.23:0/2570625246 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56284f6eb050 con 0x56284f795370 2024-03-28T05:24:55.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.169+0000 7f40f0ff9640 10 client.15585 handle_command_reply: tid=0 2024-03-28T05:24:55.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.169+0000 7f4111f8a740 1 client.15585 shutdown 2024-03-28T05:24:55.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.169+0000 7f4111f8a740 1 -- 172.21.15.23:0/2570625246 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77c240 msgr2=0x56284f7825d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:55.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.169+0000 7f4111f8a740 1 --2- 172.21.15.23:0/2570625246 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77c240 0x56284f7825d0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f410409b0f0 tx=0x7f4104000bc0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.169+0000 7f4111f8a740 1 -- 172.21.15.23:0/2570625246 shutdown_connections 2024-03-28T05:24:55.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.169+0000 7f4111f8a740 1 --2- 172.21.15.23:0/2570625246 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x56284f795370 0x56284f797760 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.169+0000 7f4111f8a740 1 --2- 172.21.15.23:0/2570625246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56284f785010 0x56284f782b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.169+0000 7f4111f8a740 1 --2- 172.21.15.23:0/2570625246 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77c240 0x56284f7825d0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.169+0000 7f40f0ff9640 0 client.15585 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:24:55.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.169+0000 7f4111f8a740 1 --2- 172.21.15.23:0/2570625246 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f77b920 0x56284f782090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.169+0000 7f4111f8a740 1 -- 172.21.15.23:0/2570625246 >> 172.21.15.23:0/2570625246 conn(0x56284f776240 msgr2=0x56284f777f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:55.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.169+0000 7f4111f8a740 1 -- 172.21.15.23:0/2570625246 shutdown_connections 2024-03-28T05:24:55.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.169+0000 7f4111f8a740 1 -- 172.21.15.23:0/2570625246 wait complete. 2024-03-28T05:24:55.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.169+0000 7f4111f8a740 20 client.15585 trim_cache size 0 max 16384 2024-03-28T05:24:55.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.183+0000 7f4111f8a740 1 Processor -- start 2024-03-28T05:24:55.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.183+0000 7f4111f8a740 1 -- start start 2024-03-28T05:24:55.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f4111f8a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f77b920 0x56284f7847c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:55.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f4111f8a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77c240 0x56284f7904d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:55.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f4111f8a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56284f785010 0x56284f790a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:55.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f4111f8a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41081aa710 con 0x56284f77c240 2024-03-28T05:24:55.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f4111f8a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41081accd0 con 0x56284f785010 2024-03-28T05:24:55.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f4111f8a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41081aad10 con 0x56284f77b920 2024-03-28T05:24:55.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f410c946640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77c240 0x56284f7904d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:55.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f410d147640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f77b920 0x56284f7847c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:55.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f410c946640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77c240 0x56284f7904d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50472/0 (socket says 172.21.15.23:50472) 2024-03-28T05:24:55.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f410d147640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f77b920 0x56284f7847c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:40414/0 (socket says 172.21.15.23:40414) 2024-03-28T05:24:55.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f410c946640 1 -- 172.21.15.23:0/2103890984 learned_addr learned my addr 172.21.15.23:0/2103890984 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:55.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f410d948640 1 --2- 172.21.15.23:0/2103890984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56284f785010 0x56284f790a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:55.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f410c946640 1 -- 172.21.15.23:0/2103890984 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f77b920 msgr2=0x56284f7847c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:55.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f410c946640 1 --2- 172.21.15.23:0/2103890984 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f77b920 0x56284f7847c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f410c946640 1 -- 172.21.15.23:0/2103890984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56284f785010 msgr2=0x56284f790a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:55.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f410c946640 1 --2- 172.21.15.23:0/2103890984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56284f785010 0x56284f790a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f410c946640 1 -- 172.21.15.23:0/2103890984 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56284f787770 con 0x56284f77c240 2024-03-28T05:24:55.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f410d948640 1 --2- 172.21.15.23:0/2103890984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56284f785010 0x56284f790a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:24:55.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f410d147640 1 --2- 172.21.15.23:0/2103890984 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f77b920 0x56284f7847c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:24:55.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f410c946640 1 --2- 172.21.15.23:0/2103890984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77c240 0x56284f7904d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f410409ae60 tx=0x7f410400bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4b68233016471e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:55.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f40f2ffd640 1 -- 172.21.15.23:0/2103890984 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41040b5070 con 0x56284f77c240 2024-03-28T05:24:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f40f2ffd640 1 -- 172.21.15.23:0/2103890984 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f410409f200 con 0x56284f77c240 2024-03-28T05:24:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.184+0000 7f40f2ffd640 1 -- 172.21.15.23:0/2103890984 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41040ccb90 con 0x56284f77c240 2024-03-28T05:24:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.185+0000 7f4111f8a740 1 -- 172.21.15.23:0/2103890984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77c240 msgr2=0x56284f7904d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.185+0000 7f4111f8a740 1 --2- 172.21.15.23:0/2103890984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77c240 0x56284f7904d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f410409ae60 tx=0x7f410400bf60 comp rx=0 tx=0).stop 2024-03-28T05:24:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.185+0000 7f4111f8a740 1 -- 172.21.15.23:0/2103890984 shutdown_connections 2024-03-28T05:24:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.185+0000 7f4111f8a740 1 --2- 172.21.15.23:0/2103890984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56284f785010 0x56284f790a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.185+0000 7f4111f8a740 1 --2- 172.21.15.23:0/2103890984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77c240 0x56284f7904d0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.185+0000 7f4111f8a740 1 --2- 172.21.15.23:0/2103890984 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f77b920 0x56284f7847c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.185+0000 7f4111f8a740 1 -- 172.21.15.23:0/2103890984 >> 172.21.15.23:0/2103890984 conn(0x56284f776240 msgr2=0x56284f76eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.185+0000 7f4111f8a740 1 -- 172.21.15.23:0/2103890984 shutdown_connections 2024-03-28T05:24:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.185+0000 7f4111f8a740 1 -- 172.21.15.23:0/2103890984 wait complete. 2024-03-28T05:24:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.185+0000 7f4111f8a740 1 -- start start 2024-03-28T05:24:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.185+0000 7f4111f8a740 1 Processor -- start 2024-03-28T05:24:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.186+0000 7f4111f8a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56284f77b920 0x56284f77ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.186+0000 7f4111f8a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77c240 0x56284f77f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.186+0000 7f4111f8a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f785010 0x56284f7e9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.186+0000 7f4111f8a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41081ab350 con 0x56284f77c240 2024-03-28T05:24:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.186+0000 7f4111f8a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41081ac670 con 0x56284f77b920 2024-03-28T05:24:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.186+0000 7f4111f8a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41081ab9b0 con 0x56284f785010 2024-03-28T05:24:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.186+0000 7f4111f8a740 10 client.0 _ll_register_callbacks cb 0x7f410409aca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:24:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.186+0000 7f410c946640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77c240 0x56284f77f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.186+0000 7f410c946640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77c240 0x56284f77f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50482/0 (socket says 172.21.15.23:50482) 2024-03-28T05:24:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.186+0000 7f410c946640 1 -- 172.21.15.23:0/1640137637 learned_addr learned my addr 172.21.15.23:0/1640137637 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:24:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.186+0000 7f410d948640 1 --2- 172.21.15.23:0/1640137637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f785010 0x56284f7e9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.186+0000 7f410c946640 1 -- 172.21.15.23:0/1640137637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f785010 msgr2=0x56284f7e9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.186+0000 7f410c946640 1 --2- 172.21.15.23:0/1640137637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f785010 0x56284f7e9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.186+0000 7f410c946640 1 -- 172.21.15.23:0/1640137637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56284f77b920 msgr2=0x56284f77ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:24:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.186+0000 7f410c946640 1 --2- 172.21.15.23:0/1640137637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56284f77b920 0x56284f77ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.186+0000 7f410c946640 1 -- 172.21.15.23:0/1640137637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56284f7ea1d0 con 0x56284f77c240 2024-03-28T05:24:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.186+0000 7f410d948640 1 --2- 172.21.15.23:0/1640137637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f785010 0x56284f7e9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:24:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.186+0000 7f410c946640 1 --2- 172.21.15.23:0/1640137637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77c240 0x56284f77f490 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f4104010fd0 tx=0x7f41040b9bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cad7c6cd0d386a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:24:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.186+0000 7f40f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:24:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.186+0000 7f40f0ff9640 1 -- 172.21.15.23:0/1640137637 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41040b5070 con 0x56284f77c240 2024-03-28T05:24:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.186+0000 7f40f0ff9640 1 -- 172.21.15.23:0/1640137637 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f41040ccda0 con 0x56284f77c240 2024-03-28T05:24:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.186+0000 7f4111f8a740 1 -- 172.21.15.23:0/1640137637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f410409f200 con 0x56284f77c240 2024-03-28T05:24:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.187+0000 7f40f0ff9640 1 -- 172.21.15.23:0/1640137637 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41040a3850 con 0x56284f77c240 2024-03-28T05:24:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.187+0000 7f4111f8a740 10 client.15591 fetch_fsmap learned FSMap version 20 2024-03-28T05:24:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.187+0000 7f4111f8a740 1 -- 172.21.15.23:0/1640137637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x56284f7ea1d0 con 0x56284f77c240 2024-03-28T05:24:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.187+0000 7f40f0ff9640 1 -- 172.21.15.23:0/1640137637 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f410409f200 con 0x56284f77c240 2024-03-28T05:24:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.187+0000 7f40f0ff9640 1 -- 172.21.15.23:0/1640137637 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f410409a020 con 0x56284f77c240 2024-03-28T05:24:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.187+0000 7f4111f8a740 10 client.15591 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:24:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.187+0000 7f4111f8a740 10 client.15591 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:24:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.187+0000 7f4111f8a740 20 client.15591 populate_metadata read hostname 'smithi023' 2024-03-28T05:24:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.187+0000 7f4111f8a740 1 --2- 172.21.15.23:0/1640137637 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x56284f7f17c0 0x56284f7f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:24:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.187+0000 7f4111f8a740 4 client.15591 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:24:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.187+0000 7f4111f8a740 1 -- 172.21.15.23:0/1640137637 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56284f6e53a0 con 0x56284f7f17c0 2024-03-28T05:24:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.187+0000 7f410d147640 1 --2- 172.21.15.23:0/1640137637 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x56284f7f17c0 0x56284f7f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:24:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.187+0000 7f40f0ff9640 10 client.15591 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:24:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.187+0000 7f410d147640 1 --2- 172.21.15.23:0/1640137637 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x56284f7f17c0 0x56284f7f3bb0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f577a38b2d69c68 server_cookie=11075e94b187a78b in_seq=0 out_seq=0 2024-03-28T05:24:55.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f40f0ff9640 1 -- 172.21.15.23:0/1640137637 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56284f6e53a0 con 0x56284f7f17c0 2024-03-28T05:24:55.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f40f0ff9640 10 client.15591 handle_command_reply: tid=0 2024-03-28T05:24:55.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f4111f8a740 1 client.15591 shutdown 2024-03-28T05:24:55.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f4111f8a740 1 -- 172.21.15.23:0/1640137637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77c240 msgr2=0x56284f77f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:55.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f4111f8a740 1 --2- 172.21.15.23:0/1640137637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77c240 0x56284f77f490 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f4104010fd0 tx=0x7f41040b9bb0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f4111f8a740 1 -- 172.21.15.23:0/1640137637 shutdown_connections 2024-03-28T05:24:55.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f4111f8a740 1 --2- 172.21.15.23:0/1640137637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56284f785010 0x56284f7e9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f4111f8a740 1 --2- 172.21.15.23:0/1640137637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56284f77c240 0x56284f77f490 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f4111f8a740 1 --2- 172.21.15.23:0/1640137637 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x56284f7f17c0 0x56284f7f3bb0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f4111f8a740 1 --2- 172.21.15.23:0/1640137637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56284f77b920 0x56284f77ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f40f0ff9640 0 client.15591 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:24:55.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f4111f8a740 1 -- 172.21.15.23:0/1640137637 >> 172.21.15.23:0/1640137637 conn(0x56284f776240 msgr2=0x56284f777640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:55.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f4111f8a740 1 -- 172.21.15.23:0/1640137637 shutdown_connections 2024-03-28T05:24:55.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f4111f8a740 1 -- 172.21.15.23:0/1640137637 wait complete. 2024-03-28T05:24:55.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f4111f8a740 20 client.15591 trim_cache size 0 max 16384 2024-03-28T05:24:55.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:24:55.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:24:55.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:24:55.194 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:24:55.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f410f3ab640 1 -- 172.21.15.23:0/2002858562 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f40f40a2b30 msgr2=0x7f40f40a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:55.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f410f3ab640 1 --2- 172.21.15.23:0/2002858562 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f40f40a2b30 0x7f40f40a4ff0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4100006010 tx=0x7f4100049000 comp rx=0 tx=0).stop 2024-03-28T05:24:55.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f410f3ab640 1 -- 172.21.15.23:0/2002858562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081011b0 msgr2=0x7f4108197300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:24:55.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f410f3ab640 1 --2- 172.21.15.23:0/2002858562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081011b0 0x7f4108197300 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f410400d950 tx=0x7f410400de10 comp rx=0 tx=0).stop 2024-03-28T05:24:55.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f410f3ab640 1 -- 172.21.15.23:0/2002858562 shutdown_connections 2024-03-28T05:24:55.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f410f3ab640 1 --2- 172.21.15.23:0/2002858562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4108105540 0x7f410819b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.195 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f410f3ab640 1 --2- 172.21.15.23:0/2002858562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41081011b0 0x7f4108197300 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.196 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f410f3ab640 1 --2- 172.21.15.23:0/2002858562 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f40f40a2b30 0x7f40f40a4ff0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.196 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.191+0000 7f410f3ab640 1 --2- 172.21.15.23:0/2002858562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41081007f0 0x7f4108196dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:24:55.196 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.192+0000 7f410f3ab640 1 -- 172.21.15.23:0/2002858562 >> 172.21.15.23:0/2002858562 conn(0x7f41080773f0 msgr2=0x7f41080fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:24:55.196 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.192+0000 7f410f3ab640 1 -- 172.21.15.23:0/2002858562 shutdown_connections 2024-03-28T05:24:55.196 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:24:55.192+0000 7f410f3ab640 1 -- 172.21.15.23:0/2002858562 wait complete. 2024-03-28T05:24:55.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:55 smithi098 ceph-mon[36087]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:55.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:55 smithi113 ceph-mon[37164]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:55.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:55 smithi023 ceph-mon[31954]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:55.847 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-28T05:24:55.904 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-28T05:24:55.944 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T05:24:55.945 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T05:24:55.945 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-28T05:24:55.984 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T05:24:55.988 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T05:24:55.988 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-28T05:24:56.042 INFO:tasks.workunit:Running workunits matching suites/blogbench.sh on client.0... 2024-03-28T05:24:56.044 INFO:tasks.workunit:Running workunit suites/blogbench.sh... 2024-03-28T05:24:56.045 DEBUG:teuthology.orchestra.run.smithi023:workunit test suites/blogbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh 2024-03-28T05:24:56.103 INFO:tasks.workunit.client.0.smithi023.stdout:getting blogbench 2024-03-28T05:24:56.103 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'getting blogbench' 2024-03-28T05:24:56.103 INFO:tasks.workunit.client.0.smithi023.stderr:+ wget http://download.ceph.com/qa/blogbench-1.0.tar.bz2 2024-03-28T05:24:56.125 INFO:tasks.workunit.client.0.smithi023.stderr:--2024-03-28 05:24:56-- http://download.ceph.com/qa/blogbench-1.0.tar.bz2 2024-03-28T05:24:56.126 INFO:tasks.workunit.client.0.smithi023.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-03-28T05:24:56.147 INFO:tasks.workunit.client.0.smithi023.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-03-28T05:24:56.168 INFO:tasks.workunit.client.0.smithi023.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-28T05:24:56.168 INFO:tasks.workunit.client.0.smithi023.stderr:Length: 103355 (101K) [application/octet-stream] 2024-03-28T05:24:56.169 INFO:tasks.workunit.client.0.smithi023.stderr:Saving to: ‘blogbench-1.0.tar.bz2’ 2024-03-28T05:24:56.172 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-28T05:24:56.210 INFO:tasks.workunit.client.0.smithi023.stderr: 0K .......... .......... .......... .......... .......... 49% 1.22M 0s 2024-03-28T05:24:56.230 INFO:tasks.workunit.client.0.smithi023.stderr: 50K .......... .......... .......... .......... .......... 99% 2.40M 0s 2024-03-28T05:24:56.230 INFO:tasks.workunit.client.0.smithi023.stderr: 100K 100% 1.74T=0.06s 2024-03-28T05:24:56.230 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-28T05:24:56.231 INFO:tasks.workunit.client.0.smithi023.stderr:2024-03-28 05:24:56 (1.64 MB/s) - ‘blogbench-1.0.tar.bz2’ saved [103355/103355] 2024-03-28T05:24:56.231 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-28T05:24:56.231 INFO:tasks.workunit.client.0.smithi023.stderr:+ tar -xvf blogbench-1.0.tar.bz2 2024-03-28T05:24:56.254 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/ 2024-03-28T05:24:56.258 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/m4/ 2024-03-28T05:24:56.259 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/m4/acx_pthread.m4 2024-03-28T05:24:56.260 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/man/ 2024-03-28T05:24:56.266 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/man/blogbench.8 2024-03-28T05:24:56.268 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/man/Makefile.am 2024-03-28T05:24:56.272 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/man/Makefile.in 2024-03-28T05:24:56.275 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/src/ 2024-03-28T05:24:56.279 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/src/mysnprintf.c 2024-03-28T05:24:56.280 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/src/mysnprintf.h 2024-03-28T05:24:56.284 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/src/blogbench.c 2024-03-28T05:24:56.287 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/src/blogbench.h 2024-03-28T05:24:56.290 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/src/fakesnprintf.c 2024-03-28T05:24:56.293 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/src/fakesnprintf.h 2024-03-28T05:24:56.296 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/src/blogbench_p.h 2024-03-28T05:24:56.299 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/src/bsd-getopt_long.c 2024-03-28T05:24:56.302 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/src/bsd-getopt_long.h 2024-03-28T05:24:56.305 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/src/Makefile.am 2024-03-28T05:24:56.307 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/src/Makefile.in 2024-03-28T05:24:56.310 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/src/process.c 2024-03-28T05:24:56.313 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/src/globals.h 2024-03-28T05:24:56.316 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/src/commenter.c 2024-03-28T05:24:56.319 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/src/rewriter.c 2024-03-28T05:24:56.322 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/src/reader.c 2024-03-28T05:24:56.324 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/src/writer.c 2024-03-28T05:24:56.327 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/src/helpers.c 2024-03-28T05:24:56.330 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/NEWS 2024-03-28T05:24:56.335 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/compile 2024-03-28T05:24:56.338 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/depcomp 2024-03-28T05:24:56.341 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/aclocal.m4 2024-03-28T05:24:56.344 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/README 2024-03-28T05:24:56.346 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/configure 2024-03-28T05:24:56.351 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/configure.ac 2024-03-28T05:24:56.354 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/config.guess 2024-03-28T05:24:56.357 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/install-sh 2024-03-28T05:24:56.359 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/config.sub 2024-03-28T05:24:56.362 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/missing 2024-03-28T05:24:56.364 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/mkinstalldirs 2024-03-28T05:24:56.367 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/Makefile.am 2024-03-28T05:24:56.370 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/Makefile.in 2024-03-28T05:24:56.372 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/config.h.in 2024-03-28T05:24:56.375 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/AUTHORS 2024-03-28T05:24:56.377 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/INSTALL 2024-03-28T05:24:56.380 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/ChangeLog 2024-03-28T05:24:56.382 INFO:tasks.workunit.client.0.smithi023.stdout:blogbench-1.0/COPYING 2024-03-28T05:24:56.387 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd blogbench-1.0/ 2024-03-28T05:24:56.387 INFO:tasks.workunit.client.0.smithi023.stdout:making blogbench 2024-03-28T05:24:56.387 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'making blogbench' 2024-03-28T05:24:56.387 INFO:tasks.workunit.client.0.smithi023.stderr:+ ./configure 2024-03-28T05:24:56.478 INFO:tasks.workunit.client.0.smithi023.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-28T05:24:57.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:57 smithi098 ceph-mon[36087]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:57.491 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether build environment is sane... yes 2024-03-28T05:24:57.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:57 smithi113 ceph-mon[37164]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:57.509 INFO:tasks.workunit.client.0.smithi023.stdout:checking for gawk... gawk 2024-03-28T05:24:57.527 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether make sets $(MAKE)... yes 2024-03-28T05:24:57.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:57 smithi023 ceph-mon[31954]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-28T05:24:57.543 INFO:tasks.workunit.client.0.smithi023.stdout:checking for gcc... gcc 2024-03-28T05:24:57.598 INFO:tasks.workunit.client.0.smithi023.stdout:checking for C compiler default output file name... a.out 2024-03-28T05:24:57.600 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether the C compiler works... yes 2024-03-28T05:24:57.606 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether we are cross compiling... no 2024-03-28T05:24:57.646 INFO:tasks.workunit.client.0.smithi023.stdout:checking for suffix of executables... 2024-03-28T05:24:57.690 INFO:tasks.workunit.client.0.smithi023.stdout:checking for suffix of object files... o 2024-03-28T05:24:57.748 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether we are using the GNU C compiler... yes 2024-03-28T05:24:57.801 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether gcc accepts -g... yes 2024-03-28T05:24:57.885 INFO:tasks.workunit.client.0.smithi023.stdout:checking for gcc option to accept ANSI C... none needed 2024-03-28T05:24:57.930 INFO:tasks.workunit.client.0.smithi023.stdout:checking for style of include used by make... GNU 2024-03-28T05:24:58.091 INFO:tasks.workunit.client.0.smithi023.stdout:checking dependency style of gcc... gcc3 2024-03-28T05:24:58.091 INFO:tasks.workunit.client.0.smithi023.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-28T05:24:58.191 INFO:tasks.workunit.client.0.smithi023.stdout:checking how to run the C preprocessor... gcc -E 2024-03-28T05:24:58.276 INFO:tasks.workunit.client.0.smithi023.stdout:checking for egrep... grep -E 2024-03-28T05:24:58.287 INFO:tasks.workunit.client.0.smithi023.stdout:checking for AIX... no 2024-03-28T05:24:58.366 INFO:tasks.workunit.client.0.smithi023.stdout:checking for library containing strerror... none required 2024-03-28T05:24:58.556 INFO:tasks.workunit.client.0.smithi023.stdout:checking for ANSI C header files... yes 2024-03-28T05:24:58.624 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-28T05:24:58.683 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/types.h... yes 2024-03-28T05:24:58.744 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/stat.h... yes 2024-03-28T05:24:58.805 INFO:tasks.workunit.client.0.smithi023.stdout:checking for stdlib.h... yes 2024-03-28T05:24:58.866 INFO:tasks.workunit.client.0.smithi023.stdout:checking for string.h... yes 2024-03-28T05:24:58.931 INFO:tasks.workunit.client.0.smithi023.stdout:checking for memory.h... yes 2024-03-28T05:24:58.992 INFO:tasks.workunit.client.0.smithi023.stdout:checking for strings.h... yes 2024-03-28T05:24:59.055 INFO:tasks.workunit.client.0.smithi023.stdout:checking for inttypes.h... yes 2024-03-28T05:24:59.119 INFO:tasks.workunit.client.0.smithi023.stdout:checking for stdint.h... yes 2024-03-28T05:24:59.182 INFO:tasks.workunit.client.0.smithi023.stdout:checking for unistd.h... yes 2024-03-28T05:24:59.189 INFO:tasks.workunit.client.0.smithi023.stdout:checking for memory.h... (cached) yes 2024-03-28T05:24:59.193 INFO:tasks.workunit.client.0.smithi023.stdout:checking for strings.h... (cached) yes 2024-03-28T05:24:59.198 INFO:tasks.workunit.client.0.smithi023.stdout:checking for unistd.h... (cached) yes 2024-03-28T05:24:59.267 INFO:tasks.workunit.client.0.smithi023.stdout:checking locale.h usability... yes 2024-03-28T05:24:59.299 INFO:tasks.workunit.client.0.smithi023.stdout:checking locale.h presence... yes 2024-03-28T05:24:59.300 INFO:tasks.workunit.client.0.smithi023.stdout:checking for locale.h... yes 2024-03-28T05:24:59.306 INFO:tasks.workunit.client.0.smithi023.stdout:checking for stdlib.h... (cached) yes 2024-03-28T05:24:59.311 INFO:tasks.workunit.client.0.smithi023.stdout:checking for string.h... (cached) yes 2024-03-28T05:24:59.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:24:59 smithi098 ceph-mon[36087]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:24:59.391 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/socket.h usability... yes 2024-03-28T05:24:59.425 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/socket.h presence... yes 2024-03-28T05:24:59.426 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/socket.h... yes 2024-03-28T05:24:59.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:24:59 smithi113 ceph-mon[37164]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:24:59.493 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/time.h usability... yes 2024-03-28T05:24:59.525 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/time.h presence... yes 2024-03-28T05:24:59.526 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/time.h... yes 2024-03-28T05:24:59.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:24:59 smithi023 ceph-mon[31954]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:24:59.588 INFO:tasks.workunit.client.0.smithi023.stdout:checking fcntl.h usability... yes 2024-03-28T05:24:59.620 INFO:tasks.workunit.client.0.smithi023.stdout:checking fcntl.h presence... yes 2024-03-28T05:24:59.621 INFO:tasks.workunit.client.0.smithi023.stdout:checking for fcntl.h... yes 2024-03-28T05:24:59.698 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/fcntl.h usability... yes 2024-03-28T05:24:59.729 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/fcntl.h presence... yes 2024-03-28T05:24:59.730 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/fcntl.h... yes 2024-03-28T05:24:59.813 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/param.h usability... yes 2024-03-28T05:24:59.849 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/param.h presence... yes 2024-03-28T05:24:59.849 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/param.h... yes 2024-03-28T05:24:59.901 INFO:tasks.workunit.client.0.smithi023.stdout:checking ioctl.h usability... no 2024-03-28T05:24:59.936 INFO:tasks.workunit.client.0.smithi023.stdout:checking ioctl.h presence... no 2024-03-28T05:24:59.937 INFO:tasks.workunit.client.0.smithi023.stdout:checking for ioctl.h... no 2024-03-28T05:25:00.001 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/ioctl.h usability... yes 2024-03-28T05:25:00.039 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/ioctl.h presence... yes 2024-03-28T05:25:00.040 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/ioctl.h... yes 2024-03-28T05:25:00.106 INFO:tasks.workunit.client.0.smithi023.stdout:checking return type of signal handlers... void 2024-03-28T05:25:00.179 INFO:tasks.workunit.client.0.smithi023.stdout:checking for size_t... yes 2024-03-28T05:25:00.245 INFO:tasks.workunit.client.0.smithi023.stdout:checking for dev_t... yes 2024-03-28T05:25:00.312 INFO:tasks.workunit.client.0.smithi023.stdout:checking for short... yes 2024-03-28T05:25:00.386 INFO:tasks.workunit.client.0.smithi023.stdout:checking size of short... 2 2024-03-28T05:25:00.453 INFO:tasks.workunit.client.0.smithi023.stdout:checking for int... yes 2024-03-28T05:25:00.528 INFO:tasks.workunit.client.0.smithi023.stdout:checking size of int... 4 2024-03-28T05:25:00.604 INFO:tasks.workunit.client.0.smithi023.stdout:checking for long... yes 2024-03-28T05:25:00.680 INFO:tasks.workunit.client.0.smithi023.stdout:checking size of long... 8 2024-03-28T05:25:00.747 INFO:tasks.workunit.client.0.smithi023.stdout:checking for long long... yes 2024-03-28T05:25:00.820 INFO:tasks.workunit.client.0.smithi023.stdout:checking size of long long... 8 2024-03-28T05:25:00.886 INFO:tasks.workunit.client.0.smithi023.stdout:checking for ssize_t... yes 2024-03-28T05:25:00.948 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sig_atomic_t... yes 2024-03-28T05:25:00.996 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether gcc needs -traditional... no 2024-03-28T05:25:01.050 INFO:tasks.workunit.client.0.smithi023.stdout:checking for an ANSI C-conforming const... yes 2024-03-28T05:25:01.085 INFO:tasks.workunit.client.0.smithi023.stdout:checking for inline... inline 2024-03-28T05:25:01.158 INFO:tasks.workunit.client.0.smithi023.stdout:checking for working alloca.h... yes 2024-03-28T05:25:01.231 INFO:tasks.workunit.client.0.smithi023.stdout:checking for alloca... yes 2024-03-28T05:25:01.298 INFO:tasks.workunit.client.0.smithi023.stdout:checking for working memcmp... yes 2024-03-28T05:25:01.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:01 smithi098 ceph-mon[36087]: pgmap v46: 129 pgs: 129 active+clean; 2.6 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 125 KiB/s wr, 33 op/s 2024-03-28T05:25:01.372 INFO:tasks.workunit.client.0.smithi023.stdout:checking for strftime... yes 2024-03-28T05:25:01.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:01 smithi113 ceph-mon[37164]: pgmap v46: 129 pgs: 129 active+clean; 2.6 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 125 KiB/s wr, 33 op/s 2024-03-28T05:25:01.522 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether lstat dereferences a symlink specified with a trailing slash... yes 2024-03-28T05:25:01.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:01 smithi023 ceph-mon[31954]: pgmap v46: 129 pgs: 129 active+clean; 2.6 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 125 KiB/s wr, 33 op/s 2024-03-28T05:25:01.588 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether stat accepts an empty string... no 2024-03-28T05:25:01.664 INFO:tasks.workunit.client.0.smithi023.stdout:checking for vprintf... yes 2024-03-28T05:25:01.740 INFO:tasks.workunit.client.0.smithi023.stdout:checking for _doprnt... no 2024-03-28T05:25:01.762 INFO:tasks.workunit.client.0.smithi023.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-28T05:25:01.773 INFO:tasks.workunit.client.0.smithi023.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-28T05:25:01.853 INFO:tasks.workunit.client.0.smithi023.stdout:checking for the pthreads library -lpthreads... no 2024-03-28T05:25:01.932 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether pthreads work without any flags... yes 2024-03-28T05:25:01.991 INFO:tasks.workunit.client.0.smithi023.stdout:checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE 2024-03-28T05:25:01.992 INFO:tasks.workunit.client.0.smithi023.stdout:checking if more special flags are required for pthreads... no 2024-03-28T05:25:01.992 INFO:tasks.workunit.client.0.smithi023.stdout:checking for cc_r... gcc 2024-03-28T05:25:02.069 INFO:tasks.workunit.client.0.smithi023.stdout:checking for setrlimit... yes 2024-03-28T05:25:02.147 INFO:tasks.workunit.client.0.smithi023.stdout:checking for setproctitle... no 2024-03-28T05:25:02.224 INFO:tasks.workunit.client.0.smithi023.stdout:checking for getopt_long... yes 2024-03-28T05:25:02.303 INFO:tasks.workunit.client.0.smithi023.stdout:checking for strtoull... yes 2024-03-28T05:25:02.430 INFO:tasks.workunit.client.0.smithi023.stdout:checking for strtoq... yes 2024-03-28T05:25:02.505 INFO:tasks.workunit.client.0.smithi023.stdout:checking for snprintf... yes 2024-03-28T05:25:02.580 INFO:tasks.workunit.client.0.smithi023.stdout:checking for vsnprintf... yes 2024-03-28T05:25:02.655 INFO:tasks.workunit.client.0.smithi023.stdout:checking for vfprintf... yes 2024-03-28T05:25:02.731 INFO:tasks.workunit.client.0.smithi023.stdout:checking for gethostname... yes 2024-03-28T05:25:02.815 INFO:tasks.workunit.client.0.smithi023.stdout:checking for setlocale... yes 2024-03-28T05:25:02.900 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sync... yes 2024-03-28T05:25:02.976 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether snprintf is C99 conformant... done 2024-03-28T05:25:02.991 INFO:tasks.workunit.client.0.smithi023.stdout:configure: creating ./config.status 2024-03-28T05:25:03.202 INFO:tasks.workunit.client.0.smithi023.stdout:config.status: creating Makefile 2024-03-28T05:25:03.223 INFO:tasks.workunit.client.0.smithi023.stdout:config.status: creating src/Makefile 2024-03-28T05:25:03.245 INFO:tasks.workunit.client.0.smithi023.stdout:config.status: creating man/Makefile 2024-03-28T05:25:03.261 INFO:tasks.workunit.client.0.smithi023.stdout:config.status: creating config.h 2024-03-28T05:25:03.334 INFO:tasks.workunit.client.0.smithi023.stdout:config.status: executing depfiles commands 2024-03-28T05:25:03.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:03 smithi098 ceph-mon[36087]: pgmap v47: 129 pgs: 129 active+clean; 2.6 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 125 KiB/s wr, 33 op/s 2024-03-28T05:25:03.486 INFO:tasks.workunit.client.0.smithi023.stderr:+ make 2024-03-28T05:25:03.491 INFO:tasks.workunit.client.0.smithi023.stdout:make all-recursive 2024-03-28T05:25:03.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:03 smithi113 ceph-mon[37164]: pgmap v47: 129 pgs: 129 active+clean; 2.6 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 125 KiB/s wr, 33 op/s 2024-03-28T05:25:03.493 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-28T05:25:03.495 INFO:tasks.workunit.client.0.smithi023.stdout:Making all in src 2024-03-28T05:25:03.504 INFO:tasks.workunit.client.0.smithi023.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-03-28T05:25:03.504 INFO:tasks.workunit.client.0.smithi023.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT blogbench.o -MD -MP -MF ".deps/blogbench.Tpo" -c -o blogbench.o blogbench.c; \ 2024-03-28T05:25:03.504 INFO:tasks.workunit.client.0.smithi023.stdout:then mv -f ".deps/blogbench.Tpo" ".deps/blogbench.Po"; else rm -f ".deps/blogbench.Tpo"; exit 1; fi 2024-03-28T05:25:03.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:03 smithi023 ceph-mon[31954]: pgmap v47: 129 pgs: 129 active+clean; 2.6 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 125 KiB/s wr, 33 op/s 2024-03-28T05:25:03.608 INFO:tasks.workunit.client.0.smithi023.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT process.o -MD -MP -MF ".deps/process.Tpo" -c -o process.o process.c; \ 2024-03-28T05:25:03.608 INFO:tasks.workunit.client.0.smithi023.stdout:then mv -f ".deps/process.Tpo" ".deps/process.Po"; else rm -f ".deps/process.Tpo"; exit 1; fi 2024-03-28T05:25:03.708 INFO:tasks.workunit.client.0.smithi023.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT writer.o -MD -MP -MF ".deps/writer.Tpo" -c -o writer.o writer.c; \ 2024-03-28T05:25:03.708 INFO:tasks.workunit.client.0.smithi023.stdout:then mv -f ".deps/writer.Tpo" ".deps/writer.Po"; else rm -f ".deps/writer.Tpo"; exit 1; fi 2024-03-28T05:25:03.784 INFO:tasks.workunit.client.0.smithi023.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT rewriter.o -MD -MP -MF ".deps/rewriter.Tpo" -c -o rewriter.o rewriter.c; \ 2024-03-28T05:25:03.784 INFO:tasks.workunit.client.0.smithi023.stdout:then mv -f ".deps/rewriter.Tpo" ".deps/rewriter.Po"; else rm -f ".deps/rewriter.Tpo"; exit 1; fi 2024-03-28T05:25:03.857 INFO:tasks.workunit.client.0.smithi023.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT reader.o -MD -MP -MF ".deps/reader.Tpo" -c -o reader.o reader.c; \ 2024-03-28T05:25:03.857 INFO:tasks.workunit.client.0.smithi023.stdout:then mv -f ".deps/reader.Tpo" ".deps/reader.Po"; else rm -f ".deps/reader.Tpo"; exit 1; fi 2024-03-28T05:25:03.929 INFO:tasks.workunit.client.0.smithi023.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT commenter.o -MD -MP -MF ".deps/commenter.Tpo" -c -o commenter.o commenter.c; \ 2024-03-28T05:25:03.929 INFO:tasks.workunit.client.0.smithi023.stdout:then mv -f ".deps/commenter.Tpo" ".deps/commenter.Po"; else rm -f ".deps/commenter.Tpo"; exit 1; fi 2024-03-28T05:25:03.994 INFO:tasks.workunit.client.0.smithi023.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT helpers.o -MD -MP -MF ".deps/helpers.Tpo" -c -o helpers.o helpers.c; \ 2024-03-28T05:25:03.994 INFO:tasks.workunit.client.0.smithi023.stdout:then mv -f ".deps/helpers.Tpo" ".deps/helpers.Po"; else rm -f ".deps/helpers.Tpo"; exit 1; fi 2024-03-28T05:25:04.089 INFO:tasks.workunit.client.0.smithi023.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT bsd-getopt_long.o -MD -MP -MF ".deps/bsd-getopt_long.Tpo" -c -o bsd-getopt_long.o bsd-getopt_long.c; \ 2024-03-28T05:25:04.090 INFO:tasks.workunit.client.0.smithi023.stdout:then mv -f ".deps/bsd-getopt_long.Tpo" ".deps/bsd-getopt_long.Po"; else rm -f ".deps/bsd-getopt_long.Tpo"; exit 1; fi 2024-03-28T05:25:04.143 INFO:tasks.workunit.client.0.smithi023.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT fakesnprintf.o -MD -MP -MF ".deps/fakesnprintf.Tpo" -c -o fakesnprintf.o fakesnprintf.c; \ 2024-03-28T05:25:04.143 INFO:tasks.workunit.client.0.smithi023.stdout:then mv -f ".deps/fakesnprintf.Tpo" ".deps/fakesnprintf.Po"; else rm -f ".deps/fakesnprintf.Tpo"; exit 1; fi 2024-03-28T05:25:04.198 INFO:tasks.workunit.client.0.smithi023.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT mysnprintf.o -MD -MP -MF ".deps/mysnprintf.Tpo" -c -o mysnprintf.o mysnprintf.c; \ 2024-03-28T05:25:04.199 INFO:tasks.workunit.client.0.smithi023.stdout:then mv -f ".deps/mysnprintf.Tpo" ".deps/mysnprintf.Po"; else rm -f ".deps/mysnprintf.Tpo"; exit 1; fi 2024-03-28T05:25:04.248 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -g -O2 -o blogbench blogbench.o process.o writer.o rewriter.o reader.o commenter.o helpers.o bsd-getopt_long.o fakesnprintf.o mysnprintf.o 2024-03-28T05:25:04.278 INFO:tasks.workunit.client.0.smithi023.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-03-28T05:25:04.279 INFO:tasks.workunit.client.0.smithi023.stdout:Making all in man 2024-03-28T05:25:04.281 INFO:tasks.workunit.client.0.smithi023.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-03-28T05:25:04.281 INFO:tasks.workunit.client.0.smithi023.stdout:make[2]: Nothing to be done for 'all'. 2024-03-28T05:25:04.281 INFO:tasks.workunit.client.0.smithi023.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-03-28T05:25:04.282 INFO:tasks.workunit.client.0.smithi023.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-28T05:25:04.283 INFO:tasks.workunit.client.0.smithi023.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-28T05:25:04.283 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-28T05:25:04.284 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd src 2024-03-28T05:25:04.284 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir blogtest_in 2024-03-28T05:25:04.289 INFO:tasks.workunit.client.0.smithi023.stdout:running blogbench 2024-03-28T05:25:04.289 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'running blogbench' 2024-03-28T05:25:04.289 INFO:tasks.workunit.client.0.smithi023.stderr:+ ./blogbench -d blogtest_in 2024-03-28T05:25:04.290 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-28T05:25:04.290 INFO:tasks.workunit.client.0.smithi023.stdout:Frequency = 10 secs 2024-03-28T05:25:04.290 INFO:tasks.workunit.client.0.smithi023.stdout:Scratch dir = [blogtest_in] 2024-03-28T05:25:05.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:05 smithi098 ceph-mon[36087]: pgmap v48: 129 pgs: 129 active+clean; 2.6 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 125 KiB/s wr, 33 op/s 2024-03-28T05:25:05.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:05 smithi113 ceph-mon[37164]: pgmap v48: 129 pgs: 129 active+clean; 2.6 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 125 KiB/s wr, 33 op/s 2024-03-28T05:25:05.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:05 smithi023 ceph-mon[31954]: pgmap v48: 129 pgs: 129 active+clean; 2.6 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 125 KiB/s wr, 33 op/s 2024-03-28T05:25:06.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:06 smithi098 ceph-mon[36087]: mgrmap e26: x(active, since 91s), standbys: y, z 2024-03-28T05:25:06.395 INFO:tasks.workunit.client.0.smithi023.stdout:Spawning 3 writers... 2024-03-28T05:25:06.396 INFO:tasks.workunit.client.0.smithi023.stdout:Spawning 1 rewriters... 2024-03-28T05:25:06.396 INFO:tasks.workunit.client.0.smithi023.stdout:Spawning 5 commenters... 2024-03-28T05:25:06.396 INFO:tasks.workunit.client.0.smithi023.stdout:Spawning 100 readers... 2024-03-28T05:25:06.398 INFO:tasks.workunit.client.0.smithi023.stdout:Benchmarking for 30 iterations. 2024-03-28T05:25:06.398 INFO:tasks.workunit.client.0.smithi023.stdout:The test will run during 5 minutes. 2024-03-28T05:25:06.398 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-28T05:25:06.398 INFO:tasks.workunit.client.0.smithi023.stdout: Nb blogs R articles W articles R pictures W pictures R comments W comments 2024-03-28T05:25:06.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:06 smithi113 ceph-mon[37164]: mgrmap e26: x(active, since 91s), standbys: y, z 2024-03-28T05:25:06.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:06 smithi023 ceph-mon[31954]: mgrmap e26: x(active, since 91s), standbys: y, z 2024-03-28T05:25:07.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:07 smithi098 ceph-mon[36087]: pgmap v49: 129 pgs: 129 active+clean; 6.3 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 536 KiB/s wr, 120 op/s 2024-03-28T05:25:07.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:07 smithi113 ceph-mon[37164]: pgmap v49: 129 pgs: 129 active+clean; 6.3 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 536 KiB/s wr, 120 op/s 2024-03-28T05:25:07.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:07 smithi023 ceph-mon[31954]: pgmap v49: 129 pgs: 129 active+clean; 6.3 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 536 KiB/s wr, 120 op/s 2024-03-28T05:25:08.289 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.287+0000 7f335cb76640 1 -- 172.21.15.23:0/3856594997 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3358101190 msgr2=0x7f335810d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.287+0000 7f335cb76640 1 --2- 172.21.15.23:0/3856594997 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3358101190 0x7f335810d3d0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3350009ca0 tx=0x7f335002c0f0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.287+0000 7f335cb76640 1 -- 172.21.15.23:0/3856594997 shutdown_connections 2024-03-28T05:25:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.287+0000 7f335cb76640 1 --2- 172.21.15.23:0/3856594997 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3358101190 0x7f335810d3d0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.287+0000 7f335cb76640 1 --2- 172.21.15.23:0/3856594997 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f33581007f0 0x7f3358100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.287+0000 7f335cb76640 1 --2- 172.21.15.23:0/3856594997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3358105520 0x7f3358105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.287+0000 7f335cb76640 1 -- 172.21.15.23:0/3856594997 >> 172.21.15.23:0/3856594997 conn(0x7f3358077370 msgr2=0x7f3358077780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.287+0000 7f335cb76640 1 -- 172.21.15.23:0/3856594997 shutdown_connections 2024-03-28T05:25:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.287+0000 7f335cb76640 1 -- 172.21.15.23:0/3856594997 wait complete. 2024-03-28T05:25:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.288+0000 7f335cb76640 1 Processor -- start 2024-03-28T05:25:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.288+0000 7f335cb76640 1 -- start start 2024-03-28T05:25:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.288+0000 7f335cb76640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33581007f0 0x7f3358197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:08.290 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.288+0000 7f335cb76640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3358101190 0x7f3358198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.288+0000 7f335cb76640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3358105520 0x7f335819c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.288+0000 7f335cb76640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3358111b50 con 0x7f3358105520 2024-03-28T05:25:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.288+0000 7f335cb76640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33581119d0 con 0x7f33581007f0 2024-03-28T05:25:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.288+0000 7f335cb76640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3358111cd0 con 0x7f3358101190 2024-03-28T05:25:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.288+0000 7f3355d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3358101190 0x7f3358198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.288+0000 7f3356d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3358105520 0x7f335819c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.288+0000 7f3355d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3358101190 0x7f3358198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:42784/0 (socket says 172.21.15.23:42784) 2024-03-28T05:25:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.288+0000 7f3355d9b640 1 -- 172.21.15.23:0/2660937594 learned_addr learned my addr 172.21.15.23:0/2660937594 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:25:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.289+0000 7f335659c640 1 --2- 172.21.15.23:0/2660937594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33581007f0 0x7f3358197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.289+0000 7f3355d9b640 1 -- 172.21.15.23:0/2660937594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33581007f0 msgr2=0x7f3358197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.289+0000 7f3355d9b640 1 --2- 172.21.15.23:0/2660937594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33581007f0 0x7f3358197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.289+0000 7f3355d9b640 1 -- 172.21.15.23:0/2660937594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3358105520 msgr2=0x7f335819c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:08.291 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.289+0000 7f3355d9b640 1 --2- 172.21.15.23:0/2660937594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3358105520 0x7f335819c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.289+0000 7f3355d9b640 1 -- 172.21.15.23:0/2660937594 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f335819ceb0 con 0x7f3358101190 2024-03-28T05:25:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.289+0000 7f3356d9d640 1 --2- 172.21.15.23:0/2660937594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3358105520 0x7f335819c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:25:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.289+0000 7f335659c640 1 --2- 172.21.15.23:0/2660937594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33581007f0 0x7f3358197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:25:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.289+0000 7f3355d9b640 1 --2- 172.21.15.23:0/2660937594 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3358101190 0x7f3358198450 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f334c00e9f0 tx=0x7f334c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.289+0000 7f33477fe640 1 -- 172.21.15.23:0/2660937594 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c00cd80 con 0x7f3358101190 2024-03-28T05:25:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.289+0000 7f335cb76640 1 -- 172.21.15.23:0/2660937594 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f335819d1a0 con 0x7f3358101190 2024-03-28T05:25:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.289+0000 7f33477fe640 1 -- 172.21.15.23:0/2660937594 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f334c005500 con 0x7f3358101190 2024-03-28T05:25:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.289+0000 7f33477fe640 1 -- 172.21.15.23:0/2660937594 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c010690 con 0x7f3358101190 2024-03-28T05:25:08.292 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.289+0000 7f335cb76640 1 -- 172.21.15.23:0/2660937594 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33581a4a30 con 0x7f3358101190 2024-03-28T05:25:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.291+0000 7f33477fe640 1 -- 172.21.15.23:0/2660937594 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f334c0046c0 con 0x7f3358101190 2024-03-28T05:25:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.292+0000 7f335f834740 1 Processor -- start 2024-03-28T05:25:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.292+0000 7f335f834740 1 -- start start 2024-03-28T05:25:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.292+0000 7f335f834740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5b710 0x559641a5baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.292+0000 7f335f834740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559641a5c030 0x559641a648c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.292+0000 7f335f834740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559641a64e00 0x559641a671f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.292+0000 7f335f834740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596419bfb10 con 0x559641a5b710 2024-03-28T05:25:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.292+0000 7f335f834740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x559641968900 con 0x559641a64e00 2024-03-28T05:25:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.292+0000 7f335f834740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596419bbbb0 con 0x559641a5c030 2024-03-28T05:25:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.292+0000 7f3356d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5b710 0x559641a5baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.292+0000 7f3356d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5b710 0x559641a5baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32780/0 (socket says 172.21.15.23:32780) 2024-03-28T05:25:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.292+0000 7f3356d9d640 1 -- 172.21.15.23:0/3406093716 learned_addr learned my addr 172.21.15.23:0/3406093716 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:25:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.292+0000 7f335659c640 1 --2- 172.21.15.23:0/3406093716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559641a5c030 0x559641a648c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.292+0000 7f33477fe640 1 --2- 172.21.15.23:0/2660937594 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f33400a2a10 0x7f33400a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.292+0000 7f3355d9b640 1 --2- 172.21.15.23:0/3406093716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559641a64e00 0x559641a671f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.292+0000 7f3356d9d640 1 --2- 172.21.15.23:0/2660937594 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f33400a2a10 0x7f33400a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.293+0000 7f335659c640 1 -- 172.21.15.23:0/3406093716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559641a64e00 msgr2=0x559641a671f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.293+0000 7f335659c640 1 --2- 172.21.15.23:0/3406093716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559641a64e00 0x559641a671f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.293+0000 7f335659c640 1 -- 172.21.15.23:0/3406093716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5b710 msgr2=0x559641a5baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.293+0000 7f335659c640 1 --2- 172.21.15.23:0/3406093716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5b710 0x559641a5baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.293+0000 7f335659c640 1 -- 172.21.15.23:0/3406093716 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5596416d4210 con 0x559641a5c030 2024-03-28T05:25:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.293+0000 7f3355d9b640 1 --2- 172.21.15.23:0/3406093716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559641a64e00 0x559641a671f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:25:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.293+0000 7f33477fe640 1 -- 172.21.15.23:0/2660937594 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f334c0cf690 con 0x7f3358101190 2024-03-28T05:25:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.293+0000 7f335659c640 1 --2- 172.21.15.23:0/3406093716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559641a5c030 0x559641a648c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3348009c30 tx=0x7f33480117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9d7c5432254626d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.293+0000 7f3356d9d640 1 --2- 172.21.15.23:0/2660937594 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f33400a2a10 0x7f33400a4ed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3350006010 tx=0x7f335002c0f0 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.293+0000 7f3344ff9640 1 -- 172.21.15.23:0/3406093716 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348012450 con 0x559641a5c030 2024-03-28T05:25:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.293+0000 7f3344ff9640 1 -- 172.21.15.23:0/3406093716 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3348012ad0 con 0x559641a5c030 2024-03-28T05:25:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.293+0000 7f3344ff9640 1 -- 172.21.15.23:0/3406093716 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334800fb00 con 0x559641a5c030 2024-03-28T05:25:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.294+0000 7f335f834740 1 -- 172.21.15.23:0/3406093716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559641a5c030 msgr2=0x559641a648c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.294+0000 7f335f834740 1 --2- 172.21.15.23:0/3406093716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559641a5c030 0x559641a648c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3348009c30 tx=0x7f33480117b0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.294+0000 7f335f834740 1 -- 172.21.15.23:0/3406093716 shutdown_connections 2024-03-28T05:25:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.294+0000 7f335f834740 1 --2- 172.21.15.23:0/3406093716 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559641a64e00 0x559641a671f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.294+0000 7f335f834740 1 --2- 172.21.15.23:0/3406093716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559641a5c030 0x559641a648c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.294+0000 7f335f834740 1 --2- 172.21.15.23:0/3406093716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5b710 0x559641a5baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.294+0000 7f335f834740 1 -- 172.21.15.23:0/3406093716 >> 172.21.15.23:0/3406093716 conn(0x559641a56030 msgr2=0x559641a56410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.294+0000 7f335f834740 1 -- 172.21.15.23:0/3406093716 shutdown_connections 2024-03-28T05:25:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.294+0000 7f335f834740 1 -- 172.21.15.23:0/3406093716 wait complete. 2024-03-28T05:25:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.294+0000 7f335f834740 1 -- start start 2024-03-28T05:25:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.294+0000 7f335f834740 1 Processor -- start 2024-03-28T05:25:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f335f834740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559641a5b710 0x559641a61dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f335f834740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5c030 0x559641a62310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f335f834740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559641a64e00 0x559641a62850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f335f834740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559641a67a30 con 0x559641a5c030 2024-03-28T05:25:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f335f834740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55964197bd50 con 0x559641a5b710 2024-03-28T05:25:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f335f834740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x559641a67bb0 con 0x559641a64e00 2024-03-28T05:25:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f335f834740 10 client.0 _ll_register_callbacks cb 0x55964197b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:25:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f3355d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5c030 0x559641a62310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f3355d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5c030 0x559641a62310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32796/0 (socket says 172.21.15.23:32796) 2024-03-28T05:25:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f3355d9b640 1 -- 172.21.15.23:0/1526644026 learned_addr learned my addr 172.21.15.23:0/1526644026 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:25:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f3356d9d640 1 --2- 172.21.15.23:0/1526644026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559641a64e00 0x559641a62850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f335659c640 1 --2- 172.21.15.23:0/1526644026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559641a5b710 0x559641a61dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f3355d9b640 1 -- 172.21.15.23:0/1526644026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559641a64e00 msgr2=0x559641a62850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f3355d9b640 1 --2- 172.21.15.23:0/1526644026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559641a64e00 0x559641a62850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f3355d9b640 1 -- 172.21.15.23:0/1526644026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559641a5b710 msgr2=0x559641a61dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f3355d9b640 1 --2- 172.21.15.23:0/1526644026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559641a5b710 0x559641a61dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f3355d9b640 1 -- 172.21.15.23:0/1526644026 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5596417a0620 con 0x559641a5c030 2024-03-28T05:25:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f3355d9b640 1 --2- 172.21.15.23:0/1526644026 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5c030 0x559641a62310 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f334c097dc0 tx=0x7f334c09b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fee69d373120b025 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f333dffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:25:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f333dffb640 1 -- 172.21.15.23:0/1526644026 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c0cf4a0 con 0x559641a5c030 2024-03-28T05:25:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f333dffb640 1 -- 172.21.15.23:0/1526644026 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f334c0bbbe0 con 0x559641a5c030 2024-03-28T05:25:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f335f834740 1 -- 172.21.15.23:0/1526644026 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3348012ad0 con 0x559641a5c030 2024-03-28T05:25:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f333dffb640 1 -- 172.21.15.23:0/1526644026 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f334c09b960 con 0x559641a5c030 2024-03-28T05:25:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f335f834740 10 client.15603 fetch_fsmap learned FSMap version 20 2024-03-28T05:25:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.295+0000 7f335f834740 1 -- 172.21.15.23:0/1526644026 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x559641a73ce0 con 0x559641a5c030 2024-03-28T05:25:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.296+0000 7f333dffb640 1 -- 172.21.15.23:0/1526644026 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3348012ad0 con 0x559641a5c030 2024-03-28T05:25:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.296+0000 7f333dffb640 1 -- 172.21.15.23:0/1526644026 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f334c0bdce0 con 0x559641a5c030 2024-03-28T05:25:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.296+0000 7f335f834740 10 client.15603 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:25:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.296+0000 7f335f834740 10 client.15603 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:25:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.296+0000 7f335f834740 20 client.15603 populate_metadata read hostname 'smithi023' 2024-03-28T05:25:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.296+0000 7f335f834740 1 --2- 172.21.15.23:0/1526644026 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x559641a751a0 0x559641a77590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.296+0000 7f335f834740 4 client.15603 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:25:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.296+0000 7f335f834740 1 -- 172.21.15.23:0/1526644026 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5596419cd050 con 0x559641a751a0 2024-03-28T05:25:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.296+0000 7f335659c640 1 --2- 172.21.15.23:0/1526644026 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x559641a751a0 0x559641a77590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.296+0000 7f333dffb640 10 client.15603 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:25:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.296+0000 7f335659c640 1 --2- 172.21.15.23:0/1526644026 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x559641a751a0 0x559641a77590 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcdf572f6256d0 server_cookie=6c2c3ec168e8dfa3 in_seq=0 out_seq=0 2024-03-28T05:25:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.297+0000 7f333dffb640 1 -- 172.21.15.23:0/1526644026 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5596419cd050 con 0x559641a751a0 2024-03-28T05:25:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.297+0000 7f333dffb640 10 client.15603 handle_command_reply: tid=0 2024-03-28T05:25:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.297+0000 7f335f834740 1 client.15603 shutdown 2024-03-28T05:25:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.297+0000 7f335f834740 1 -- 172.21.15.23:0/1526644026 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5c030 msgr2=0x559641a62310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.297+0000 7f335f834740 1 --2- 172.21.15.23:0/1526644026 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5c030 0x559641a62310 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f334c097dc0 tx=0x7f334c09b7f0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.297+0000 7f335f834740 1 -- 172.21.15.23:0/1526644026 shutdown_connections 2024-03-28T05:25:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.297+0000 7f335f834740 1 --2- 172.21.15.23:0/1526644026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559641a64e00 0x559641a62850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.297+0000 7f335f834740 1 --2- 172.21.15.23:0/1526644026 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5c030 0x559641a62310 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.297+0000 7f335f834740 1 --2- 172.21.15.23:0/1526644026 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x559641a751a0 0x559641a77590 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.297+0000 7f335f834740 1 --2- 172.21.15.23:0/1526644026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559641a5b710 0x559641a61dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.297+0000 7f333dffb640 0 client.15603 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:25:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.297+0000 7f335f834740 1 -- 172.21.15.23:0/1526644026 >> 172.21.15.23:0/1526644026 conn(0x559641a56030 msgr2=0x559641a57d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.297+0000 7f335f834740 1 -- 172.21.15.23:0/1526644026 shutdown_connections 2024-03-28T05:25:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.297+0000 7f335f834740 1 -- 172.21.15.23:0/1526644026 wait complete. 2024-03-28T05:25:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.297+0000 7f335f834740 20 client.15603 trim_cache size 0 max 16384 2024-03-28T05:25:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.311+0000 7f335f834740 1 Processor -- start 2024-03-28T05:25:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.311+0000 7f335f834740 1 -- start start 2024-03-28T05:25:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.311+0000 7f335f834740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559641a5b710 0x559641a64500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.311+0000 7f335f834740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5c030 0x559641a702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.311+0000 7f335f834740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559641a64e00 0x559641a70800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.311+0000 7f335f834740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33581aca00 con 0x559641a5c030 2024-03-28T05:25:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.311+0000 7f335f834740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33581aefc0 con 0x559641a5b710 2024-03-28T05:25:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.311+0000 7f335f834740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33581ad000 con 0x559641a64e00 2024-03-28T05:25:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.311+0000 7f3355d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5c030 0x559641a702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.311+0000 7f3355d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5c030 0x559641a702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32800/0 (socket says 172.21.15.23:32800) 2024-03-28T05:25:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.311+0000 7f3355d9b640 1 -- 172.21.15.23:0/831607385 learned_addr learned my addr 172.21.15.23:0/831607385 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:25:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.312+0000 7f3356d9d640 1 --2- 172.21.15.23:0/831607385 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559641a64e00 0x559641a70800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.312+0000 7f335659c640 1 --2- 172.21.15.23:0/831607385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559641a5b710 0x559641a64500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.312+0000 7f3355d9b640 1 -- 172.21.15.23:0/831607385 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559641a64e00 msgr2=0x559641a70800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.312+0000 7f3355d9b640 1 --2- 172.21.15.23:0/831607385 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559641a64e00 0x559641a70800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.312+0000 7f3355d9b640 1 -- 172.21.15.23:0/831607385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559641a5b710 msgr2=0x559641a64500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.312+0000 7f3355d9b640 1 --2- 172.21.15.23:0/831607385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559641a5b710 0x559641a64500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.312+0000 7f3355d9b640 1 -- 172.21.15.23:0/831607385 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559641a77bf0 con 0x559641a5c030 2024-03-28T05:25:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.312+0000 7f3355d9b640 1 --2- 172.21.15.23:0/831607385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5c030 0x559641a702c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f334c0ccb00 tx=0x7f334c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=994ee2872ebb3a78 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.312+0000 7f333ffff640 1 -- 172.21.15.23:0/831607385 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c09b2c0 con 0x559641a5c030 2024-03-28T05:25:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.312+0000 7f333ffff640 1 -- 172.21.15.23:0/831607385 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f334c09fab0 con 0x559641a5c030 2024-03-28T05:25:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.312+0000 7f333ffff640 1 -- 172.21.15.23:0/831607385 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c0bbbe0 con 0x559641a5c030 2024-03-28T05:25:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.312+0000 7f335f834740 1 -- 172.21.15.23:0/831607385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5c030 msgr2=0x559641a702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f335f834740 1 --2- 172.21.15.23:0/831607385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5c030 0x559641a702c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f334c0ccb00 tx=0x7f334c004a40 comp rx=0 tx=0).stop 2024-03-28T05:25:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f335f834740 1 -- 172.21.15.23:0/831607385 shutdown_connections 2024-03-28T05:25:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f335f834740 1 --2- 172.21.15.23:0/831607385 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559641a64e00 0x559641a70800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f335f834740 1 --2- 172.21.15.23:0/831607385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5c030 0x559641a702c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f335f834740 1 --2- 172.21.15.23:0/831607385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559641a5b710 0x559641a64500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f335f834740 1 -- 172.21.15.23:0/831607385 >> 172.21.15.23:0/831607385 conn(0x559641a56030 msgr2=0x559641a5c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f335f834740 1 -- 172.21.15.23:0/831607385 shutdown_connections 2024-03-28T05:25:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f335f834740 1 -- 172.21.15.23:0/831607385 wait complete. 2024-03-28T05:25:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f335f834740 1 -- start start 2024-03-28T05:25:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f335f834740 1 Processor -- start 2024-03-28T05:25:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f335f834740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559641a5b710 0x559641a68a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f335f834740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5c030 0x559641a68fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f335f834740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559641a64e00 0x559641ac99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f335f834740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33581ad640 con 0x559641a5c030 2024-03-28T05:25:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f335f834740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33581ae960 con 0x559641a5b710 2024-03-28T05:25:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f335f834740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33581adca0 con 0x559641a64e00 2024-03-28T05:25:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f335f834740 10 client.0 _ll_register_callbacks cb 0x7f334c004ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:25:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f3355d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5c030 0x559641a68fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f335659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559641a5b710 0x559641a68a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f3356d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559641a64e00 0x559641ac99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f3355d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5c030 0x559641a68fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32810/0 (socket says 172.21.15.23:32810) 2024-03-28T05:25:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f335659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559641a5b710 0x559641a68a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.23:34172/0 (socket says 172.21.15.23:34172) 2024-03-28T05:25:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.313+0000 7f3355d9b640 1 -- 172.21.15.23:0/1130952555 learned_addr learned my addr 172.21.15.23:0/1130952555 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:25:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.314+0000 7f3355d9b640 1 -- 172.21.15.23:0/1130952555 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559641a64e00 msgr2=0x559641ac99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.314+0000 7f3355d9b640 1 --2- 172.21.15.23:0/1130952555 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559641a64e00 0x559641ac99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.314+0000 7f3355d9b640 1 -- 172.21.15.23:0/1130952555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559641a5b710 msgr2=0x559641a68a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.314+0000 7f3355d9b640 1 --2- 172.21.15.23:0/1130952555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559641a5b710 0x559641a68a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.314+0000 7f3355d9b640 1 -- 172.21.15.23:0/1130952555 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559641a671f0 con 0x559641a5c030 2024-03-28T05:25:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.314+0000 7f3356d9d640 1 --2- 172.21.15.23:0/1130952555 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559641a64e00 0x559641ac99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:25:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.314+0000 7f3355d9b640 1 --2- 172.21.15.23:0/1130952555 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5c030 0x559641a68fb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f334c0a3bd0 tx=0x7f334c0a3c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=575ed1e046cfec9d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.314+0000 7f333dffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:25:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.314+0000 7f333dffb640 1 -- 172.21.15.23:0/1130952555 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c09b2c0 con 0x559641a5c030 2024-03-28T05:25:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.314+0000 7f333dffb640 1 -- 172.21.15.23:0/1130952555 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f334c0bbbe0 con 0x559641a5c030 2024-03-28T05:25:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.314+0000 7f335f834740 1 -- 172.21.15.23:0/1130952555 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f334c09fab0 con 0x559641a5c030 2024-03-28T05:25:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.314+0000 7f333dffb640 1 -- 172.21.15.23:0/1130952555 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f334c0cf4b0 con 0x559641a5c030 2024-03-28T05:25:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.314+0000 7f335f834740 10 client.15609 fetch_fsmap learned FSMap version 20 2024-03-28T05:25:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.314+0000 7f335f834740 1 -- 172.21.15.23:0/1130952555 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x559641a671f0 con 0x559641a5c030 2024-03-28T05:25:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.314+0000 7f333dffb640 1 -- 172.21.15.23:0/1130952555 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f334c09fab0 con 0x559641a5c030 2024-03-28T05:25:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.314+0000 7f333dffb640 1 -- 172.21.15.23:0/1130952555 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f334c0cfb60 con 0x559641a5c030 2024-03-28T05:25:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.314+0000 7f335f834740 10 client.15609 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:25:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.314+0000 7f335f834740 10 client.15609 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:25:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.314+0000 7f335f834740 20 client.15609 populate_metadata read hostname 'smithi023' 2024-03-28T05:25:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.315+0000 7f335f834740 1 --2- 172.21.15.23:0/1130952555 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x559641ad1480 0x559641ad38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.315+0000 7f335f834740 4 client.15609 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:25:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.315+0000 7f335f834740 1 -- 172.21.15.23:0/1130952555 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5596419c73a0 con 0x559641ad1480 2024-03-28T05:25:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.315+0000 7f335659c640 1 --2- 172.21.15.23:0/1130952555 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x559641ad1480 0x559641ad38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.315+0000 7f333dffb640 10 client.15609 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:25:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.315+0000 7f335659c640 1 --2- 172.21.15.23:0/1130952555 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x559641ad1480 0x559641ad38a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a035dc05df3498a server_cookie=9b854c872ab3e0bd in_seq=0 out_seq=0 2024-03-28T05:25:08.592 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.591+0000 7f333dffb640 1 -- 172.21.15.23:0/1130952555 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5596419c73a0 con 0x559641ad1480 2024-03-28T05:25:08.592 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.591+0000 7f333dffb640 10 client.15609 handle_command_reply: tid=0 2024-03-28T05:25:08.592 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.591+0000 7f335f834740 1 client.15609 shutdown 2024-03-28T05:25:08.592 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.591+0000 7f335f834740 1 -- 172.21.15.23:0/1130952555 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5c030 msgr2=0x559641a68fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:08.592 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.591+0000 7f335f834740 1 --2- 172.21.15.23:0/1130952555 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5c030 0x559641a68fb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f334c0a3bd0 tx=0x7f334c0a3c70 comp rx=0 tx=0).stop 2024-03-28T05:25:08.592 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.591+0000 7f335f834740 1 -- 172.21.15.23:0/1130952555 shutdown_connections 2024-03-28T05:25:08.592 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.591+0000 7f335f834740 1 --2- 172.21.15.23:0/1130952555 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559641a64e00 0x559641ac99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.592 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.591+0000 7f335f834740 1 --2- 172.21.15.23:0/1130952555 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559641a5c030 0x559641a68fb0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.592 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.591+0000 7f335f834740 1 --2- 172.21.15.23:0/1130952555 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x559641ad1480 0x559641ad38a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.592 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.591+0000 7f335f834740 1 --2- 172.21.15.23:0/1130952555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559641a5b710 0x559641a68a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.593 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.591+0000 7f333dffb640 0 client.15609 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:25:08.593 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.591+0000 7f335f834740 1 -- 172.21.15.23:0/1130952555 >> 172.21.15.23:0/1130952555 conn(0x559641a56030 msgr2=0x559641a54790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:08.593 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.591+0000 7f335f834740 1 -- 172.21.15.23:0/1130952555 shutdown_connections 2024-03-28T05:25:08.593 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.591+0000 7f335f834740 1 -- 172.21.15.23:0/1130952555 wait complete. 2024-03-28T05:25:08.593 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.591+0000 7f335f834740 20 client.15609 trim_cache size 0 max 16384 2024-03-28T05:25:08.593 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:25:08.593 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:25:08.593 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:25:08.593 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:25:08.594 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.592+0000 7f335cb76640 1 -- 172.21.15.23:0/2660937594 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f33400a2a10 msgr2=0x7f33400a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:08.594 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.592+0000 7f335cb76640 1 --2- 172.21.15.23:0/2660937594 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f33400a2a10 0x7f33400a4ed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3350006010 tx=0x7f335002c0f0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.594 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.592+0000 7f335cb76640 1 -- 172.21.15.23:0/2660937594 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3358101190 msgr2=0x7f3358198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:08.594 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.592+0000 7f335cb76640 1 --2- 172.21.15.23:0/2660937594 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3358101190 0x7f3358198450 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f334c00e9f0 tx=0x7f334c00edf0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.594 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.592+0000 7f335cb76640 1 -- 172.21.15.23:0/2660937594 shutdown_connections 2024-03-28T05:25:08.594 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.592+0000 7f335cb76640 1 --2- 172.21.15.23:0/2660937594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3358105520 0x7f335819c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.595 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.592+0000 7f335cb76640 1 --2- 172.21.15.23:0/2660937594 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3358101190 0x7f3358198450 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.595 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.592+0000 7f335cb76640 1 --2- 172.21.15.23:0/2660937594 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f33400a2a10 0x7f33400a4ed0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.595 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.592+0000 7f335cb76640 1 --2- 172.21.15.23:0/2660937594 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33581007f0 0x7f3358197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:08.595 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.592+0000 7f335cb76640 1 -- 172.21.15.23:0/2660937594 >> 172.21.15.23:0/2660937594 conn(0x7f3358077370 msgr2=0x7f33580fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:08.595 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.592+0000 7f335cb76640 1 -- 172.21.15.23:0/2660937594 shutdown_connections 2024-03-28T05:25:08.595 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:08.592+0000 7f335cb76640 1 -- 172.21.15.23:0/2660937594 wait complete. 2024-03-28T05:25:09.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:09 smithi098 ceph-mon[36087]: pgmap v50: 129 pgs: 129 active+clean; 6.3 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 536 KiB/s wr, 119 op/s 2024-03-28T05:25:09.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:09 smithi113 ceph-mon[37164]: pgmap v50: 129 pgs: 129 active+clean; 6.3 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 536 KiB/s wr, 119 op/s 2024-03-28T05:25:09.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:09 smithi023 ceph-mon[31954]: pgmap v50: 129 pgs: 129 active+clean; 6.3 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 536 KiB/s wr, 119 op/s 2024-03-28T05:25:11.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:11 smithi113 ceph-mon[37164]: pgmap v51: 129 pgs: 129 active+clean; 4.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 845 KiB/s wr, 317 op/s 2024-03-28T05:25:11.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:11 smithi023 ceph-mon[31954]: pgmap v51: 129 pgs: 129 active+clean; 4.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 845 KiB/s wr, 317 op/s 2024-03-28T05:25:11.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:11 smithi098 ceph-mon[36087]: pgmap v51: 129 pgs: 129 active+clean; 4.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 845 KiB/s wr, 317 op/s 2024-03-28T05:25:13.492 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:13 smithi113 ceph-mon[37164]: pgmap v52: 129 pgs: 129 active+clean; 4.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 719 KiB/s wr, 285 op/s 2024-03-28T05:25:13.530 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:13 smithi023 ceph-mon[31954]: pgmap v52: 129 pgs: 129 active+clean; 4.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 719 KiB/s wr, 285 op/s 2024-03-28T05:25:13.615 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:13 smithi098 ceph-mon[36087]: pgmap v52: 129 pgs: 129 active+clean; 4.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 719 KiB/s wr, 285 op/s 2024-03-28T05:25:15.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:15 smithi113 ceph-mon[37164]: pgmap v53: 129 pgs: 129 active+clean; 4.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 719 KiB/s wr, 285 op/s 2024-03-28T05:25:16.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:15 smithi023 ceph-mon[31954]: pgmap v53: 129 pgs: 129 active+clean; 4.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 719 KiB/s wr, 285 op/s 2024-03-28T05:25:16.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:15 smithi098 ceph-mon[36087]: pgmap v53: 129 pgs: 129 active+clean; 4.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 719 KiB/s wr, 285 op/s 2024-03-28T05:25:16.398 INFO:tasks.workunit.client.0.smithi023.stdout: 4 26931 446 15315 199 7976 353 2024-03-28T05:25:17.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:17 smithi113 ceph-mon[37164]: pgmap v54: 129 pgs: 129 active+clean; 6.9 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 939 KiB/s wr, 314 op/s 2024-03-28T05:25:18.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:17 smithi023 ceph-mon[31954]: pgmap v54: 129 pgs: 129 active+clean; 6.9 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 939 KiB/s wr, 314 op/s 2024-03-28T05:25:18.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:17 smithi098 ceph-mon[36087]: pgmap v54: 129 pgs: 129 active+clean; 6.9 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 939 KiB/s wr, 314 op/s 2024-03-28T05:25:19.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:19 smithi113 ceph-mon[37164]: pgmap v55: 129 pgs: 129 active+clean; 6.9 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 528 KiB/s wr, 227 op/s 2024-03-28T05:25:20.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:19 smithi023 ceph-mon[31954]: pgmap v55: 129 pgs: 129 active+clean; 6.9 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 528 KiB/s wr, 227 op/s 2024-03-28T05:25:20.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:19 smithi098 ceph-mon[36087]: pgmap v55: 129 pgs: 129 active+clean; 6.9 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 528 KiB/s wr, 227 op/s 2024-03-28T05:25:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.682+0000 7f8655819640 1 -- 172.21.15.23:0/3707951580 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8650105540 msgr2=0x7f8650105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.682+0000 7f8655819640 1 --2- 172.21.15.23:0/3707951580 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8650105540 0x7f8650105920 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8640009900 tx=0x7f864002bc30 comp rx=0 tx=0).stop 2024-03-28T05:25:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.682+0000 7f8655819640 1 -- 172.21.15.23:0/3707951580 shutdown_connections 2024-03-28T05:25:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.682+0000 7f8655819640 1 --2- 172.21.15.23:0/3707951580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86501011b0 0x7f865010d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.682+0000 7f8655819640 1 --2- 172.21.15.23:0/3707951580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86501007f0 0x7f8650100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.682+0000 7f8655819640 1 --2- 172.21.15.23:0/3707951580 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8650105540 0x7f8650105920 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.682+0000 7f8655819640 1 -- 172.21.15.23:0/3707951580 >> 172.21.15.23:0/3707951580 conn(0x7f86500773f0 msgr2=0x7f8650077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.682+0000 7f8655819640 1 -- 172.21.15.23:0/3707951580 shutdown_connections 2024-03-28T05:25:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.682+0000 7f8655819640 1 -- 172.21.15.23:0/3707951580 wait complete. 2024-03-28T05:25:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.682+0000 7f8655819640 1 Processor -- start 2024-03-28T05:25:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.682+0000 7f8655819640 1 -- start start 2024-03-28T05:25:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.683+0000 7f8655819640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86501007f0 0x7f8650198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.683+0000 7f8655819640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86501011b0 0x7f8650199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.683+0000 7f8655819640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8650105540 0x7f86501a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.683+0000 7f8655819640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8650110a60 con 0x7f8650105540 2024-03-28T05:25:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.683+0000 7f8655819640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86501108e0 con 0x7f86501007f0 2024-03-28T05:25:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.683+0000 7f8655819640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8650110be0 con 0x7f86501011b0 2024-03-28T05:25:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.683+0000 7f864f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8650105540 0x7f86501a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.683+0000 7f864f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8650105540 0x7f86501a3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32880/0 (socket says 172.21.15.23:32880) 2024-03-28T05:25:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.683+0000 7f864e7fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86501011b0 0x7f8650199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.683+0000 7f864f7fe640 1 -- 172.21.15.23:0/4199195311 learned_addr learned my addr 172.21.15.23:0/4199195311 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:25:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.683+0000 7f864effd640 1 --2- 172.21.15.23:0/4199195311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86501007f0 0x7f8650198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.683+0000 7f864f7fe640 1 -- 172.21.15.23:0/4199195311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86501011b0 msgr2=0x7f8650199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.683+0000 7f864f7fe640 1 --2- 172.21.15.23:0/4199195311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86501011b0 0x7f8650199300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.683+0000 7f864f7fe640 1 -- 172.21.15.23:0/4199195311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86501007f0 msgr2=0x7f8650198dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.683+0000 7f864f7fe640 1 --2- 172.21.15.23:0/4199195311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86501007f0 0x7f8650198dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.683+0000 7f864f7fe640 1 -- 172.21.15.23:0/4199195311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86501a4380 con 0x7f8650105540 2024-03-28T05:25:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.683+0000 7f864e7fc640 1 --2- 172.21.15.23:0/4199195311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86501011b0 0x7f8650199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:25:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.683+0000 7f864f7fe640 1 --2- 172.21.15.23:0/4199195311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8650105540 0x7f86501a3e40 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f864800ea60 tx=0x7f864800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.683+0000 7f8654817640 1 -- 172.21.15.23:0/4199195311 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864800ce40 con 0x7f8650105540 2024-03-28T05:25:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.684+0000 7f8655819640 1 -- 172.21.15.23:0/4199195311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86501a4550 con 0x7f8650105540 2024-03-28T05:25:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.684+0000 7f8654817640 1 -- 172.21.15.23:0/4199195311 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8648005020 con 0x7f8650105540 2024-03-28T05:25:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.684+0000 7f8654817640 1 -- 172.21.15.23:0/4199195311 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8648010690 con 0x7f8650105540 2024-03-28T05:25:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.684+0000 7f8655819640 1 -- 172.21.15.23:0/4199195311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86501a4aa0 con 0x7f8650105540 2024-03-28T05:25:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.686+0000 7f8654817640 1 -- 172.21.15.23:0/4199195311 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f8648005520 con 0x7f8650105540 2024-03-28T05:25:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.686+0000 7f8658409740 1 Processor -- start 2024-03-28T05:25:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.686+0000 7f8658409740 1 -- start start 2024-03-28T05:25:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.686+0000 7f8654817640 1 --2- 172.21.15.23:0/4199195311 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f86380a2a10 0x7f86380a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.686+0000 7f8658409740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed219710 0x5584ed219af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.686+0000 7f8658409740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed21a030 0x5584ed2228c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.686+0000 7f8658409740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584ed222e00 0x5584ed2251f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.686+0000 7f8658409740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584ed17db10 con 0x5584ed21a030 2024-03-28T05:25:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.686+0000 7f8658409740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584ed126900 con 0x5584ed222e00 2024-03-28T05:25:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.686+0000 7f8658409740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584ed179bb0 con 0x5584ed219710 2024-03-28T05:25:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.686+0000 7f864e7fc640 1 --2- 172.21.15.23:0/4199195311 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f86380a2a10 0x7f86380a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.687+0000 7f864effd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed219710 0x5584ed219af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.687+0000 7f8654817640 1 -- 172.21.15.23:0/4199195311 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f86480cf660 con 0x7f8650105540 2024-03-28T05:25:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.687+0000 7f864effd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed219710 0x5584ed219af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:48810/0 (socket says 172.21.15.23:48810) 2024-03-28T05:25:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.687+0000 7f864effd640 1 -- 172.21.15.23:0/584010661 learned_addr learned my addr 172.21.15.23:0/584010661 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:25:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.687+0000 7f864f7fe640 1 --2- 172.21.15.23:0/584010661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584ed222e00 0x5584ed2251f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.687+0000 7f864e7fc640 1 --2- 172.21.15.23:0/584010661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed21a030 0x5584ed2228c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.687+0000 7f864effd640 1 -- 172.21.15.23:0/584010661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584ed222e00 msgr2=0x5584ed2251f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.687+0000 7f864effd640 1 --2- 172.21.15.23:0/584010661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584ed222e00 0x5584ed2251f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.687+0000 7f864effd640 1 -- 172.21.15.23:0/584010661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed21a030 msgr2=0x5584ed2228c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.687+0000 7f864effd640 1 --2- 172.21.15.23:0/584010661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed21a030 0x5584ed2228c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.687+0000 7f864effd640 1 -- 172.21.15.23:0/584010661 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584ece92210 con 0x5584ed219710 2024-03-28T05:25:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.687+0000 7f864e7fc640 1 --2- 172.21.15.23:0/4199195311 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f86380a2a10 0x7f86380a4ed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8650069890 tx=0x7f8644013c30 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.687+0000 7f864effd640 1 --2- 172.21.15.23:0/584010661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed219710 0x5584ed219af0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8640009900 tx=0x7f8640005be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a301d0130c18a0b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.687+0000 7f8635ffb640 1 -- 172.21.15.23:0/584010661 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8640042070 con 0x5584ed219710 2024-03-28T05:25:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.687+0000 7f8635ffb640 1 -- 172.21.15.23:0/584010661 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f864002c8b0 con 0x5584ed219710 2024-03-28T05:25:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.687+0000 7f8635ffb640 1 -- 172.21.15.23:0/584010661 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8640004ec0 con 0x5584ed219710 2024-03-28T05:25:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.688+0000 7f8658409740 1 -- 172.21.15.23:0/584010661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed219710 msgr2=0x5584ed219af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.688+0000 7f8658409740 1 --2- 172.21.15.23:0/584010661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed219710 0x5584ed219af0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8640009900 tx=0x7f8640005be0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.688+0000 7f8658409740 1 -- 172.21.15.23:0/584010661 shutdown_connections 2024-03-28T05:25:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.688+0000 7f8658409740 1 --2- 172.21.15.23:0/584010661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584ed222e00 0x5584ed2251f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.688+0000 7f8658409740 1 --2- 172.21.15.23:0/584010661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed21a030 0x5584ed2228c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.688+0000 7f8658409740 1 --2- 172.21.15.23:0/584010661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed219710 0x5584ed219af0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.688+0000 7f8658409740 1 -- 172.21.15.23:0/584010661 >> 172.21.15.23:0/584010661 conn(0x5584ed214030 msgr2=0x5584ed214410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.688+0000 7f8658409740 1 -- 172.21.15.23:0/584010661 shutdown_connections 2024-03-28T05:25:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.688+0000 7f8658409740 1 -- 172.21.15.23:0/584010661 wait complete. 2024-03-28T05:25:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.688+0000 7f8658409740 1 -- start start 2024-03-28T05:25:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.689+0000 7f8658409740 1 Processor -- start 2024-03-28T05:25:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.689+0000 7f8658409740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed219710 0x5584ed21fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.689+0000 7f8658409740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584ed21a030 0x5584ed220400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.689+0000 7f8658409740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed222e00 0x5584ed220940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.689+0000 7f8658409740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584ed2259d0 con 0x5584ed222e00 2024-03-28T05:25:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.689+0000 7f8658409740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584ed139d50 con 0x5584ed21a030 2024-03-28T05:25:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.689+0000 7f8658409740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584ed225b50 con 0x5584ed219710 2024-03-28T05:25:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.689+0000 7f8658409740 10 client.0 _ll_register_callbacks cb 0x5584ed139720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:25:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.689+0000 7f864f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed222e00 0x5584ed220940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.689+0000 7f864e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584ed21a030 0x5584ed220400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.689+0000 7f864effd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed219710 0x5584ed21fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.689+0000 7f864f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed222e00 0x5584ed220940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32884/0 (socket says 172.21.15.23:32884) 2024-03-28T05:25:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.689+0000 7f864f7fe640 1 -- 172.21.15.23:0/341511507 learned_addr learned my addr 172.21.15.23:0/341511507 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:25:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.689+0000 7f864f7fe640 1 -- 172.21.15.23:0/341511507 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed219710 msgr2=0x5584ed21fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.689+0000 7f864f7fe640 1 --2- 172.21.15.23:0/341511507 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed219710 0x5584ed21fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.689+0000 7f864f7fe640 1 -- 172.21.15.23:0/341511507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584ed21a030 msgr2=0x5584ed220400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.689+0000 7f864f7fe640 1 --2- 172.21.15.23:0/341511507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584ed21a030 0x5584ed220400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.689+0000 7f864f7fe640 1 -- 172.21.15.23:0/341511507 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5584ecf5e620 con 0x5584ed222e00 2024-03-28T05:25:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.689+0000 7f864e7fc640 1 --2- 172.21.15.23:0/341511507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584ed21a030 0x5584ed220400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:25:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.689+0000 7f864effd640 1 --2- 172.21.15.23:0/341511507 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed219710 0x5584ed21fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:25:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.689+0000 7f864f7fe640 1 --2- 172.21.15.23:0/341511507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed222e00 0x5584ed220940 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8648097d90 tx=0x7f864809b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf73e3ada5402a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.689+0000 7f862effd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:25:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.690+0000 7f862effd640 1 -- 172.21.15.23:0/341511507 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86480ce040 con 0x5584ed222e00 2024-03-28T05:25:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.690+0000 7f862effd640 1 -- 172.21.15.23:0/341511507 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f86480051d0 con 0x5584ed222e00 2024-03-28T05:25:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.690+0000 7f8658409740 1 -- 172.21.15.23:0/341511507 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f864002c8b0 con 0x5584ed222e00 2024-03-28T05:25:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.690+0000 7f862effd640 1 -- 172.21.15.23:0/341511507 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86480a40d0 con 0x5584ed222e00 2024-03-28T05:25:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.690+0000 7f8658409740 10 client.15621 fetch_fsmap learned FSMap version 20 2024-03-28T05:25:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.690+0000 7f8658409740 1 -- 172.21.15.23:0/341511507 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5584ed231d20 con 0x5584ed222e00 2024-03-28T05:25:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.690+0000 7f862effd640 1 -- 172.21.15.23:0/341511507 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f864002c8b0 con 0x5584ed222e00 2024-03-28T05:25:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.690+0000 7f862effd640 1 -- 172.21.15.23:0/341511507 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f86480a3c40 con 0x5584ed222e00 2024-03-28T05:25:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.690+0000 7f8658409740 10 client.15621 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:25:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.690+0000 7f8658409740 10 client.15621 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:25:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.690+0000 7f8658409740 20 client.15621 populate_metadata read hostname 'smithi023' 2024-03-28T05:25:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.690+0000 7f8658409740 1 --2- 172.21.15.23:0/341511507 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5584ed2331e0 0x5584ed2355d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.690+0000 7f8658409740 4 client.15621 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:25:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.690+0000 7f8658409740 1 -- 172.21.15.23:0/341511507 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5584ed18b050 con 0x5584ed2331e0 2024-03-28T05:25:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.690+0000 7f864effd640 1 --2- 172.21.15.23:0/341511507 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5584ed2331e0 0x5584ed2355d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.691+0000 7f864effd640 1 --2- 172.21.15.23:0/341511507 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5584ed2331e0 0x5584ed2355d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=922000789d802dc3 server_cookie=b04b69cd0cf33e48 in_seq=0 out_seq=0 2024-03-28T05:25:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.691+0000 7f862effd640 10 client.15621 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:25:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.692+0000 7f862effd640 1 -- 172.21.15.23:0/341511507 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5584ed18b050 con 0x5584ed2331e0 2024-03-28T05:25:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.692+0000 7f862effd640 10 client.15621 handle_command_reply: tid=0 2024-03-28T05:25:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.692+0000 7f8658409740 1 client.15621 shutdown 2024-03-28T05:25:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.692+0000 7f8658409740 1 -- 172.21.15.23:0/341511507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed222e00 msgr2=0x5584ed220940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.692+0000 7f8658409740 1 --2- 172.21.15.23:0/341511507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed222e00 0x5584ed220940 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8648097d90 tx=0x7f864809b930 comp rx=0 tx=0).stop 2024-03-28T05:25:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.692+0000 7f8658409740 1 -- 172.21.15.23:0/341511507 shutdown_connections 2024-03-28T05:25:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.692+0000 7f8658409740 1 --2- 172.21.15.23:0/341511507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed222e00 0x5584ed220940 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.692+0000 7f8658409740 1 --2- 172.21.15.23:0/341511507 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5584ed2331e0 0x5584ed2355d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.692+0000 7f8658409740 1 --2- 172.21.15.23:0/341511507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584ed21a030 0x5584ed220400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.692+0000 7f8658409740 1 --2- 172.21.15.23:0/341511507 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed219710 0x5584ed21fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.692+0000 7f8658409740 1 -- 172.21.15.23:0/341511507 >> 172.21.15.23:0/341511507 conn(0x5584ed214030 msgr2=0x5584ed215d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.692+0000 7f862effd640 0 client.15621 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:25:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.692+0000 7f8658409740 1 -- 172.21.15.23:0/341511507 shutdown_connections 2024-03-28T05:25:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.692+0000 7f8658409740 1 -- 172.21.15.23:0/341511507 wait complete. 2024-03-28T05:25:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.692+0000 7f8658409740 20 client.15621 trim_cache size 0 max 16384 2024-03-28T05:25:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.708+0000 7f8658409740 1 Processor -- start 2024-03-28T05:25:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.708+0000 7f8658409740 1 -- start start 2024-03-28T05:25:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.708+0000 7f8658409740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584ed219710 0x5584ed2225f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.708+0000 7f8658409740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed21a030 0x5584ed212ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.708+0000 7f8658409740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed222e00 0x5584ed22e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.708+0000 7f8658409740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86501ac9b0 con 0x5584ed222e00 2024-03-28T05:25:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.708+0000 7f8658409740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86501aef70 con 0x5584ed219710 2024-03-28T05:25:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.708+0000 7f8658409740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86501acfb0 con 0x5584ed21a030 2024-03-28T05:25:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.708+0000 7f864f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed222e00 0x5584ed22e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.708+0000 7f864f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed222e00 0x5584ed22e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32888/0 (socket says 172.21.15.23:32888) 2024-03-28T05:25:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.708+0000 7f864f7fe640 1 -- 172.21.15.23:0/1506614045 learned_addr learned my addr 172.21.15.23:0/1506614045 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:25:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.708+0000 7f864e7fc640 1 --2- 172.21.15.23:0/1506614045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed21a030 0x5584ed212ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.708+0000 7f864effd640 1 --2- 172.21.15.23:0/1506614045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584ed219710 0x5584ed2225f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.708+0000 7f864f7fe640 1 -- 172.21.15.23:0/1506614045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed21a030 msgr2=0x5584ed212ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.708+0000 7f864f7fe640 1 --2- 172.21.15.23:0/1506614045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed21a030 0x5584ed212ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.708+0000 7f864f7fe640 1 -- 172.21.15.23:0/1506614045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584ed219710 msgr2=0x5584ed2225f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.708+0000 7f864f7fe640 1 --2- 172.21.15.23:0/1506614045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584ed219710 0x5584ed2225f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.708+0000 7f864f7fe640 1 -- 172.21.15.23:0/1506614045 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584ed213b60 con 0x5584ed222e00 2024-03-28T05:25:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.709+0000 7f864f7fe640 1 --2- 172.21.15.23:0/1506614045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed222e00 0x5584ed22e260 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f864800c760 tx=0x7f8648005920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eda23430b1a1d50 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.709+0000 7f86357fa640 1 -- 172.21.15.23:0/1506614045 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86480051d0 con 0x5584ed222e00 2024-03-28T05:25:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.709+0000 7f86357fa640 1 -- 172.21.15.23:0/1506614045 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f864807f070 con 0x5584ed222e00 2024-03-28T05:25:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.709+0000 7f86357fa640 1 -- 172.21.15.23:0/1506614045 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864809aec0 con 0x5584ed222e00 2024-03-28T05:25:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.709+0000 7f8658409740 1 -- 172.21.15.23:0/1506614045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed222e00 msgr2=0x5584ed22e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.709+0000 7f8658409740 1 --2- 172.21.15.23:0/1506614045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed222e00 0x5584ed22e260 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f864800c760 tx=0x7f8648005920 comp rx=0 tx=0).stop 2024-03-28T05:25:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.709+0000 7f8658409740 1 -- 172.21.15.23:0/1506614045 shutdown_connections 2024-03-28T05:25:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.709+0000 7f8658409740 1 --2- 172.21.15.23:0/1506614045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed222e00 0x5584ed22e260 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.709+0000 7f8658409740 1 --2- 172.21.15.23:0/1506614045 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed21a030 0x5584ed212ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.709+0000 7f8658409740 1 --2- 172.21.15.23:0/1506614045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584ed219710 0x5584ed2225f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.710+0000 7f8658409740 1 -- 172.21.15.23:0/1506614045 >> 172.21.15.23:0/1506614045 conn(0x5584ed214030 msgr2=0x5584ed212790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.710+0000 7f8658409740 1 -- 172.21.15.23:0/1506614045 shutdown_connections 2024-03-28T05:25:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.710+0000 7f8658409740 1 -- 172.21.15.23:0/1506614045 wait complete. 2024-03-28T05:25:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.710+0000 7f8658409740 1 -- start start 2024-03-28T05:25:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.710+0000 7f8658409740 1 Processor -- start 2024-03-28T05:25:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.710+0000 7f8658409740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed219710 0x5584ed22a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.710+0000 7f8658409740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed21a030 0x5584ed225d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.710+0000 7f8658409740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584ed222e00 0x5584ed2262a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.710+0000 7f8658409740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86501ad5f0 con 0x5584ed219710 2024-03-28T05:25:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.710+0000 7f8658409740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86501ae910 con 0x5584ed222e00 2024-03-28T05:25:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.710+0000 7f8658409740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86501adc50 con 0x5584ed21a030 2024-03-28T05:25:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.710+0000 7f8658409740 10 client.0 _ll_register_callbacks cb 0x7f864809c030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:25:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.710+0000 7f864effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed219710 0x5584ed22a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.710+0000 7f864effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed219710 0x5584ed22a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32896/0 (socket says 172.21.15.23:32896) 2024-03-28T05:25:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.710+0000 7f864effd640 1 -- 172.21.15.23:0/3498189284 learned_addr learned my addr 172.21.15.23:0/3498189284 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:25:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.710+0000 7f864e7fc640 1 --2- 172.21.15.23:0/3498189284 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed21a030 0x5584ed225d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.710+0000 7f864f7fe640 1 --2- 172.21.15.23:0/3498189284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584ed222e00 0x5584ed2262a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.711+0000 7f864effd640 1 -- 172.21.15.23:0/3498189284 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed21a030 msgr2=0x5584ed225d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.711+0000 7f864effd640 1 --2- 172.21.15.23:0/3498189284 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed21a030 0x5584ed225d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.711+0000 7f864effd640 1 -- 172.21.15.23:0/3498189284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584ed222e00 msgr2=0x5584ed2262a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.711+0000 7f864effd640 1 --2- 172.21.15.23:0/3498189284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584ed222e00 0x5584ed2262a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.711+0000 7f864effd640 1 -- 172.21.15.23:0/3498189284 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5584ed2267e0 con 0x5584ed219710 2024-03-28T05:25:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.711+0000 7f864f7fe640 1 --2- 172.21.15.23:0/3498189284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584ed222e00 0x5584ed2262a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:25:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.711+0000 7f864effd640 1 --2- 172.21.15.23:0/3498189284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed219710 0x5584ed22a850 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f86400048b0 tx=0x7f86400048e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f6e037a61961965 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.711+0000 7f862effd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:25:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.711+0000 7f862effd640 1 -- 172.21.15.23:0/3498189284 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8640054060 con 0x5584ed219710 2024-03-28T05:25:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.711+0000 7f862effd640 1 -- 172.21.15.23:0/3498189284 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f8640004230 con 0x5584ed219710 2024-03-28T05:25:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.711+0000 7f8658409740 1 -- 172.21.15.23:0/3498189284 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f864807f070 con 0x5584ed219710 2024-03-28T05:25:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.711+0000 7f862effd640 1 -- 172.21.15.23:0/3498189284 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f864005b050 con 0x5584ed219710 2024-03-28T05:25:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.711+0000 7f8658409740 10 client.15627 fetch_fsmap learned FSMap version 20 2024-03-28T05:25:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.711+0000 7f8658409740 1 -- 172.21.15.23:0/3498189284 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5584ed28e7a0 con 0x5584ed219710 2024-03-28T05:25:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.711+0000 7f862effd640 1 -- 172.21.15.23:0/3498189284 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f864807f070 con 0x5584ed219710 2024-03-28T05:25:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.712+0000 7f862effd640 1 -- 172.21.15.23:0/3498189284 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f86400035f0 con 0x5584ed219710 2024-03-28T05:25:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.712+0000 7f8658409740 10 client.15627 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:25:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.712+0000 7f8658409740 10 client.15627 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:25:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.712+0000 7f8658409740 20 client.15627 populate_metadata read hostname 'smithi023' 2024-03-28T05:25:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.712+0000 7f8658409740 1 --2- 172.21.15.23:0/3498189284 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5584ed28eb30 0x5584ed290f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.712+0000 7f8658409740 4 client.15627 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:25:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.712+0000 7f8658409740 1 -- 172.21.15.23:0/3498189284 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5584ed1853a0 con 0x5584ed28eb30 2024-03-28T05:25:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.712+0000 7f864e7fc640 1 --2- 172.21.15.23:0/3498189284 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5584ed28eb30 0x5584ed290f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.712+0000 7f864e7fc640 1 --2- 172.21.15.23:0/3498189284 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5584ed28eb30 0x5584ed290f20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ae5d7a683cc3f86 server_cookie=e4a1a7614f64c1b1 in_seq=0 out_seq=0 2024-03-28T05:25:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:21.712+0000 7f862effd640 10 client.15627 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:25:21.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:21 smithi113 ceph-mon[37164]: pgmap v56: 129 pgs: 129 active+clean; 27 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 2.3 MiB/s wr, 375 op/s 2024-03-28T05:25:22.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:21 smithi023 ceph-mon[31954]: pgmap v56: 129 pgs: 129 active+clean; 27 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 2.3 MiB/s wr, 375 op/s 2024-03-28T05:25:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.112+0000 7f862effd640 1 -- 172.21.15.23:0/3498189284 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5584ed1853a0 con 0x5584ed28eb30 2024-03-28T05:25:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.112+0000 7f862effd640 10 client.15627 handle_command_reply: tid=0 2024-03-28T05:25:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.112+0000 7f8658409740 1 client.15627 shutdown 2024-03-28T05:25:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.113+0000 7f8658409740 1 -- 172.21.15.23:0/3498189284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed219710 msgr2=0x5584ed22a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.113+0000 7f8658409740 1 --2- 172.21.15.23:0/3498189284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed219710 0x5584ed22a850 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f86400048b0 tx=0x7f86400048e0 comp rx=0 tx=0).stop 2024-03-28T05:25:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.113+0000 7f8658409740 1 -- 172.21.15.23:0/3498189284 shutdown_connections 2024-03-28T05:25:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.113+0000 7f8658409740 1 --2- 172.21.15.23:0/3498189284 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5584ed28eb30 0x5584ed290f20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.113+0000 7f8658409740 1 --2- 172.21.15.23:0/3498189284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5584ed222e00 0x5584ed2262a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.113+0000 7f8658409740 1 --2- 172.21.15.23:0/3498189284 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5584ed21a030 0x5584ed225d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.113+0000 7f8658409740 1 --2- 172.21.15.23:0/3498189284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5584ed219710 0x5584ed22a850 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.113+0000 7f862effd640 0 client.15627 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:25:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.113+0000 7f8658409740 1 -- 172.21.15.23:0/3498189284 >> 172.21.15.23:0/3498189284 conn(0x5584ed214030 msgr2=0x5584ed212790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.113+0000 7f8658409740 1 -- 172.21.15.23:0/3498189284 shutdown_connections 2024-03-28T05:25:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.113+0000 7f8658409740 1 -- 172.21.15.23:0/3498189284 wait complete. 2024-03-28T05:25:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.113+0000 7f8658409740 20 client.15627 trim_cache size 0 max 16384 2024-03-28T05:25:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:25:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:25:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:25:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:25:22.115 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.113+0000 7f8655819640 1 -- 172.21.15.23:0/4199195311 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f86380a2a10 msgr2=0x7f86380a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:22.115 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.113+0000 7f8655819640 1 --2- 172.21.15.23:0/4199195311 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f86380a2a10 0x7f86380a4ed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8650069890 tx=0x7f8644013c30 comp rx=0 tx=0).stop 2024-03-28T05:25:22.115 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.113+0000 7f8655819640 1 -- 172.21.15.23:0/4199195311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8650105540 msgr2=0x7f86501a3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:22.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.113+0000 7f8655819640 1 --2- 172.21.15.23:0/4199195311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8650105540 0x7f86501a3e40 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f864800ea60 tx=0x7f864800ef20 comp rx=0 tx=0).stop 2024-03-28T05:25:22.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.113+0000 7f8655819640 1 -- 172.21.15.23:0/4199195311 shutdown_connections 2024-03-28T05:25:22.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.113+0000 7f8655819640 1 --2- 172.21.15.23:0/4199195311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8650105540 0x7f86501a3e40 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:22.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.113+0000 7f8655819640 1 --2- 172.21.15.23:0/4199195311 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f86501011b0 0x7f8650199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:22.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.113+0000 7f8655819640 1 --2- 172.21.15.23:0/4199195311 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f86380a2a10 0x7f86380a4ed0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:22.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.113+0000 7f8655819640 1 --2- 172.21.15.23:0/4199195311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86501007f0 0x7f8650198dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:22.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.113+0000 7f8655819640 1 -- 172.21.15.23:0/4199195311 >> 172.21.15.23:0/4199195311 conn(0x7f86500773f0 msgr2=0x7f86500fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:22.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.114+0000 7f8655819640 1 -- 172.21.15.23:0/4199195311 shutdown_connections 2024-03-28T05:25:22.116 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:22.114+0000 7f8655819640 1 -- 172.21.15.23:0/4199195311 wait complete. 2024-03-28T05:25:22.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:21 smithi098 ceph-mon[36087]: pgmap v56: 129 pgs: 129 active+clean; 27 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 2.3 MiB/s wr, 375 op/s 2024-03-28T05:25:23.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:23 smithi113 ceph-mon[37164]: pgmap v57: 129 pgs: 129 active+clean; 27 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 2.0 MiB/s wr, 177 op/s 2024-03-28T05:25:24.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:23 smithi023 ceph-mon[31954]: pgmap v57: 129 pgs: 129 active+clean; 27 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 2.0 MiB/s wr, 177 op/s 2024-03-28T05:25:24.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:23 smithi098 ceph-mon[36087]: pgmap v57: 129 pgs: 129 active+clean; 27 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 2.0 MiB/s wr, 177 op/s 2024-03-28T05:25:25.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:25 smithi113 ceph-mon[37164]: pgmap v58: 129 pgs: 129 active+clean; 27 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 2.0 MiB/s wr, 177 op/s 2024-03-28T05:25:26.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:25 smithi023 ceph-mon[31954]: pgmap v58: 129 pgs: 129 active+clean; 27 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 2.0 MiB/s wr, 177 op/s 2024-03-28T05:25:26.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:25 smithi098 ceph-mon[36087]: pgmap v58: 129 pgs: 129 active+clean; 27 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 2.0 MiB/s wr, 177 op/s 2024-03-28T05:25:26.398 INFO:tasks.workunit.client.0.smithi023.stdout: 7 60596 128 40749 238 21814 357 2024-03-28T05:25:27.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:27 smithi113 ceph-mon[37164]: pgmap v59: 129 pgs: 129 active+clean; 21 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 476 op/s 2024-03-28T05:25:28.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:27 smithi023 ceph-mon[31954]: pgmap v59: 129 pgs: 129 active+clean; 21 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 476 op/s 2024-03-28T05:25:28.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:27 smithi098 ceph-mon[36087]: pgmap v59: 129 pgs: 129 active+clean; 21 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 476 op/s 2024-03-28T05:25:29.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:29 smithi113 ceph-mon[37164]: pgmap v60: 129 pgs: 129 active+clean; 21 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 446 op/s 2024-03-28T05:25:30.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:29 smithi023 ceph-mon[31954]: pgmap v60: 129 pgs: 129 active+clean; 21 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 446 op/s 2024-03-28T05:25:30.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:29 smithi098 ceph-mon[36087]: pgmap v60: 129 pgs: 129 active+clean; 21 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 446 op/s 2024-03-28T05:25:31.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:31 smithi113 ceph-mon[37164]: pgmap v61: 129 pgs: 129 active+clean; 23 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 474 op/s 2024-03-28T05:25:32.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:31 smithi023 ceph-mon[31954]: pgmap v61: 129 pgs: 129 active+clean; 23 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 474 op/s 2024-03-28T05:25:32.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:31 smithi098 ceph-mon[36087]: pgmap v61: 129 pgs: 129 active+clean; 23 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 474 op/s 2024-03-28T05:25:33.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:33 smithi113 ceph-mon[37164]: pgmap v62: 129 pgs: 129 active+clean; 23 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 500 KiB/s wr, 327 op/s 2024-03-28T05:25:34.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:33 smithi023 ceph-mon[31954]: pgmap v62: 129 pgs: 129 active+clean; 23 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 500 KiB/s wr, 327 op/s 2024-03-28T05:25:34.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:33 smithi098 ceph-mon[36087]: pgmap v62: 129 pgs: 129 active+clean; 23 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 500 KiB/s wr, 327 op/s 2024-03-28T05:25:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.205+0000 7efbfd93b640 1 -- 172.21.15.23:0/2308861378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbf81026e0 msgr2=0x7efbf8102ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.205+0000 7efbfd93b640 1 --2- 172.21.15.23:0/2308861378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbf81026e0 0x7efbf8102ac0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7efbe8009900 tx=0x7efbe802bcd0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.205+0000 7efbfd93b640 1 -- 172.21.15.23:0/2308861378 shutdown_connections 2024-03-28T05:25:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.205+0000 7efbfd93b640 1 --2- 172.21.15.23:0/2308861378 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efbf810bab0 0x7efbf810df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.205+0000 7efbfd93b640 1 --2- 172.21.15.23:0/2308861378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efbf8103000 0x7efbf810b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.205+0000 7efbfd93b640 1 --2- 172.21.15.23:0/2308861378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbf81026e0 0x7efbf8102ac0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.205+0000 7efbfd93b640 1 -- 172.21.15.23:0/2308861378 >> 172.21.15.23:0/2308861378 conn(0x7efbf80fbee0 msgr2=0x7efbf80fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.205+0000 7efbfd93b640 1 -- 172.21.15.23:0/2308861378 shutdown_connections 2024-03-28T05:25:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.205+0000 7efbfd93b640 1 -- 172.21.15.23:0/2308861378 wait complete. 2024-03-28T05:25:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.206+0000 7efbfd93b640 1 Processor -- start 2024-03-28T05:25:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.206+0000 7efbfd93b640 1 -- start start 2024-03-28T05:25:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.206+0000 7efbfd93b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efbf81026e0 0x7efbf8197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.206+0000 7efbfd93b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbf8103000 0x7efbf8198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.206+0000 7efbfd93b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efbf810bab0 0x7efbf819c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.206+0000 7efbfd93b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf8111880 con 0x7efbf8103000 2024-03-28T05:25:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.206+0000 7efbfd93b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf8111700 con 0x7efbf81026e0 2024-03-28T05:25:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.206+0000 7efbfd93b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf8111a00 con 0x7efbf810bab0 2024-03-28T05:25:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.206+0000 7efbee7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbf8103000 0x7efbf8198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.206+0000 7efbf77fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efbf810bab0 0x7efbf819c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.206+0000 7efbee7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbf8103000 0x7efbf8198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48462/0 (socket says 172.21.15.23:48462) 2024-03-28T05:25:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.206+0000 7efbf77fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efbf810bab0 0x7efbf819c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:43316/0 (socket says 172.21.15.23:43316) 2024-03-28T05:25:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.206+0000 7efbee7fc640 1 -- 172.21.15.23:0/1845074379 learned_addr learned my addr 172.21.15.23:0/1845074379 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:25:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.207+0000 7efbf6ffd640 1 --2- 172.21.15.23:0/1845074379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efbf81026e0 0x7efbf8197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.207+0000 7efbee7fc640 1 -- 172.21.15.23:0/1845074379 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efbf810bab0 msgr2=0x7efbf819c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:35.209 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.207+0000 7efbee7fc640 1 --2- 172.21.15.23:0/1845074379 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efbf810bab0 0x7efbf819c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.207+0000 7efbee7fc640 1 -- 172.21.15.23:0/1845074379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efbf81026e0 msgr2=0x7efbf8197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.207+0000 7efbee7fc640 1 --2- 172.21.15.23:0/1845074379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efbf81026e0 0x7efbf8197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.207+0000 7efbee7fc640 1 -- 172.21.15.23:0/1845074379 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf819ce40 con 0x7efbf8103000 2024-03-28T05:25:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.207+0000 7efbee7fc640 1 --2- 172.21.15.23:0/1845074379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbf8103000 0x7efbf8198390 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7efbe000c890 tx=0x7efbe000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.207+0000 7efbfc939640 1 -- 172.21.15.23:0/1845074379 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe0004220 con 0x7efbf8103000 2024-03-28T05:25:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.207+0000 7efbfd93b640 1 -- 172.21.15.23:0/1845074379 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf819d130 con 0x7efbf8103000 2024-03-28T05:25:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.207+0000 7efbfc939640 1 -- 172.21.15.23:0/1845074379 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efbe00043c0 con 0x7efbf8103000 2024-03-28T05:25:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.207+0000 7efbfc939640 1 -- 172.21.15.23:0/1845074379 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe0004a00 con 0x7efbf8103000 2024-03-28T05:25:35.210 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.207+0000 7efbfd93b640 1 -- 172.21.15.23:0/1845074379 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf81a49c0 con 0x7efbf8103000 2024-03-28T05:25:35.217 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.209+0000 7efbfc939640 1 -- 172.21.15.23:0/1845074379 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7efbe00079b0 con 0x7efbf8103000 2024-03-28T05:25:35.217 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.210+0000 7efc00600740 1 Processor -- start 2024-03-28T05:25:35.217 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.210+0000 7efc00600740 1 -- start start 2024-03-28T05:25:35.217 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.210+0000 7efbfc939640 1 --2- 172.21.15.23:0/1845074379 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7efbe40a2960 0x7efbe40a4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:35.217 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.210+0000 7efc00600740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a15e91d910 0x55a15e91dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:35.217 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.210+0000 7efc00600740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a15e91e230 0x55a15e926ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:35.217 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.210+0000 7efc00600740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e927000 0x55a15e9293f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.210+0000 7efc00600740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a15e881b10 con 0x55a15e927000 2024-03-28T05:25:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.210+0000 7efc00600740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a15e82a900 con 0x55a15e91e230 2024-03-28T05:25:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.210+0000 7efc00600740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a15e87dbb0 con 0x55a15e91d910 2024-03-28T05:25:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.210+0000 7efbfc939640 1 -- 172.21.15.23:0/1845074379 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7efbe00cee10 con 0x7efbf8103000 2024-03-28T05:25:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.210+0000 7efbf77fe640 1 --2- 172.21.15.23:0/1845074379 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7efbe40a2960 0x7efbe40a4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.210+0000 7efbf6ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a15e91d910 0x55a15e91dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.210+0000 7efbf77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e927000 0x55a15e9293f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.210+0000 7efbf6ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a15e91d910 0x55a15e91dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:43330/0 (socket says 172.21.15.23:43330) 2024-03-28T05:25:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.210+0000 7efbf6ffd640 1 -- 172.21.15.23:0/1315870252 learned_addr learned my addr 172.21.15.23:0/1315870252 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:25:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.210+0000 7efbee7fc640 1 --2- 172.21.15.23:0/1315870252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a15e91e230 0x55a15e926ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.210+0000 7efbf77fe640 1 -- 172.21.15.23:0/1315870252 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a15e91d910 msgr2=0x55a15e91dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.210+0000 7efbf77fe640 1 --2- 172.21.15.23:0/1315870252 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a15e91d910 0x55a15e91dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.211+0000 7efbf77fe640 1 -- 172.21.15.23:0/1315870252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a15e91e230 msgr2=0x55a15e926ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:35.218 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.211+0000 7efbf77fe640 1 --2- 172.21.15.23:0/1315870252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a15e91e230 0x55a15e926ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.211+0000 7efbf77fe640 1 -- 172.21.15.23:0/1315870252 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a15e596210 con 0x55a15e927000 2024-03-28T05:25:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.211+0000 7efbf77fe640 1 --2- 172.21.15.23:0/1845074379 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7efbe40a2960 0x7efbe40a4e20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efbf0005fd0 tx=0x7efbf0009e90 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.211+0000 7efbf77fe640 1 --2- 172.21.15.23:0/1315870252 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e927000 0x55a15e9293f0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7efbf0009fd0 tx=0x7efbf001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4093a22936a923b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.211+0000 7efbedffb640 1 -- 172.21.15.23:0/1315870252 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf0024070 con 0x55a15e927000 2024-03-28T05:25:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.211+0000 7efbedffb640 1 -- 172.21.15.23:0/1315870252 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efbf0003e50 con 0x55a15e927000 2024-03-28T05:25:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.211+0000 7efbedffb640 1 -- 172.21.15.23:0/1315870252 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf0004300 con 0x55a15e927000 2024-03-28T05:25:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.211+0000 7efc00600740 1 -- 172.21.15.23:0/1315870252 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e927000 msgr2=0x55a15e9293f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.211+0000 7efc00600740 1 --2- 172.21.15.23:0/1315870252 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e927000 0x55a15e9293f0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7efbf0009fd0 tx=0x7efbf001d000 comp rx=0 tx=0).stop 2024-03-28T05:25:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.212+0000 7efc00600740 1 -- 172.21.15.23:0/1315870252 shutdown_connections 2024-03-28T05:25:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.212+0000 7efc00600740 1 --2- 172.21.15.23:0/1315870252 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e927000 0x55a15e9293f0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.212+0000 7efc00600740 1 --2- 172.21.15.23:0/1315870252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a15e91e230 0x55a15e926ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.212+0000 7efc00600740 1 --2- 172.21.15.23:0/1315870252 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a15e91d910 0x55a15e91dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.212+0000 7efc00600740 1 -- 172.21.15.23:0/1315870252 >> 172.21.15.23:0/1315870252 conn(0x55a15e918230 msgr2=0x55a15e918610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.212+0000 7efc00600740 1 -- 172.21.15.23:0/1315870252 shutdown_connections 2024-03-28T05:25:35.219 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.212+0000 7efc00600740 1 -- 172.21.15.23:0/1315870252 wait complete. 2024-03-28T05:25:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.212+0000 7efc00600740 1 -- start start 2024-03-28T05:25:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.212+0000 7efc00600740 1 Processor -- start 2024-03-28T05:25:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.212+0000 7efc00600740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e91d910 0x55a15e923fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.212+0000 7efc00600740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a15e91e230 0x55a15e924520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.212+0000 7efc00600740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a15e927000 0x55a15e924a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.212+0000 7efc00600740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a15e929bd0 con 0x55a15e91d910 2024-03-28T05:25:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.212+0000 7efc00600740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a15e83dd50 con 0x55a15e927000 2024-03-28T05:25:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.212+0000 7efc00600740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a15e929d50 con 0x55a15e91e230 2024-03-28T05:25:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.212+0000 7efc00600740 10 client.0 _ll_register_callbacks cb 0x55a15e83d720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:25:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.212+0000 7efbf6ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e91d910 0x55a15e923fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.213+0000 7efbf6ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e91d910 0x55a15e923fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48486/0 (socket says 172.21.15.23:48486) 2024-03-28T05:25:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.213+0000 7efbf6ffd640 1 -- 172.21.15.23:0/323937294 learned_addr learned my addr 172.21.15.23:0/323937294 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:25:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.213+0000 7efbee7fc640 1 --2- 172.21.15.23:0/323937294 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a15e91e230 0x55a15e924520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:35.220 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.213+0000 7efbf77fe640 1 --2- 172.21.15.23:0/323937294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a15e927000 0x55a15e924a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.213+0000 7efbf6ffd640 1 -- 172.21.15.23:0/323937294 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a15e91e230 msgr2=0x55a15e924520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.213+0000 7efbf6ffd640 1 --2- 172.21.15.23:0/323937294 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a15e91e230 0x55a15e924520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.213+0000 7efbf6ffd640 1 -- 172.21.15.23:0/323937294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a15e927000 msgr2=0x55a15e924a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.213+0000 7efbf6ffd640 1 --2- 172.21.15.23:0/323937294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a15e927000 0x55a15e924a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.213+0000 7efbf6ffd640 1 -- 172.21.15.23:0/323937294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a15e662620 con 0x55a15e91d910 2024-03-28T05:25:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.213+0000 7efbf77fe640 1 --2- 172.21.15.23:0/323937294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a15e927000 0x55a15e924a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:25:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.213+0000 7efbf6ffd640 1 --2- 172.21.15.23:0/323937294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e91d910 0x55a15e923fe0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7efbe8005ab0 tx=0x7efbe8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2318da6d5cf591ce server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.213+0000 7efbd6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:25:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.213+0000 7efbd6ffd640 1 -- 172.21.15.23:0/323937294 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe80046c0 con 0x55a15e91d910 2024-03-28T05:25:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.213+0000 7efbd6ffd640 1 -- 172.21.15.23:0/323937294 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7efbe8004860 con 0x55a15e91d910 2024-03-28T05:25:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.213+0000 7efc00600740 1 -- 172.21.15.23:0/323937294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efbf0003e50 con 0x55a15e91d910 2024-03-28T05:25:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.213+0000 7efbd6ffd640 1 -- 172.21.15.23:0/323937294 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7efbe8004a60 con 0x55a15e91d910 2024-03-28T05:25:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.213+0000 7efc00600740 10 client.15639 fetch_fsmap learned FSMap version 20 2024-03-28T05:25:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.213+0000 7efc00600740 1 -- 172.21.15.23:0/323937294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55a15e935e20 con 0x55a15e91d910 2024-03-28T05:25:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.214+0000 7efbd6ffd640 1 -- 172.21.15.23:0/323937294 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efbf0003e50 con 0x55a15e91d910 2024-03-28T05:25:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.214+0000 7efbd6ffd640 1 -- 172.21.15.23:0/323937294 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7efbe8046070 con 0x55a15e91d910 2024-03-28T05:25:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.214+0000 7efc00600740 10 client.15639 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:25:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.214+0000 7efc00600740 10 client.15639 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:25:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.214+0000 7efc00600740 20 client.15639 populate_metadata read hostname 'smithi023' 2024-03-28T05:25:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.214+0000 7efc00600740 1 --2- 172.21.15.23:0/323937294 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a15e937290 0x55a15e939680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.214+0000 7efc00600740 4 client.15639 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:25:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.214+0000 7efc00600740 1 -- 172.21.15.23:0/323937294 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a15e88f050 con 0x55a15e937290 2024-03-28T05:25:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.214+0000 7efbee7fc640 1 --2- 172.21.15.23:0/323937294 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a15e937290 0x55a15e939680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.214+0000 7efbd6ffd640 10 client.15639 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:25:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.214+0000 7efbee7fc640 1 --2- 172.21.15.23:0/323937294 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a15e937290 0x55a15e939680 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=febd7f949b13f782 server_cookie=70869547654c78d6 in_seq=0 out_seq=0 2024-03-28T05:25:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.216+0000 7efbd6ffd640 1 -- 172.21.15.23:0/323937294 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a15e88f050 con 0x55a15e937290 2024-03-28T05:25:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.216+0000 7efbd6ffd640 10 client.15639 handle_command_reply: tid=0 2024-03-28T05:25:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.216+0000 7efc00600740 1 client.15639 shutdown 2024-03-28T05:25:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.216+0000 7efc00600740 1 -- 172.21.15.23:0/323937294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e91d910 msgr2=0x55a15e923fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.216+0000 7efc00600740 1 --2- 172.21.15.23:0/323937294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e91d910 0x55a15e923fe0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7efbe8005ab0 tx=0x7efbe8004040 comp rx=0 tx=0).stop 2024-03-28T05:25:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.216+0000 7efc00600740 1 -- 172.21.15.23:0/323937294 shutdown_connections 2024-03-28T05:25:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.216+0000 7efc00600740 1 --2- 172.21.15.23:0/323937294 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a15e937290 0x55a15e939680 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.216+0000 7efc00600740 1 --2- 172.21.15.23:0/323937294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a15e927000 0x55a15e924a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.216+0000 7efc00600740 1 --2- 172.21.15.23:0/323937294 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a15e91e230 0x55a15e924520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.216+0000 7efbd6ffd640 0 client.15639 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:25:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.216+0000 7efc00600740 1 --2- 172.21.15.23:0/323937294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e91d910 0x55a15e923fe0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.216+0000 7efc00600740 1 -- 172.21.15.23:0/323937294 >> 172.21.15.23:0/323937294 conn(0x55a15e918230 msgr2=0x55a15e919f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.216+0000 7efc00600740 1 -- 172.21.15.23:0/323937294 shutdown_connections 2024-03-28T05:25:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.216+0000 7efc00600740 1 -- 172.21.15.23:0/323937294 wait complete. 2024-03-28T05:25:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.216+0000 7efc00600740 20 client.15639 trim_cache size 0 max 16384 2024-03-28T05:25:35.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.230+0000 7efc00600740 1 Processor -- start 2024-03-28T05:25:35.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.230+0000 7efc00600740 1 -- start start 2024-03-28T05:25:35.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.230+0000 7efc00600740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e91d910 0x55a15e926710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:35.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.230+0000 7efc00600740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a15e91e230 0x55a15e932460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:35.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.230+0000 7efc00600740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a15e927000 0x55a15e9329a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:35.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.230+0000 7efc00600740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf81ac940 con 0x55a15e91d910 2024-03-28T05:25:35.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.230+0000 7efc00600740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf81aef00 con 0x55a15e91e230 2024-03-28T05:25:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.230+0000 7efc00600740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf81acf40 con 0x55a15e927000 2024-03-28T05:25:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.231+0000 7efbf6ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e91d910 0x55a15e926710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.231+0000 7efbf6ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e91d910 0x55a15e926710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48500/0 (socket says 172.21.15.23:48500) 2024-03-28T05:25:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.231+0000 7efbf6ffd640 1 -- 172.21.15.23:0/1343960974 learned_addr learned my addr 172.21.15.23:0/1343960974 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:25:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.231+0000 7efbf77fe640 1 --2- 172.21.15.23:0/1343960974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a15e927000 0x55a15e9329a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.231+0000 7efbee7fc640 1 --2- 172.21.15.23:0/1343960974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a15e91e230 0x55a15e932460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.231+0000 7efbf6ffd640 1 -- 172.21.15.23:0/1343960974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a15e927000 msgr2=0x55a15e9329a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.231+0000 7efbf6ffd640 1 --2- 172.21.15.23:0/1343960974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a15e927000 0x55a15e9329a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.231+0000 7efbf6ffd640 1 -- 172.21.15.23:0/1343960974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a15e91e230 msgr2=0x55a15e932460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.231+0000 7efbf6ffd640 1 --2- 172.21.15.23:0/1343960974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a15e91e230 0x55a15e932460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.231+0000 7efbf6ffd640 1 -- 172.21.15.23:0/1343960974 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a15e9399f0 con 0x55a15e91d910 2024-03-28T05:25:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.231+0000 7efbf6ffd640 1 --2- 172.21.15.23:0/1343960974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e91d910 0x55a15e926710 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7efbe802c210 tx=0x7efbe8005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3afa2d9711d76f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.231+0000 7efbed7fa640 1 -- 172.21.15.23:0/1343960974 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe8009520 con 0x55a15e91d910 2024-03-28T05:25:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.231+0000 7efbed7fa640 1 -- 172.21.15.23:0/1343960974 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efbe8003e60 con 0x55a15e91d910 2024-03-28T05:25:35.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.231+0000 7efbed7fa640 1 -- 172.21.15.23:0/1343960974 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe80053f0 con 0x55a15e91d910 2024-03-28T05:25:35.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.232+0000 7efc00600740 1 -- 172.21.15.23:0/1343960974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e91d910 msgr2=0x55a15e926710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:35.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.232+0000 7efc00600740 1 --2- 172.21.15.23:0/1343960974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e91d910 0x55a15e926710 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7efbe802c210 tx=0x7efbe8005e80 comp rx=0 tx=0).stop 2024-03-28T05:25:35.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.232+0000 7efc00600740 1 -- 172.21.15.23:0/1343960974 shutdown_connections 2024-03-28T05:25:35.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.232+0000 7efc00600740 1 --2- 172.21.15.23:0/1343960974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a15e927000 0x55a15e9329a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.232+0000 7efc00600740 1 --2- 172.21.15.23:0/1343960974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a15e91e230 0x55a15e932460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.232+0000 7efc00600740 1 --2- 172.21.15.23:0/1343960974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e91d910 0x55a15e926710 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.232+0000 7efc00600740 1 -- 172.21.15.23:0/1343960974 >> 172.21.15.23:0/1343960974 conn(0x55a15e918230 msgr2=0x55a15e91f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:35.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.232+0000 7efc00600740 1 -- 172.21.15.23:0/1343960974 shutdown_connections 2024-03-28T05:25:35.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.232+0000 7efc00600740 1 -- 172.21.15.23:0/1343960974 wait complete. 2024-03-28T05:25:35.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.232+0000 7efc00600740 1 -- start start 2024-03-28T05:25:35.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.232+0000 7efc00600740 1 Processor -- start 2024-03-28T05:25:35.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.233+0000 7efc00600740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a15e91d910 0x55a15e92ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:35.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.233+0000 7efc00600740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e91e230 0x55a15e921260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:35.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.233+0000 7efc00600740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a15e927000 0x55a15e9217a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:35.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.233+0000 7efc00600740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf81ad580 con 0x55a15e91e230 2024-03-28T05:25:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.233+0000 7efc00600740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf81ae8a0 con 0x55a15e927000 2024-03-28T05:25:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.233+0000 7efc00600740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf81adbe0 con 0x55a15e91d910 2024-03-28T05:25:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.233+0000 7efc00600740 10 client.0 _ll_register_callbacks cb 0x7efbe802ce20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:25:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.233+0000 7efbee7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e91e230 0x55a15e921260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.233+0000 7efbee7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e91e230 0x55a15e921260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48516/0 (socket says 172.21.15.23:48516) 2024-03-28T05:25:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.233+0000 7efbee7fc640 1 -- 172.21.15.23:0/474094495 learned_addr learned my addr 172.21.15.23:0/474094495 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:25:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.233+0000 7efbf6ffd640 1 --2- 172.21.15.23:0/474094495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a15e91d910 0x55a15e92ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.233+0000 7efbee7fc640 1 -- 172.21.15.23:0/474094495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a15e91d910 msgr2=0x55a15e92ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.233+0000 7efbee7fc640 1 --2- 172.21.15.23:0/474094495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a15e91d910 0x55a15e92ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.233+0000 7efbee7fc640 1 -- 172.21.15.23:0/474094495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a15e927000 msgr2=0x55a15e9217a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.233+0000 7efbf77fe640 1 --2- 172.21.15.23:0/474094495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a15e927000 0x55a15e9217a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.233+0000 7efbee7fc640 1 --2- 172.21.15.23:0/474094495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a15e927000 0x55a15e9217a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.233+0000 7efbee7fc640 1 -- 172.21.15.23:0/474094495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a15e921e10 con 0x55a15e91e230 2024-03-28T05:25:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.233+0000 7efbee7fc640 1 --2- 172.21.15.23:0/474094495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e91e230 0x55a15e921260 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7efbe009b120 tx=0x7efbe00ce040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3926370113677e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.233+0000 7efbd6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:25:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.233+0000 7efbd6ffd640 1 -- 172.21.15.23:0/474094495 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe001d040 con 0x55a15e91e230 2024-03-28T05:25:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.233+0000 7efbd6ffd640 1 -- 172.21.15.23:0/474094495 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7efbe0097800 con 0x55a15e91e230 2024-03-28T05:25:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.233+0000 7efc00600740 1 -- 172.21.15.23:0/474094495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efbe8003e60 con 0x55a15e91e230 2024-03-28T05:25:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.234+0000 7efbd6ffd640 1 -- 172.21.15.23:0/474094495 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7efbe00a3880 con 0x55a15e91e230 2024-03-28T05:25:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.234+0000 7efc00600740 10 client.15645 fetch_fsmap learned FSMap version 20 2024-03-28T05:25:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.234+0000 7efc00600740 1 -- 172.21.15.23:0/474094495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55a15e993a10 con 0x55a15e91e230 2024-03-28T05:25:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.234+0000 7efbd6ffd640 1 -- 172.21.15.23:0/474094495 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efbe8003e60 con 0x55a15e91e230 2024-03-28T05:25:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.234+0000 7efbd6ffd640 1 -- 172.21.15.23:0/474094495 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7efbe00a33f0 con 0x55a15e91e230 2024-03-28T05:25:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.234+0000 7efc00600740 10 client.15645 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:25:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.234+0000 7efc00600740 10 client.15645 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:25:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.234+0000 7efc00600740 20 client.15645 populate_metadata read hostname 'smithi023' 2024-03-28T05:25:35.240 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.234+0000 7efc00600740 1 --2- 172.21.15.23:0/474094495 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a15e994140 0x55a15e996560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:35.240 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.234+0000 7efc00600740 4 client.15645 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:25:35.240 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.234+0000 7efc00600740 1 -- 172.21.15.23:0/474094495 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a15e8893a0 con 0x55a15e994140 2024-03-28T05:25:35.240 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.234+0000 7efbf6ffd640 1 --2- 172.21.15.23:0/474094495 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a15e994140 0x55a15e996560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:35.240 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.235+0000 7efbf6ffd640 1 --2- 172.21.15.23:0/474094495 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a15e994140 0x55a15e996560 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d97f7dc69c3aa01d server_cookie=db4eac0826c98567 in_seq=0 out_seq=0 2024-03-28T05:25:35.240 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.235+0000 7efbd6ffd640 10 client.15645 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:25:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.658+0000 7efbd6ffd640 1 -- 172.21.15.23:0/474094495 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a15e8893a0 con 0x55a15e994140 2024-03-28T05:25:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.658+0000 7efbd6ffd640 10 client.15645 handle_command_reply: tid=0 2024-03-28T05:25:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.658+0000 7efc00600740 1 client.15645 shutdown 2024-03-28T05:25:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.659+0000 7efc00600740 1 -- 172.21.15.23:0/474094495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e91e230 msgr2=0x55a15e921260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.659+0000 7efc00600740 1 --2- 172.21.15.23:0/474094495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e91e230 0x55a15e921260 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7efbe009b120 tx=0x7efbe00ce040 comp rx=0 tx=0).stop 2024-03-28T05:25:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.659+0000 7efc00600740 1 -- 172.21.15.23:0/474094495 shutdown_connections 2024-03-28T05:25:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.659+0000 7efc00600740 1 --2- 172.21.15.23:0/474094495 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a15e994140 0x55a15e996560 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.659+0000 7efc00600740 1 --2- 172.21.15.23:0/474094495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a15e927000 0x55a15e9217a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.659+0000 7efc00600740 1 --2- 172.21.15.23:0/474094495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a15e91e230 0x55a15e921260 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.659+0000 7efbd6ffd640 0 client.15645 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:25:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.659+0000 7efc00600740 1 --2- 172.21.15.23:0/474094495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a15e91d910 0x55a15e92ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.659+0000 7efc00600740 1 -- 172.21.15.23:0/474094495 >> 172.21.15.23:0/474094495 conn(0x55a15e918230 msgr2=0x55a15e916a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.659+0000 7efc00600740 1 -- 172.21.15.23:0/474094495 shutdown_connections 2024-03-28T05:25:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.659+0000 7efc00600740 1 -- 172.21.15.23:0/474094495 wait complete. 2024-03-28T05:25:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.659+0000 7efc00600740 20 client.15645 trim_cache size 0 max 16384 2024-03-28T05:25:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:25:35.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:25:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:25:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:25:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.659+0000 7efbfd93b640 1 -- 172.21.15.23:0/1845074379 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7efbe40a2960 msgr2=0x7efbe40a4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.659+0000 7efbfd93b640 1 --2- 172.21.15.23:0/1845074379 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7efbe40a2960 0x7efbe40a4e20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efbf0005fd0 tx=0x7efbf0009e90 comp rx=0 tx=0).stop 2024-03-28T05:25:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.659+0000 7efbfd93b640 1 -- 172.21.15.23:0/1845074379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbf8103000 msgr2=0x7efbf8198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.659+0000 7efbfd93b640 1 --2- 172.21.15.23:0/1845074379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbf8103000 0x7efbf8198390 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7efbe000c890 tx=0x7efbe000cd50 comp rx=0 tx=0).stop 2024-03-28T05:25:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.660+0000 7efbfd93b640 1 -- 172.21.15.23:0/1845074379 shutdown_connections 2024-03-28T05:25:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.660+0000 7efbfd93b640 1 --2- 172.21.15.23:0/1845074379 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efbf810bab0 0x7efbf819c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.660+0000 7efbfd93b640 1 --2- 172.21.15.23:0/1845074379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efbf8103000 0x7efbf8198390 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.660+0000 7efbfd93b640 1 --2- 172.21.15.23:0/1845074379 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7efbe40a2960 0x7efbe40a4e20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.660+0000 7efbfd93b640 1 --2- 172.21.15.23:0/1845074379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efbf81026e0 0x7efbf8197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.660+0000 7efbfd93b640 1 -- 172.21.15.23:0/1845074379 >> 172.21.15.23:0/1845074379 conn(0x7efbf80fbee0 msgr2=0x7efbf80fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.660+0000 7efbfd93b640 1 -- 172.21.15.23:0/1845074379 shutdown_connections 2024-03-28T05:25:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:35.660+0000 7efbfd93b640 1 -- 172.21.15.23:0/1845074379 wait complete. 2024-03-28T05:25:35.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:35 smithi113 ceph-mon[37164]: pgmap v63: 129 pgs: 129 active+clean; 23 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 500 KiB/s wr, 327 op/s 2024-03-28T05:25:36.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:35 smithi023 ceph-mon[31954]: pgmap v63: 129 pgs: 129 active+clean; 23 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 500 KiB/s wr, 327 op/s 2024-03-28T05:25:36.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:35 smithi098 ceph-mon[36087]: pgmap v63: 129 pgs: 129 active+clean; 23 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 500 KiB/s wr, 327 op/s 2024-03-28T05:25:36.398 INFO:tasks.workunit.client.0.smithi023.stdout: 9 57764 90 41879 59 20533 131 2024-03-28T05:25:37.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:37 smithi113 ceph-mon[37164]: pgmap v64: 129 pgs: 129 active+clean; 24 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 589 KiB/s wr, 341 op/s 2024-03-28T05:25:38.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:37 smithi023 ceph-mon[31954]: pgmap v64: 129 pgs: 129 active+clean; 24 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 589 KiB/s wr, 341 op/s 2024-03-28T05:25:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:37 smithi098 ceph-mon[36087]: pgmap v64: 129 pgs: 129 active+clean; 24 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 589 KiB/s wr, 341 op/s 2024-03-28T05:25:39.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:39 smithi113 ceph-mon[37164]: pgmap v65: 129 pgs: 129 active+clean; 24 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 265 KiB/s wr, 42 op/s 2024-03-28T05:25:40.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:39 smithi023 ceph-mon[31954]: pgmap v65: 129 pgs: 129 active+clean; 24 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 265 KiB/s wr, 42 op/s 2024-03-28T05:25:40.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:39 smithi098 ceph-mon[36087]: pgmap v65: 129 pgs: 129 active+clean; 24 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 265 KiB/s wr, 42 op/s 2024-03-28T05:25:41.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:40 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:25:41.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:40 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:25:41.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:40 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:25:42.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:41 smithi023 ceph-mon[31954]: pgmap v66: 129 pgs: 129 active+clean; 38 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 2.0 MiB/s wr, 377 op/s 2024-03-28T05:25:42.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:41 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:25:42.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:41 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:25:42.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:41 smithi098 ceph-mon[36087]: pgmap v66: 129 pgs: 129 active+clean; 38 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 2.0 MiB/s wr, 377 op/s 2024-03-28T05:25:42.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:41 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:25:42.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:41 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:25:42.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:41 smithi113 ceph-mon[37164]: pgmap v66: 129 pgs: 129 active+clean; 38 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 2.0 MiB/s wr, 377 op/s 2024-03-28T05:25:42.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:41 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:25:42.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:41 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:25:43.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:42 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:25:43.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:42 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:25:43.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:42 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:25:43.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:42 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:25:43.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:42 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:25:43.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:42 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:25:43.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:42 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:25:43.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:42 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:25:43.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:42 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:25:44.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:43 smithi023 ceph-mon[31954]: pgmap v67: 129 pgs: 129 active+clean; 38 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 1.8 MiB/s wr, 349 op/s 2024-03-28T05:25:44.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:43 smithi098 ceph-mon[36087]: pgmap v67: 129 pgs: 129 active+clean; 38 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 1.8 MiB/s wr, 349 op/s 2024-03-28T05:25:44.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:43 smithi113 ceph-mon[37164]: pgmap v67: 129 pgs: 129 active+clean; 38 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 1.8 MiB/s wr, 349 op/s 2024-03-28T05:25:46.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:45 smithi023 ceph-mon[31954]: pgmap v68: 129 pgs: 129 active+clean; 38 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 1.8 MiB/s wr, 349 op/s 2024-03-28T05:25:46.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:45 smithi098 ceph-mon[36087]: pgmap v68: 129 pgs: 129 active+clean; 38 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 1.8 MiB/s wr, 349 op/s 2024-03-28T05:25:46.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:45 smithi113 ceph-mon[37164]: pgmap v68: 129 pgs: 129 active+clean; 38 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 1.8 MiB/s wr, 349 op/s 2024-03-28T05:25:46.399 INFO:tasks.workunit.client.0.smithi023.stdout: 10 153243 137 107892 82 59912 308 2024-03-28T05:25:48.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:47 smithi023 ceph-mon[31954]: pgmap v69: 129 pgs: 129 active+clean; 40 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 2.0 MiB/s wr, 397 op/s 2024-03-28T05:25:48.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:47 smithi098 ceph-mon[36087]: pgmap v69: 129 pgs: 129 active+clean; 40 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 2.0 MiB/s wr, 397 op/s 2024-03-28T05:25:48.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:47 smithi113 ceph-mon[37164]: pgmap v69: 129 pgs: 129 active+clean; 40 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 2.0 MiB/s wr, 397 op/s 2024-03-28T05:25:48.753 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.751+0000 7f296afb9640 1 -- 172.21.15.23:0/87268196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2964102030 msgr2=0x7f296410e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:48.753 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.751+0000 7f296afb9640 1 --2- 172.21.15.23:0/87268196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2964102030 0x7f296410e540 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f295c009cd0 tx=0x7f295c02c0f0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.753 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.751+0000 7f296afb9640 1 -- 172.21.15.23:0/87268196 shutdown_connections 2024-03-28T05:25:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.751+0000 7f296afb9640 1 --2- 172.21.15.23:0/87268196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2964102030 0x7f296410e540 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.751+0000 7f296afb9640 1 --2- 172.21.15.23:0/87268196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2964101670 0x7f2964101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.751+0000 7f296afb9640 1 --2- 172.21.15.23:0/87268196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2964107680 0x7f2964107a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.751+0000 7f296afb9640 1 -- 172.21.15.23:0/87268196 >> 172.21.15.23:0/87268196 conn(0x7f29640fd180 msgr2=0x7f29640ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.751+0000 7f296afb9640 1 -- 172.21.15.23:0/87268196 shutdown_connections 2024-03-28T05:25:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.751+0000 7f296afb9640 1 -- 172.21.15.23:0/87268196 wait complete. 2024-03-28T05:25:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.752+0000 7f296afb9640 1 Processor -- start 2024-03-28T05:25:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.752+0000 7f296afb9640 1 -- start start 2024-03-28T05:25:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.752+0000 7f296afb9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2964101670 0x7f2964197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.752+0000 7f296afb9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2964102030 0x7f2964198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.752+0000 7f296afb9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2964107680 0x7f296419c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.752+0000 7f296afb9640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2964111a90 con 0x7f2964107680 2024-03-28T05:25:48.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.752+0000 7f296afb9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2964111910 con 0x7f2964101670 2024-03-28T05:25:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.752+0000 7f296afb9640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2964111c10 con 0x7f2964102030 2024-03-28T05:25:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.752+0000 7f2969556640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2964107680 0x7f296419c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.752+0000 7f2969556640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2964107680 0x7f296419c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39270/0 (socket says 172.21.15.23:39270) 2024-03-28T05:25:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.752+0000 7f2969556640 1 -- 172.21.15.23:0/3913418248 learned_addr learned my addr 172.21.15.23:0/3913418248 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:25:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.752+0000 7f2963fff640 1 --2- 172.21.15.23:0/3913418248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2964102030 0x7f2964198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.752+0000 7f2968d55640 1 --2- 172.21.15.23:0/3913418248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2964101670 0x7f2964197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.752+0000 7f2969556640 1 -- 172.21.15.23:0/3913418248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2964102030 msgr2=0x7f2964198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.752+0000 7f2969556640 1 --2- 172.21.15.23:0/3913418248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2964102030 0x7f2964198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.752+0000 7f2969556640 1 -- 172.21.15.23:0/3913418248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2964101670 msgr2=0x7f2964197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.752+0000 7f2969556640 1 --2- 172.21.15.23:0/3913418248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2964101670 0x7f2964197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.752+0000 7f2969556640 1 -- 172.21.15.23:0/3913418248 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296419ce70 con 0x7f2964107680 2024-03-28T05:25:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.752+0000 7f2969556640 1 --2- 172.21.15.23:0/3913418248 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2964107680 0x7f296419c740 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f295c005e30 tx=0x7f295c032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.753+0000 7f2961ffb640 1 -- 172.21.15.23:0/3913418248 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c041020 con 0x7f2964107680 2024-03-28T05:25:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.753+0000 7f2961ffb640 1 -- 172.21.15.23:0/3913418248 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f295c034cd0 con 0x7f2964107680 2024-03-28T05:25:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.753+0000 7f296afb9640 1 -- 172.21.15.23:0/3913418248 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f296419d100 con 0x7f2964107680 2024-03-28T05:25:48.755 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.753+0000 7f2961ffb640 1 -- 172.21.15.23:0/3913418248 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c039040 con 0x7f2964107680 2024-03-28T05:25:48.756 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.753+0000 7f296afb9640 1 -- 172.21.15.23:0/3913418248 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29641a49f0 con 0x7f2964107680 2024-03-28T05:25:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.755+0000 7f2961ffb640 1 -- 172.21.15.23:0/3913418248 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f295c044020 con 0x7f2964107680 2024-03-28T05:25:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.755+0000 7f296dba2740 1 Processor -- start 2024-03-28T05:25:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.755+0000 7f296dba2740 1 -- start start 2024-03-28T05:25:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.756+0000 7f296dba2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a08831b0 0x5556a0883590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.756+0000 7f296dba2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556a0883ad0 0x5556a0890cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.756+0000 7f2961ffb640 1 --2- 172.21.15.23:0/3913418248 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f29540a2a10 0x7f29540a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.756+0000 7f296dba2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a0891210 0x5556a0893600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.756+0000 7f296dba2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a07e7b10 con 0x5556a0891210 2024-03-28T05:25:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.756+0000 7f296dba2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a0790900 con 0x5556a0883ad0 2024-03-28T05:25:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.756+0000 7f296dba2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a07e3bb0 con 0x5556a08831b0 2024-03-28T05:25:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.756+0000 7f2963fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a08831b0 0x5556a0883590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.756+0000 7f2969556640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a0891210 0x5556a0893600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.756+0000 7f2963fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a08831b0 0x5556a0883590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:56410/0 (socket says 172.21.15.23:56410) 2024-03-28T05:25:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.756+0000 7f2963fff640 1 -- 172.21.15.23:0/285738427 learned_addr learned my addr 172.21.15.23:0/285738427 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:25:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.756+0000 7f2963fff640 1 --2- 172.21.15.23:0/3913418248 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f29540a2a10 0x7f29540a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.756+0000 7f2968d55640 1 --2- 172.21.15.23:0/285738427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556a0883ad0 0x5556a0890cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.756+0000 7f2961ffb640 1 -- 172.21.15.23:0/3913418248 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f295c0f0d20 con 0x7f2964107680 2024-03-28T05:25:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.756+0000 7f2963fff640 1 -- 172.21.15.23:0/285738427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556a0883ad0 msgr2=0x5556a0890cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.756+0000 7f2963fff640 1 --2- 172.21.15.23:0/285738427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556a0883ad0 0x5556a0890cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.756+0000 7f2963fff640 1 -- 172.21.15.23:0/285738427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a0891210 msgr2=0x5556a0893600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.756+0000 7f2963fff640 1 --2- 172.21.15.23:0/285738427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a0891210 0x5556a0893600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.756+0000 7f2963fff640 1 -- 172.21.15.23:0/285738427 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556a04fc210 con 0x5556a08831b0 2024-03-28T05:25:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.757+0000 7f2963fff640 1 --2- 172.21.15.23:0/3913418248 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f29540a2a10 0x7f29540a4ed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f295000f8c0 tx=0x7f2950018000 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.757+0000 7f2963fff640 1 --2- 172.21.15.23:0/285738427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a08831b0 0x5556a0883590 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2950011ae0 tx=0x7f295000f3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd576a9016cfe0be server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.757+0000 7f294effd640 1 -- 172.21.15.23:0/285738427 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2950011050 con 0x5556a08831b0 2024-03-28T05:25:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.757+0000 7f294effd640 1 -- 172.21.15.23:0/285738427 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29500111f0 con 0x5556a08831b0 2024-03-28T05:25:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.757+0000 7f294effd640 1 -- 172.21.15.23:0/285738427 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29500035f0 con 0x5556a08831b0 2024-03-28T05:25:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.758+0000 7f296dba2740 1 -- 172.21.15.23:0/285738427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a08831b0 msgr2=0x5556a0883590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.758+0000 7f296dba2740 1 --2- 172.21.15.23:0/285738427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a08831b0 0x5556a0883590 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2950011ae0 tx=0x7f295000f3c0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.758+0000 7f296dba2740 1 -- 172.21.15.23:0/285738427 shutdown_connections 2024-03-28T05:25:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.758+0000 7f296dba2740 1 --2- 172.21.15.23:0/285738427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a0891210 0x5556a0893600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.758+0000 7f296dba2740 1 --2- 172.21.15.23:0/285738427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556a0883ad0 0x5556a0890cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.758+0000 7f296dba2740 1 --2- 172.21.15.23:0/285738427 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a08831b0 0x5556a0883590 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.758+0000 7f296dba2740 1 -- 172.21.15.23:0/285738427 >> 172.21.15.23:0/285738427 conn(0x5556a087e030 msgr2=0x5556a087e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.758+0000 7f296dba2740 1 -- 172.21.15.23:0/285738427 shutdown_connections 2024-03-28T05:25:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.758+0000 7f296dba2740 1 -- 172.21.15.23:0/285738427 wait complete. 2024-03-28T05:25:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.758+0000 7f296dba2740 1 -- start start 2024-03-28T05:25:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.758+0000 7f296dba2740 1 Processor -- start 2024-03-28T05:25:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.758+0000 7f296dba2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556a08831b0 0x5556a088e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.758+0000 7f296dba2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a0883ad0 0x5556a088e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.758+0000 7f296dba2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a0891210 0x5556a088ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.758+0000 7f296dba2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a0893de0 con 0x5556a0883ad0 2024-03-28T05:25:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.758+0000 7f296dba2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a07a3d50 con 0x5556a08831b0 2024-03-28T05:25:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.758+0000 7f296dba2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556a0893f60 con 0x5556a0891210 2024-03-28T05:25:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.758+0000 7f296dba2740 10 client.0 _ll_register_callbacks cb 0x5556a07a3720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:25:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.758+0000 7f2963fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a0883ad0 0x5556a088e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.759+0000 7f2963fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a0883ad0 0x5556a088e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39282/0 (socket says 172.21.15.23:39282) 2024-03-28T05:25:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.759+0000 7f2963fff640 1 -- 172.21.15.23:0/52325483 learned_addr learned my addr 172.21.15.23:0/52325483 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:25:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.759+0000 7f2969556640 1 --2- 172.21.15.23:0/52325483 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a0891210 0x5556a088ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.759+0000 7f2968d55640 1 --2- 172.21.15.23:0/52325483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556a08831b0 0x5556a088e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.759+0000 7f2963fff640 1 -- 172.21.15.23:0/52325483 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a0891210 msgr2=0x5556a088ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.759+0000 7f2963fff640 1 --2- 172.21.15.23:0/52325483 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a0891210 0x5556a088ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.759+0000 7f2963fff640 1 -- 172.21.15.23:0/52325483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556a08831b0 msgr2=0x5556a088e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.759+0000 7f2963fff640 1 --2- 172.21.15.23:0/52325483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556a08831b0 0x5556a088e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.759+0000 7f2963fff640 1 -- 172.21.15.23:0/52325483 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5556a05c8620 con 0x5556a0883ad0 2024-03-28T05:25:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.759+0000 7f2963fff640 1 --2- 172.21.15.23:0/52325483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a0883ad0 0x5556a088e720 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f295000eb50 tx=0x7f2950005530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d58244c4e0dc1b71 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.759+0000 7f2943fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:25:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.759+0000 7f2943fff640 1 -- 172.21.15.23:0/52325483 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2950011050 con 0x5556a0883ad0 2024-03-28T05:25:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.759+0000 7f2943fff640 1 -- 172.21.15.23:0/52325483 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f295000ed50 con 0x5556a0883ad0 2024-03-28T05:25:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.759+0000 7f296dba2740 1 -- 172.21.15.23:0/52325483 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f29500111f0 con 0x5556a0883ad0 2024-03-28T05:25:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.759+0000 7f2943fff640 1 -- 172.21.15.23:0/52325483 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29500035f0 con 0x5556a0883ad0 2024-03-28T05:25:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.759+0000 7f296dba2740 10 client.15657 fetch_fsmap learned FSMap version 20 2024-03-28T05:25:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.759+0000 7f296dba2740 1 -- 172.21.15.23:0/52325483 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5556a08a0060 con 0x5556a0883ad0 2024-03-28T05:25:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.760+0000 7f2943fff640 1 -- 172.21.15.23:0/52325483 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29500111f0 con 0x5556a0883ad0 2024-03-28T05:25:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.760+0000 7f2943fff640 1 -- 172.21.15.23:0/52325483 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f2950032420 con 0x5556a0883ad0 2024-03-28T05:25:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.760+0000 7f296dba2740 10 client.15657 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:25:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.760+0000 7f296dba2740 10 client.15657 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:25:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.760+0000 7f296dba2740 20 client.15657 populate_metadata read hostname 'smithi023' 2024-03-28T05:25:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.760+0000 7f296dba2740 1 --2- 172.21.15.23:0/52325483 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5556a08a14c0 0x5556a08a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.760+0000 7f296dba2740 4 client.15657 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:25:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.760+0000 7f296dba2740 1 -- 172.21.15.23:0/52325483 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5556a07f5050 con 0x5556a08a14c0 2024-03-28T05:25:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.760+0000 7f2968d55640 1 --2- 172.21.15.23:0/52325483 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5556a08a14c0 0x5556a08a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.760+0000 7f2968d55640 1 --2- 172.21.15.23:0/52325483 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5556a08a14c0 0x5556a08a38b0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=767312779fb83ec9 server_cookie=74a51d18b59ff41f in_seq=0 out_seq=0 2024-03-28T05:25:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.760+0000 7f2943fff640 10 client.15657 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:25:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.762+0000 7f2943fff640 1 -- 172.21.15.23:0/52325483 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5556a07f5050 con 0x5556a08a14c0 2024-03-28T05:25:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.762+0000 7f2943fff640 10 client.15657 handle_command_reply: tid=0 2024-03-28T05:25:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.762+0000 7f296dba2740 1 client.15657 shutdown 2024-03-28T05:25:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.762+0000 7f296dba2740 1 -- 172.21.15.23:0/52325483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a0883ad0 msgr2=0x5556a088e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.762+0000 7f296dba2740 1 --2- 172.21.15.23:0/52325483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a0883ad0 0x5556a088e720 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f295000eb50 tx=0x7f2950005530 comp rx=0 tx=0).stop 2024-03-28T05:25:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.762+0000 7f296dba2740 1 -- 172.21.15.23:0/52325483 shutdown_connections 2024-03-28T05:25:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.762+0000 7f296dba2740 1 --2- 172.21.15.23:0/52325483 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a0891210 0x5556a088ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.762+0000 7f296dba2740 1 --2- 172.21.15.23:0/52325483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a0883ad0 0x5556a088e720 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.762+0000 7f296dba2740 1 --2- 172.21.15.23:0/52325483 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5556a08a14c0 0x5556a08a38b0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.762+0000 7f296dba2740 1 --2- 172.21.15.23:0/52325483 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556a08831b0 0x5556a088e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.762+0000 7f2943fff640 0 client.15657 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:25:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.762+0000 7f296dba2740 1 -- 172.21.15.23:0/52325483 >> 172.21.15.23:0/52325483 conn(0x5556a087e030 msgr2=0x5556a087fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.762+0000 7f296dba2740 1 -- 172.21.15.23:0/52325483 shutdown_connections 2024-03-28T05:25:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.762+0000 7f296dba2740 1 -- 172.21.15.23:0/52325483 wait complete. 2024-03-28T05:25:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.762+0000 7f296dba2740 20 client.15657 trim_cache size 0 max 16384 2024-03-28T05:25:48.781 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.779+0000 7f296dba2740 1 Processor -- start 2024-03-28T05:25:48.781 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.779+0000 7f296dba2740 1 -- start start 2024-03-28T05:25:48.781 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.779+0000 7f296dba2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a08831b0 0x5556a0890910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:48.781 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.779+0000 7f296dba2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a0883ad0 0x5556a087fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:48.781 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.779+0000 7f296dba2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556a0891210 0x5556a089c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:48.781 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.779+0000 7f296dba2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29641aa870 con 0x5556a08831b0 2024-03-28T05:25:48.781 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.779+0000 7f296dba2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29641ace30 con 0x5556a0891210 2024-03-28T05:25:48.781 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.779+0000 7f296dba2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29641aae70 con 0x5556a0883ad0 2024-03-28T05:25:48.781 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.779+0000 7f2968d55640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a08831b0 0x5556a0890910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:48.781 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.779+0000 7f2968d55640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a08831b0 0x5556a0890910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39292/0 (socket says 172.21.15.23:39292) 2024-03-28T05:25:48.781 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.779+0000 7f2968d55640 1 -- 172.21.15.23:0/1424209274 learned_addr learned my addr 172.21.15.23:0/1424209274 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:25:48.781 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.779+0000 7f2963fff640 1 --2- 172.21.15.23:0/1424209274 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a0883ad0 0x5556a087fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:48.781 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.779+0000 7f2969556640 1 --2- 172.21.15.23:0/1424209274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556a0891210 0x5556a089c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:48.781 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.779+0000 7f2968d55640 1 -- 172.21.15.23:0/1424209274 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a0883ad0 msgr2=0x5556a087fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:48.782 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.779+0000 7f2968d55640 1 --2- 172.21.15.23:0/1424209274 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a0883ad0 0x5556a087fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.782 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.779+0000 7f2968d55640 1 -- 172.21.15.23:0/1424209274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556a0891210 msgr2=0x5556a089c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:48.782 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.779+0000 7f2968d55640 1 --2- 172.21.15.23:0/1424209274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556a0891210 0x5556a089c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.782 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.779+0000 7f2968d55640 1 -- 172.21.15.23:0/1424209274 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556a088c770 con 0x5556a08831b0 2024-03-28T05:25:48.782 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.780+0000 7f2968d55640 1 --2- 172.21.15.23:0/1424209274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a08831b0 0x5556a0890910 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2958009e00 tx=0x7f2958005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31280756b1541839 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:48.782 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.780+0000 7f294e7fc640 1 -- 172.21.15.23:0/1424209274 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29580201b0 con 0x5556a08831b0 2024-03-28T05:25:48.782 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.780+0000 7f294e7fc640 1 -- 172.21.15.23:0/1424209274 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2958005020 con 0x5556a08831b0 2024-03-28T05:25:48.782 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.780+0000 7f294e7fc640 1 -- 172.21.15.23:0/1424209274 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2958004bc0 con 0x5556a08831b0 2024-03-28T05:25:48.782 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.781+0000 7f296dba2740 1 -- 172.21.15.23:0/1424209274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a08831b0 msgr2=0x5556a0890910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:48.782 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.781+0000 7f296dba2740 1 --2- 172.21.15.23:0/1424209274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a08831b0 0x5556a0890910 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2958009e00 tx=0x7f2958005f90 comp rx=0 tx=0).stop 2024-03-28T05:25:48.782 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.781+0000 7f296dba2740 1 -- 172.21.15.23:0/1424209274 shutdown_connections 2024-03-28T05:25:48.782 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.781+0000 7f296dba2740 1 --2- 172.21.15.23:0/1424209274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556a0891210 0x5556a089c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.782 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.781+0000 7f296dba2740 1 --2- 172.21.15.23:0/1424209274 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a0883ad0 0x5556a087fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.782 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.781+0000 7f296dba2740 1 --2- 172.21.15.23:0/1424209274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a08831b0 0x5556a0890910 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.782 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.781+0000 7f296dba2740 1 -- 172.21.15.23:0/1424209274 >> 172.21.15.23:0/1424209274 conn(0x5556a087e030 msgr2=0x5556a087f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:48.783 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.781+0000 7f296dba2740 1 -- 172.21.15.23:0/1424209274 shutdown_connections 2024-03-28T05:25:48.783 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.781+0000 7f296dba2740 1 -- 172.21.15.23:0/1424209274 wait complete. 2024-03-28T05:25:48.783 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.781+0000 7f296dba2740 1 -- start start 2024-03-28T05:25:48.783 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.781+0000 7f296dba2740 1 Processor -- start 2024-03-28T05:25:48.785 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.781+0000 7f296dba2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556a08831b0 0x5556a088ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:48.785 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.781+0000 7f296dba2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a0883ad0 0x5556a088b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:48.785 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.781+0000 7f296dba2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a0891210 0x5556a08fbb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:48.785 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.781+0000 7f296dba2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29641ab4b0 con 0x5556a0883ad0 2024-03-28T05:25:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.781+0000 7f296dba2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29641ac7d0 con 0x5556a08831b0 2024-03-28T05:25:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.781+0000 7f296dba2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29641abb10 con 0x5556a0891210 2024-03-28T05:25:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.782+0000 7f296dba2740 10 client.0 _ll_register_callbacks cb 0x7f29500118b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:25:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.782+0000 7f2963fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a0883ad0 0x5556a088b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.782+0000 7f2963fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a0883ad0 0x5556a088b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39294/0 (socket says 172.21.15.23:39294) 2024-03-28T05:25:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.782+0000 7f2963fff640 1 -- 172.21.15.23:0/1734027453 learned_addr learned my addr 172.21.15.23:0/1734027453 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:25:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.782+0000 7f2969556640 1 --2- 172.21.15.23:0/1734027453 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a0891210 0x5556a08fbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.782+0000 7f2968d55640 1 --2- 172.21.15.23:0/1734027453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556a08831b0 0x5556a088ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.782+0000 7f2963fff640 1 -- 172.21.15.23:0/1734027453 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a0891210 msgr2=0x5556a08fbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.782+0000 7f2963fff640 1 --2- 172.21.15.23:0/1734027453 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a0891210 0x5556a08fbb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.782+0000 7f2963fff640 1 -- 172.21.15.23:0/1734027453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556a08831b0 msgr2=0x5556a088ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:25:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.782+0000 7f2963fff640 1 --2- 172.21.15.23:0/1734027453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556a08831b0 0x5556a088ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.782+0000 7f2963fff640 1 -- 172.21.15.23:0/1734027453 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5556a08fc040 con 0x5556a0883ad0 2024-03-28T05:25:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.782+0000 7f2963fff640 1 --2- 172.21.15.23:0/1734027453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a0883ad0 0x5556a088b140 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f29500117f0 tx=0x7f2950032cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=848041366e7410b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:25:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.782+0000 7f2943fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:25:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.782+0000 7f2943fff640 1 -- 172.21.15.23:0/1734027453 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2950009b90 con 0x5556a0883ad0 2024-03-28T05:25:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.782+0000 7f2943fff640 1 -- 172.21.15.23:0/1734027453 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f2950011030 con 0x5556a0883ad0 2024-03-28T05:25:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.782+0000 7f296dba2740 1 -- 172.21.15.23:0/1734027453 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2958005020 con 0x5556a0883ad0 2024-03-28T05:25:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.782+0000 7f2943fff640 1 -- 172.21.15.23:0/1734027453 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2950005350 con 0x5556a0883ad0 2024-03-28T05:25:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.782+0000 7f296dba2740 10 client.15663 fetch_fsmap learned FSMap version 20 2024-03-28T05:25:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.782+0000 7f296dba2740 1 -- 172.21.15.23:0/1734027453 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5556a08fd130 con 0x5556a0883ad0 2024-03-28T05:25:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.783+0000 7f2943fff640 1 -- 172.21.15.23:0/1734027453 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2958005020 con 0x5556a0883ad0 2024-03-28T05:25:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.783+0000 7f2943fff640 1 -- 172.21.15.23:0/1734027453 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f295000eb80 con 0x5556a0883ad0 2024-03-28T05:25:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.783+0000 7f296dba2740 10 client.15663 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:25:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.783+0000 7f296dba2740 10 client.15663 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:25:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.783+0000 7f296dba2740 20 client.15663 populate_metadata read hostname 'smithi023' 2024-03-28T05:25:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.783+0000 7f296dba2740 1 --2- 172.21.15.23:0/1734027453 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5556a08fd470 0x5556a08ff860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:25:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.783+0000 7f296dba2740 4 client.15663 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:25:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.783+0000 7f296dba2740 1 -- 172.21.15.23:0/1734027453 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5556a07ef3a0 con 0x5556a08fd470 2024-03-28T05:25:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.783+0000 7f2968d55640 1 --2- 172.21.15.23:0/1734027453 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5556a08fd470 0x5556a08ff860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:25:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.783+0000 7f2943fff640 10 client.15663 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:25:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:48.783+0000 7f2968d55640 1 --2- 172.21.15.23:0/1734027453 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5556a08fd470 0x5556a08ff860 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=859db6af5414b02f server_cookie=464fddb5b18b45af in_seq=0 out_seq=0 2024-03-28T05:25:49.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.160+0000 7f2943fff640 1 -- 172.21.15.23:0/1734027453 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5556a07ef3a0 con 0x5556a08fd470 2024-03-28T05:25:49.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.160+0000 7f2943fff640 10 client.15663 handle_command_reply: tid=0 2024-03-28T05:25:49.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.160+0000 7f296dba2740 1 client.15663 shutdown 2024-03-28T05:25:49.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.160+0000 7f296dba2740 1 -- 172.21.15.23:0/1734027453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a0883ad0 msgr2=0x5556a088b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:49.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.160+0000 7f296dba2740 1 --2- 172.21.15.23:0/1734027453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a0883ad0 0x5556a088b140 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f29500117f0 tx=0x7f2950032cb0 comp rx=0 tx=0).stop 2024-03-28T05:25:49.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.160+0000 7f296dba2740 1 -- 172.21.15.23:0/1734027453 shutdown_connections 2024-03-28T05:25:49.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.160+0000 7f296dba2740 1 --2- 172.21.15.23:0/1734027453 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5556a0891210 0x5556a08fbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:49.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.160+0000 7f296dba2740 1 --2- 172.21.15.23:0/1734027453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556a0883ad0 0x5556a088b140 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:49.161 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.160+0000 7f296dba2740 1 --2- 172.21.15.23:0/1734027453 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5556a08fd470 0x5556a08ff860 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.160+0000 7f296dba2740 1 --2- 172.21.15.23:0/1734027453 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556a08831b0 0x5556a088ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.160+0000 7f2943fff640 0 client.15663 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:25:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.160+0000 7f296dba2740 1 -- 172.21.15.23:0/1734027453 >> 172.21.15.23:0/1734027453 conn(0x5556a087e030 msgr2=0x5556a087fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.161+0000 7f296dba2740 1 -- 172.21.15.23:0/1734027453 shutdown_connections 2024-03-28T05:25:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.161+0000 7f296dba2740 1 -- 172.21.15.23:0/1734027453 wait complete. 2024-03-28T05:25:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.161+0000 7f296dba2740 20 client.15663 trim_cache size 0 max 16384 2024-03-28T05:25:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:25:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:25:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:25:49.162 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:25:49.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.162+0000 7f296afb9640 1 -- 172.21.15.23:0/3913418248 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f29540a2a10 msgr2=0x7f29540a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:49.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.162+0000 7f296afb9640 1 --2- 172.21.15.23:0/3913418248 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f29540a2a10 0x7f29540a4ed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f295000f8c0 tx=0x7f2950018000 comp rx=0 tx=0).stop 2024-03-28T05:25:49.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.162+0000 7f296afb9640 1 -- 172.21.15.23:0/3913418248 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2964107680 msgr2=0x7f296419c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:25:49.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.162+0000 7f296afb9640 1 --2- 172.21.15.23:0/3913418248 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2964107680 0x7f296419c740 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f295c005e30 tx=0x7f295c032ea0 comp rx=0 tx=0).stop 2024-03-28T05:25:49.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.162+0000 7f296afb9640 1 -- 172.21.15.23:0/3913418248 shutdown_connections 2024-03-28T05:25:49.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.162+0000 7f296afb9640 1 --2- 172.21.15.23:0/3913418248 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2964107680 0x7f296419c740 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:49.163 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.162+0000 7f296afb9640 1 --2- 172.21.15.23:0/3913418248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2964102030 0x7f2964198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:49.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.162+0000 7f296afb9640 1 --2- 172.21.15.23:0/3913418248 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f29540a2a10 0x7f29540a4ed0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:49.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.162+0000 7f296afb9640 1 --2- 172.21.15.23:0/3913418248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2964101670 0x7f2964197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:25:49.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.162+0000 7f296afb9640 1 -- 172.21.15.23:0/3913418248 >> 172.21.15.23:0/3913418248 conn(0x7f29640fd180 msgr2=0x7f29640fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:25:49.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.162+0000 7f296afb9640 1 -- 172.21.15.23:0/3913418248 shutdown_connections 2024-03-28T05:25:49.164 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:25:49.162+0000 7f296afb9640 1 -- 172.21.15.23:0/3913418248 wait complete. 2024-03-28T05:25:50.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:49 smithi023 ceph-mon[31954]: pgmap v70: 129 pgs: 129 active+clean; 40 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 2.0 MiB/s wr, 383 op/s 2024-03-28T05:25:50.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:49 smithi098 ceph-mon[36087]: pgmap v70: 129 pgs: 129 active+clean; 40 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 2.0 MiB/s wr, 383 op/s 2024-03-28T05:25:50.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:49 smithi113 ceph-mon[37164]: pgmap v70: 129 pgs: 129 active+clean; 40 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 2.0 MiB/s wr, 383 op/s 2024-03-28T05:25:52.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:51 smithi023 ceph-mon[31954]: pgmap v71: 129 pgs: 129 active+clean; 33 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 2.2 MiB/s wr, 614 op/s 2024-03-28T05:25:52.044 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:51 smithi113 ceph-mon[37164]: pgmap v71: 129 pgs: 129 active+clean; 33 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 2.2 MiB/s wr, 614 op/s 2024-03-28T05:25:52.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:51 smithi098 ceph-mon[36087]: pgmap v71: 129 pgs: 129 active+clean; 33 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 2.2 MiB/s wr, 614 op/s 2024-03-28T05:25:54.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:53 smithi023 ceph-mon[31954]: pgmap v72: 129 pgs: 129 active+clean; 33 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 475 KiB/s wr, 279 op/s 2024-03-28T05:25:54.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:53 smithi098 ceph-mon[36087]: pgmap v72: 129 pgs: 129 active+clean; 33 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 475 KiB/s wr, 279 op/s 2024-03-28T05:25:54.242 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:53 smithi113 ceph-mon[37164]: pgmap v72: 129 pgs: 129 active+clean; 33 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 475 KiB/s wr, 279 op/s 2024-03-28T05:25:56.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:55 smithi023 ceph-mon[31954]: pgmap v73: 129 pgs: 129 active+clean; 33 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 475 KiB/s wr, 279 op/s 2024-03-28T05:25:56.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:55 smithi098 ceph-mon[36087]: pgmap v73: 129 pgs: 129 active+clean; 33 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 475 KiB/s wr, 279 op/s 2024-03-28T05:25:56.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:55 smithi113 ceph-mon[37164]: pgmap v73: 129 pgs: 129 active+clean; 33 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 475 KiB/s wr, 279 op/s 2024-03-28T05:25:56.399 INFO:tasks.workunit.client.0.smithi023.stdout: 13 73622 95 49177 190 27780 404 2024-03-28T05:25:58.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:57 smithi098 ceph-mon[36087]: pgmap v74: 129 pgs: 129 active+clean; 35 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 734 KiB/s wr, 308 op/s 2024-03-28T05:25:58.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:57 smithi113 ceph-mon[37164]: pgmap v74: 129 pgs: 129 active+clean; 35 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 734 KiB/s wr, 308 op/s 2024-03-28T05:25:58.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:57 smithi023 ceph-mon[31954]: pgmap v74: 129 pgs: 129 active+clean; 35 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 734 KiB/s wr, 308 op/s 2024-03-28T05:26:00.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:25:59 smithi098 ceph-mon[36087]: pgmap v75: 129 pgs: 129 active+clean; 35 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 505 KiB/s wr, 259 op/s 2024-03-28T05:26:00.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:25:59 smithi113 ceph-mon[37164]: pgmap v75: 129 pgs: 129 active+clean; 35 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 505 KiB/s wr, 259 op/s 2024-03-28T05:26:00.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:25:59 smithi023 ceph-mon[31954]: pgmap v75: 129 pgs: 129 active+clean; 35 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 505 KiB/s wr, 259 op/s 2024-03-28T05:26:02.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:01 smithi098 ceph-mon[36087]: pgmap v76: 129 pgs: 129 active+clean; 60 MiB data, 582 MiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 2.6 MiB/s wr, 400 op/s 2024-03-28T05:26:02.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:01 smithi113 ceph-mon[37164]: pgmap v76: 129 pgs: 129 active+clean; 60 MiB data, 582 MiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 2.6 MiB/s wr, 400 op/s 2024-03-28T05:26:02.252 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.252+0000 7f3e02a75640 1 -- 172.21.15.23:0/183785542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3dfc105540 msgr2=0x7f3dfc105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:02.252 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.252+0000 7f3e02a75640 1 --2- 172.21.15.23:0/183785542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3dfc105540 0x7f3dfc105920 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3df0009900 tx=0x7f3df002bc30 comp rx=0 tx=0).stop 2024-03-28T05:26:02.252 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.252+0000 7f3e02a75640 1 -- 172.21.15.23:0/183785542 shutdown_connections 2024-03-28T05:26:02.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.252+0000 7f3e02a75640 1 --2- 172.21.15.23:0/183785542 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3dfc1011b0 0x7f3dfc10d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.252+0000 7f3e02a75640 1 --2- 172.21.15.23:0/183785542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dfc1007f0 0x7f3dfc100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.252+0000 7f3e02a75640 1 --2- 172.21.15.23:0/183785542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3dfc105540 0x7f3dfc105920 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.252+0000 7f3e02a75640 1 -- 172.21.15.23:0/183785542 >> 172.21.15.23:0/183785542 conn(0x7f3dfc0773f0 msgr2=0x7f3dfc077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:02.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.252+0000 7f3e02a75640 1 -- 172.21.15.23:0/183785542 shutdown_connections 2024-03-28T05:26:02.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.252+0000 7f3e02a75640 1 -- 172.21.15.23:0/183785542 wait complete. 2024-03-28T05:26:02.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.253+0000 7f3e02a75640 1 Processor -- start 2024-03-28T05:26:02.253 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.253+0000 7f3e02a75640 1 -- start start 2024-03-28T05:26:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.253+0000 7f3e02a75640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dfc1007f0 0x7f3dfc102210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.253+0000 7f3e02a75640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3dfc1011b0 0x7f3dfc102750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.253+0000 7f3e02a75640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3dfc105540 0x7f3dfc102c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.253+0000 7f3e02a75640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc110ab0 con 0x7f3dfc105540 2024-03-28T05:26:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.253+0000 7f3e02a75640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc110930 con 0x7f3dfc1007f0 2024-03-28T05:26:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.253+0000 7f3e02a75640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc110c30 con 0x7f3dfc1011b0 2024-03-28T05:26:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.253+0000 7f3e01012640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3dfc105540 0x7f3dfc102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.253+0000 7f3e01012640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3dfc105540 0x7f3dfc102c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52486/0 (socket says 172.21.15.23:52486) 2024-03-28T05:26:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.253+0000 7f3e01012640 1 -- 172.21.15.23:0/1835732650 learned_addr learned my addr 172.21.15.23:0/1835732650 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.253+0000 7f3dfbfff640 1 --2- 172.21.15.23:0/1835732650 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3dfc1011b0 0x7f3dfc102750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.253+0000 7f3e00811640 1 --2- 172.21.15.23:0/1835732650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dfc1007f0 0x7f3dfc102210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.253+0000 7f3e01012640 1 -- 172.21.15.23:0/1835732650 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3dfc1011b0 msgr2=0x7f3dfc102750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.253+0000 7f3e01012640 1 --2- 172.21.15.23:0/1835732650 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3dfc1011b0 0x7f3dfc102750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.253+0000 7f3e01012640 1 -- 172.21.15.23:0/1835732650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dfc1007f0 msgr2=0x7f3dfc102210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.253+0000 7f3e01012640 1 --2- 172.21.15.23:0/1835732650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dfc1007f0 0x7f3dfc102210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.253+0000 7f3e01012640 1 -- 172.21.15.23:0/1835732650 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dfc1a6420 con 0x7f3dfc105540 2024-03-28T05:26:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.253+0000 7f3e00811640 1 --2- 172.21.15.23:0/1835732650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dfc1007f0 0x7f3dfc102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:26:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.254+0000 7f3e01012640 1 --2- 172.21.15.23:0/1835732650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3dfc105540 0x7f3dfc102c90 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f3df400ea60 tx=0x7f3df400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.254+0000 7f3df9ffb640 1 -- 172.21.15.23:0/1835732650 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df400ce40 con 0x7f3dfc105540 2024-03-28T05:26:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.254+0000 7f3df9ffb640 1 -- 172.21.15.23:0/1835732650 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3df4005020 con 0x7f3dfc105540 2024-03-28T05:26:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.254+0000 7f3e02a75640 1 -- 172.21.15.23:0/1835732650 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dfc1a6710 con 0x7f3dfc105540 2024-03-28T05:26:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.254+0000 7f3df9ffb640 1 -- 172.21.15.23:0/1835732650 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df4010690 con 0x7f3dfc105540 2024-03-28T05:26:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.254+0000 7f3e02a75640 1 -- 172.21.15.23:0/1835732650 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dfc1a6cb0 con 0x7f3dfc105540 2024-03-28T05:26:02.262 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.256+0000 7f3df9ffb640 1 -- 172.21.15.23:0/1835732650 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f3df4005520 con 0x7f3dfc105540 2024-03-28T05:26:02.262 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.256+0000 7f3e0565c740 1 Processor -- start 2024-03-28T05:26:02.262 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.256+0000 7f3e0565c740 1 -- start start 2024-03-28T05:26:02.262 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.256+0000 7f3df9ffb640 1 --2- 172.21.15.23:0/1835732650 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f3dec0a2a90 0x7f3dec0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:02.262 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3e0565c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f45710 0x55a0b5f45af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:02.262 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3e0565c740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a0b5f46030 0x55a0b5f4e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:02.262 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3df9ffb640 1 -- 172.21.15.23:0/1835732650 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f3df40cf5c0 con 0x7f3dfc105540 2024-03-28T05:26:02.262 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3dfbfff640 1 --2- 172.21.15.23:0/1835732650 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f3dec0a2a90 0x7f3dec0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3e0565c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f4ee00 0x55a0b5f511f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3e0565c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b5ea9b10 con 0x55a0b5f45710 2024-03-28T05:26:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3e0565c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b5e52900 con 0x55a0b5f4ee00 2024-03-28T05:26:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3e0565c740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b5ea5bb0 con 0x55a0b5f46030 2024-03-28T05:26:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3e00811640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f45710 0x55a0b5f45af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3e00811640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f45710 0x55a0b5f45af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52488/0 (socket says 172.21.15.23:52488) 2024-03-28T05:26:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3e00811640 1 -- 172.21.15.23:0/2327505295 learned_addr learned my addr 172.21.15.23:0/2327505295 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3dfbfff640 1 --2- 172.21.15.23:0/2327505295 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a0b5f46030 0x55a0b5f4e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3e01012640 1 --2- 172.21.15.23:0/2327505295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f4ee00 0x55a0b5f511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3e00811640 1 -- 172.21.15.23:0/2327505295 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a0b5f46030 msgr2=0x55a0b5f4e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3e00811640 1 --2- 172.21.15.23:0/2327505295 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a0b5f46030 0x55a0b5f4e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3e00811640 1 -- 172.21.15.23:0/2327505295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f4ee00 msgr2=0x55a0b5f511f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3e00811640 1 --2- 172.21.15.23:0/2327505295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f4ee00 0x55a0b5f511f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3e00811640 1 -- 172.21.15.23:0/2327505295 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0b5bbe210 con 0x55a0b5f45710 2024-03-28T05:26:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3e01012640 1 --2- 172.21.15.23:0/2327505295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f4ee00 0x55a0b5f511f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:26:02.264 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3dfbfff640 1 --2- 172.21.15.23:0/1835732650 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f3dec0a2a90 0x7f3dec0a4f50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc103ca0 tx=0x7f3de8013880 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:02.264 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3e00811640 1 --2- 172.21.15.23:0/2327505295 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f45710 0x55a0b5f45af0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3df0009230 tx=0x7f3df0009670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a145d39b5a25ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:02.264 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3de6ffd640 1 -- 172.21.15.23:0/2327505295 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df0042070 con 0x55a0b5f45710 2024-03-28T05:26:02.264 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3de6ffd640 1 -- 172.21.15.23:0/2327505295 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3df0004a70 con 0x55a0b5f45710 2024-03-28T05:26:02.264 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.257+0000 7f3de6ffd640 1 -- 172.21.15.23:0/2327505295 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df0005180 con 0x55a0b5f45710 2024-03-28T05:26:02.264 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.258+0000 7f3e0565c740 1 -- 172.21.15.23:0/2327505295 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f45710 msgr2=0x55a0b5f45af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:02.264 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.258+0000 7f3e0565c740 1 --2- 172.21.15.23:0/2327505295 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f45710 0x55a0b5f45af0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3df0009230 tx=0x7f3df0009670 comp rx=0 tx=0).stop 2024-03-28T05:26:02.264 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.258+0000 7f3e0565c740 1 -- 172.21.15.23:0/2327505295 shutdown_connections 2024-03-28T05:26:02.264 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.258+0000 7f3e0565c740 1 --2- 172.21.15.23:0/2327505295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f4ee00 0x55a0b5f511f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.264 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.258+0000 7f3e0565c740 1 --2- 172.21.15.23:0/2327505295 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a0b5f46030 0x55a0b5f4e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.264 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.258+0000 7f3e0565c740 1 --2- 172.21.15.23:0/2327505295 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f45710 0x55a0b5f45af0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.264 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.258+0000 7f3e0565c740 1 -- 172.21.15.23:0/2327505295 >> 172.21.15.23:0/2327505295 conn(0x55a0b5f40030 msgr2=0x55a0b5f40410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:02.264 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.258+0000 7f3e0565c740 1 -- 172.21.15.23:0/2327505295 shutdown_connections 2024-03-28T05:26:02.264 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.258+0000 7f3e0565c740 1 -- 172.21.15.23:0/2327505295 wait complete. 2024-03-28T05:26:02.264 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.258+0000 7f3e0565c740 1 -- start start 2024-03-28T05:26:02.264 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.259+0000 7f3e0565c740 1 Processor -- start 2024-03-28T05:26:02.265 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.259+0000 7f3e0565c740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a0b5f45710 0x55a0b5f4be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:02.265 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.259+0000 7f3e0565c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f46030 0x55a0b5f4c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:02.265 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.259+0000 7f3e0565c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f4ee00 0x55a0b5f4c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:02.265 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.259+0000 7f3e0565c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b5f519d0 con 0x55a0b5f4ee00 2024-03-28T05:26:02.265 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.259+0000 7f3e0565c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b5e65d50 con 0x55a0b5f46030 2024-03-28T05:26:02.265 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.259+0000 7f3e0565c740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b5f51b50 con 0x55a0b5f45710 2024-03-28T05:26:02.265 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.259+0000 7f3e0565c740 10 client.0 _ll_register_callbacks cb 0x55a0b5e65720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:26:02.265 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.259+0000 7f3e01012640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f4ee00 0x55a0b5f4c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:02.265 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.259+0000 7f3e00811640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a0b5f45710 0x55a0b5f4be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:02.265 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.259+0000 7f3e01012640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f4ee00 0x55a0b5f4c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52494/0 (socket says 172.21.15.23:52494) 2024-03-28T05:26:02.265 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.259+0000 7f3dfbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f46030 0x55a0b5f4c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:02.265 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.259+0000 7f3e01012640 1 -- 172.21.15.23:0/3925407310 learned_addr learned my addr 172.21.15.23:0/3925407310 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:02.265 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.259+0000 7f3e00811640 1 -- 172.21.15.23:0/3925407310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f46030 msgr2=0x55a0b5f4c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:02.265 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.259+0000 7f3e00811640 1 --2- 172.21.15.23:0/3925407310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f46030 0x55a0b5f4c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.265 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.259+0000 7f3e00811640 1 -- 172.21.15.23:0/3925407310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f4ee00 msgr2=0x55a0b5f4c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.259+0000 7f3e00811640 1 --2- 172.21.15.23:0/3925407310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f4ee00 0x55a0b5f4c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.259+0000 7f3e00811640 1 -- 172.21.15.23:0/3925407310 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a0b5c8a620 con 0x55a0b5f45710 2024-03-28T05:26:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.259+0000 7f3dfbfff640 1 --2- 172.21.15.23:0/3925407310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f46030 0x55a0b5f4c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:26:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.259+0000 7f3e01012640 1 --2- 172.21.15.23:0/3925407310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f4ee00 0x55a0b5f4c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:26:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.260+0000 7f3e00811640 1 --2- 172.21.15.23:0/3925407310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a0b5f45710 0x55a0b5f4be60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3df0006010 tx=0x7f3df002c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3dbd2402a21ab47f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.260+0000 7f3ddbfff640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-28T05:26:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.260+0000 7f3ddbfff640 1 -- 172.21.15.23:0/3925407310 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df004b070 con 0x55a0b5f45710 2024-03-28T05:26:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.260+0000 7f3e0565c740 1 -- 172.21.15.23:0/3925407310 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3df0004a70 con 0x55a0b5f45710 2024-03-28T05:26:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.260+0000 7f3ddbfff640 1 -- 172.21.15.23:0/3925407310 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f3df0004450 con 0x55a0b5f45710 2024-03-28T05:26:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.260+0000 7f3ddbfff640 1 -- 172.21.15.23:0/3925407310 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3df0057050 con 0x55a0b5f45710 2024-03-28T05:26:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.260+0000 7f3e0565c740 10 client.25510 fetch_fsmap learned FSMap version 20 2024-03-28T05:26:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.260+0000 7f3e0565c740 1 -- 172.21.15.23:0/3925407310 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55a0b5f519d0 con 0x55a0b5f45710 2024-03-28T05:26:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.260+0000 7f3ddbfff640 1 -- 172.21.15.23:0/3925407310 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3df0004a70 con 0x55a0b5f45710 2024-03-28T05:26:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.260+0000 7f3ddbfff640 1 -- 172.21.15.23:0/3925407310 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f3df0003b90 con 0x55a0b5f45710 2024-03-28T05:26:02.267 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.260+0000 7f3e0565c740 10 client.25510 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:26:02.267 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.260+0000 7f3e0565c740 10 client.25510 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:26:02.267 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.260+0000 7f3e0565c740 20 client.25510 populate_metadata read hostname 'smithi023' 2024-03-28T05:26:02.267 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.260+0000 7f3e0565c740 1 --2- 172.21.15.23:0/3925407310 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a0b5f5ede0 0x55a0b5f611d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:02.267 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.260+0000 7f3e0565c740 4 client.25510 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:26:02.267 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.260+0000 7f3e0565c740 1 -- 172.21.15.23:0/3925407310 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a0b5eb7050 con 0x55a0b5f5ede0 2024-03-28T05:26:02.267 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.261+0000 7f3dfbfff640 1 --2- 172.21.15.23:0/3925407310 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a0b5f5ede0 0x55a0b5f611d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:02.267 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.261+0000 7f3dfbfff640 1 --2- 172.21.15.23:0/3925407310 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a0b5f5ede0 0x55a0b5f611d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fdb6a641cdea14b server_cookie=85f785f2e31a248e in_seq=0 out_seq=0 2024-03-28T05:26:02.267 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.261+0000 7f3ddbfff640 10 client.25510 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:26:02.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.263+0000 7f3ddbfff640 1 -- 172.21.15.23:0/3925407310 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a0b5eb7050 con 0x55a0b5f5ede0 2024-03-28T05:26:02.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.263+0000 7f3ddbfff640 10 client.25510 handle_command_reply: tid=0 2024-03-28T05:26:02.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.263+0000 7f3e0565c740 1 client.25510 shutdown 2024-03-28T05:26:02.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.263+0000 7f3e0565c740 1 -- 172.21.15.23:0/3925407310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a0b5f45710 msgr2=0x55a0b5f4be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:02.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.263+0000 7f3e0565c740 1 --2- 172.21.15.23:0/3925407310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a0b5f45710 0x55a0b5f4be60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3df0006010 tx=0x7f3df002c2d0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.263+0000 7f3e0565c740 1 -- 172.21.15.23:0/3925407310 shutdown_connections 2024-03-28T05:26:02.268 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.263+0000 7f3e0565c740 1 --2- 172.21.15.23:0/3925407310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f4ee00 0x55a0b5f4c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.263+0000 7f3e0565c740 1 --2- 172.21.15.23:0/3925407310 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a0b5f5ede0 0x55a0b5f611d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.263+0000 7f3e0565c740 1 --2- 172.21.15.23:0/3925407310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f46030 0x55a0b5f4c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.263+0000 7f3e0565c740 1 --2- 172.21.15.23:0/3925407310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a0b5f45710 0x55a0b5f4be60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.263+0000 7f3ddbfff640 0 client.25510 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:26:02.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.263+0000 7f3e0565c740 1 -- 172.21.15.23:0/3925407310 >> 172.21.15.23:0/3925407310 conn(0x55a0b5f40030 msgr2=0x55a0b5f41d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:02.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.263+0000 7f3e0565c740 1 -- 172.21.15.23:0/3925407310 shutdown_connections 2024-03-28T05:26:02.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.263+0000 7f3e0565c740 1 -- 172.21.15.23:0/3925407310 wait complete. 2024-03-28T05:26:02.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.263+0000 7f3e0565c740 20 client.25510 trim_cache size 0 max 16384 2024-03-28T05:26:02.281 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3e0565c740 1 Processor -- start 2024-03-28T05:26:02.281 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3e0565c740 1 -- start start 2024-03-28T05:26:02.281 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3e0565c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f45710 0x55a0b5f4e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:02.281 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3e0565c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f46030 0x55a0b5f471c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:02.282 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3e0565c740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a0b5f4ee00 0x55a0b5f5a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:02.282 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3e0565c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc1aec10 con 0x55a0b5f46030 2024-03-28T05:26:02.282 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3e0565c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc1b11d0 con 0x55a0b5f45710 2024-03-28T05:26:02.282 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3e0565c740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc1af210 con 0x55a0b5f4ee00 2024-03-28T05:26:02.282 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3dfbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f46030 0x55a0b5f471c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:02.282 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3dfbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f46030 0x55a0b5f471c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52500/0 (socket says 172.21.15.23:52500) 2024-03-28T05:26:02.282 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3dfbfff640 1 -- 172.21.15.23:0/278321455 learned_addr learned my addr 172.21.15.23:0/278321455 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:02.282 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3e01012640 1 --2- 172.21.15.23:0/278321455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a0b5f4ee00 0x55a0b5f5a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:02.282 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3e00811640 1 --2- 172.21.15.23:0/278321455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f45710 0x55a0b5f4e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:02.282 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3dfbfff640 1 -- 172.21.15.23:0/278321455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a0b5f4ee00 msgr2=0x55a0b5f5a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:02.282 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3dfbfff640 1 --2- 172.21.15.23:0/278321455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a0b5f4ee00 0x55a0b5f5a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.282 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3dfbfff640 1 -- 172.21.15.23:0/278321455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f45710 msgr2=0x55a0b5f4e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:02.282 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3dfbfff640 1 --2- 172.21.15.23:0/278321455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f45710 0x55a0b5f4e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.283 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3dfbfff640 1 -- 172.21.15.23:0/278321455 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0b5f4a3b0 con 0x55a0b5f46030 2024-03-28T05:26:02.283 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3e01012640 1 --2- 172.21.15.23:0/278321455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a0b5f4ee00 0x55a0b5f5a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:26:02.283 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3e00811640 1 --2- 172.21.15.23:0/278321455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f45710 0x55a0b5f4e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:26:02.283 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3dfbfff640 1 --2- 172.21.15.23:0/278321455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f46030 0x55a0b5f471c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3de800f750 tx=0x7f3de8020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0ddb03cd1478f5e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:02.283 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3de67fc640 1 -- 172.21.15.23:0/278321455 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de8027040 con 0x55a0b5f46030 2024-03-28T05:26:02.283 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3de67fc640 1 -- 172.21.15.23:0/278321455 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3de800fd10 con 0x55a0b5f46030 2024-03-28T05:26:02.283 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.278+0000 7f3de67fc640 1 -- 172.21.15.23:0/278321455 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de802d1b0 con 0x55a0b5f46030 2024-03-28T05:26:02.283 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.279+0000 7f3e0565c740 1 -- 172.21.15.23:0/278321455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f46030 msgr2=0x55a0b5f471c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:02.283 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.279+0000 7f3e0565c740 1 --2- 172.21.15.23:0/278321455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f46030 0x55a0b5f471c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3de800f750 tx=0x7f3de8020000 comp rx=0 tx=0).stop 2024-03-28T05:26:02.283 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.279+0000 7f3e0565c740 1 -- 172.21.15.23:0/278321455 shutdown_connections 2024-03-28T05:26:02.283 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.279+0000 7f3e0565c740 1 --2- 172.21.15.23:0/278321455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a0b5f4ee00 0x55a0b5f5a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.283 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.279+0000 7f3e0565c740 1 --2- 172.21.15.23:0/278321455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f46030 0x55a0b5f471c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.283 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.279+0000 7f3e0565c740 1 --2- 172.21.15.23:0/278321455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f45710 0x55a0b5f4e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.283 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.279+0000 7f3e0565c740 1 -- 172.21.15.23:0/278321455 >> 172.21.15.23:0/278321455 conn(0x55a0b5f40030 msgr2=0x55a0b5f40410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:02.283 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.279+0000 7f3e0565c740 1 -- 172.21.15.23:0/278321455 shutdown_connections 2024-03-28T05:26:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.279+0000 7f3e0565c740 1 -- 172.21.15.23:0/278321455 wait complete. 2024-03-28T05:26:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.279+0000 7f3e0565c740 1 -- start start 2024-03-28T05:26:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.279+0000 7f3e0565c740 1 Processor -- start 2024-03-28T05:26:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.279+0000 7f3e0565c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f45710 0x55a0b5f49780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.279+0000 7f3e0565c740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a0b5f46030 0x55a0b5f480d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.279+0000 7f3e0565c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f4ee00 0x55a0b5f48610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.279+0000 7f3e0565c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc1af850 con 0x55a0b5f4ee00 2024-03-28T05:26:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.279+0000 7f3e0565c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc1b0b70 con 0x55a0b5f45710 2024-03-28T05:26:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.279+0000 7f3e0565c740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc1afeb0 con 0x55a0b5f46030 2024-03-28T05:26:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.279+0000 7f3e0565c740 10 client.0 _ll_register_callbacks cb 0x7f3df0005850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:26:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.279+0000 7f3e01012640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f4ee00 0x55a0b5f48610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:02.284 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.280+0000 7f3e01012640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f4ee00 0x55a0b5f48610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52516/0 (socket says 172.21.15.23:52516) 2024-03-28T05:26:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.280+0000 7f3e01012640 1 -- 172.21.15.23:0/216017389 learned_addr learned my addr 172.21.15.23:0/216017389 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.280+0000 7f3dfbfff640 1 --2- 172.21.15.23:0/216017389 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a0b5f46030 0x55a0b5f480d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.280+0000 7f3e00811640 1 --2- 172.21.15.23:0/216017389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f45710 0x55a0b5f49780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.280+0000 7f3e01012640 1 -- 172.21.15.23:0/216017389 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a0b5f46030 msgr2=0x55a0b5f480d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.280+0000 7f3e01012640 1 --2- 172.21.15.23:0/216017389 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a0b5f46030 0x55a0b5f480d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.280+0000 7f3e01012640 1 -- 172.21.15.23:0/216017389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f45710 msgr2=0x55a0b5f49780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.280+0000 7f3e01012640 1 --2- 172.21.15.23:0/216017389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f45710 0x55a0b5f49780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.280+0000 7f3e01012640 1 -- 172.21.15.23:0/216017389 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a0b5f48b50 con 0x55a0b5f4ee00 2024-03-28T05:26:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.280+0000 7f3e01012640 1 --2- 172.21.15.23:0/216017389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f4ee00 0x55a0b5f48610 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3df400c760 tx=0x7f3df409bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b73a57a951bbdca2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.280+0000 7f3ddbfff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:26:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.280+0000 7f3ddbfff640 1 -- 172.21.15.23:0/216017389 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df40b9060 con 0x55a0b5f4ee00 2024-03-28T05:26:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.280+0000 7f3ddbfff640 1 -- 172.21.15.23:0/216017389 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f3df40ccba0 con 0x55a0b5f4ee00 2024-03-28T05:26:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.280+0000 7f3e0565c740 1 -- 172.21.15.23:0/216017389 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3de800fd10 con 0x55a0b5f4ee00 2024-03-28T05:26:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.280+0000 7f3ddbfff640 1 -- 172.21.15.23:0/216017389 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3df40a3fd0 con 0x55a0b5f4ee00 2024-03-28T05:26:02.285 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.280+0000 7f3e0565c740 10 client.15681 fetch_fsmap learned FSMap version 20 2024-03-28T05:26:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.280+0000 7f3e0565c740 1 -- 172.21.15.23:0/216017389 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55a0b5fba060 con 0x55a0b5f4ee00 2024-03-28T05:26:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.280+0000 7f3ddbfff640 1 -- 172.21.15.23:0/216017389 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3de800fd10 con 0x55a0b5f4ee00 2024-03-28T05:26:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.281+0000 7f3ddbfff640 1 -- 172.21.15.23:0/216017389 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f3df409ba90 con 0x55a0b5f4ee00 2024-03-28T05:26:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.281+0000 7f3e0565c740 10 client.15681 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:26:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.281+0000 7f3e0565c740 10 client.15681 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:26:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.281+0000 7f3e0565c740 20 client.15681 populate_metadata read hostname 'smithi023' 2024-03-28T05:26:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.281+0000 7f3e0565c740 1 --2- 172.21.15.23:0/216017389 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a0b5fba6e0 0x55a0b5fbcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.281+0000 7f3e0565c740 4 client.15681 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:26:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.281+0000 7f3e0565c740 1 -- 172.21.15.23:0/216017389 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a0b5eb13a0 con 0x55a0b5fba6e0 2024-03-28T05:26:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.281+0000 7f3e00811640 1 --2- 172.21.15.23:0/216017389 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a0b5fba6e0 0x55a0b5fbcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:02.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:01 smithi023 ceph-mon[31954]: pgmap v76: 129 pgs: 129 active+clean; 60 MiB data, 582 MiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 2.6 MiB/s wr, 400 op/s 2024-03-28T05:26:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.281+0000 7f3e00811640 1 --2- 172.21.15.23:0/216017389 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a0b5fba6e0 0x55a0b5fbcb00 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a20c396018e76929 server_cookie=84e5b6c3f6ef7485 in_seq=0 out_seq=0 2024-03-28T05:26:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.281+0000 7f3ddbfff640 10 client.15681 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:26:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3ddbfff640 1 -- 172.21.15.23:0/216017389 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a0b5eb13a0 con 0x55a0b5fba6e0 2024-03-28T05:26:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3ddbfff640 10 client.15681 handle_command_reply: tid=0 2024-03-28T05:26:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e0565c740 1 client.15681 shutdown 2024-03-28T05:26:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e0565c740 1 -- 172.21.15.23:0/216017389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f4ee00 msgr2=0x55a0b5f48610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e0565c740 1 --2- 172.21.15.23:0/216017389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f4ee00 0x55a0b5f48610 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3df400c760 tx=0x7f3df409bf30 comp rx=0 tx=0).stop 2024-03-28T05:26:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e0565c740 1 -- 172.21.15.23:0/216017389 shutdown_connections 2024-03-28T05:26:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e0565c740 1 --2- 172.21.15.23:0/216017389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a0b5f4ee00 0x55a0b5f48610 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e0565c740 1 --2- 172.21.15.23:0/216017389 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a0b5f46030 0x55a0b5f480d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e0565c740 1 --2- 172.21.15.23:0/216017389 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55a0b5fba6e0 0x55a0b5fbcb00 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e0565c740 1 --2- 172.21.15.23:0/216017389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0b5f45710 0x55a0b5f49780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3ddbfff640 0 client.15681 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:26:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e0565c740 1 -- 172.21.15.23:0/216017389 >> 172.21.15.23:0/216017389 conn(0x55a0b5f40030 msgr2=0x55a0b5f471c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e0565c740 1 -- 172.21.15.23:0/216017389 shutdown_connections 2024-03-28T05:26:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e0565c740 1 -- 172.21.15.23:0/216017389 wait complete. 2024-03-28T05:26:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e0565c740 20 client.15681 trim_cache size 0 max 16384 2024-03-28T05:26:02.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:26:02.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:26:02.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:26:02.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:26:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e02a75640 1 -- 172.21.15.23:0/1835732650 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f3dec0a2a90 msgr2=0x7f3dec0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e02a75640 1 --2- 172.21.15.23:0/1835732650 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f3dec0a2a90 0x7f3dec0a4f50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc103ca0 tx=0x7f3de8013880 comp rx=0 tx=0).stop 2024-03-28T05:26:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e02a75640 1 -- 172.21.15.23:0/1835732650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3dfc105540 msgr2=0x7f3dfc102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e02a75640 1 --2- 172.21.15.23:0/1835732650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3dfc105540 0x7f3dfc102c90 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f3df400ea60 tx=0x7f3df400ef20 comp rx=0 tx=0).stop 2024-03-28T05:26:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e02a75640 1 -- 172.21.15.23:0/1835732650 shutdown_connections 2024-03-28T05:26:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e02a75640 1 --2- 172.21.15.23:0/1835732650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3dfc105540 0x7f3dfc102c90 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e02a75640 1 --2- 172.21.15.23:0/1835732650 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3dfc1011b0 0x7f3dfc102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e02a75640 1 --2- 172.21.15.23:0/1835732650 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f3dec0a2a90 0x7f3dec0a4f50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e02a75640 1 --2- 172.21.15.23:0/1835732650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dfc1007f0 0x7f3dfc102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e02a75640 1 -- 172.21.15.23:0/1835732650 >> 172.21.15.23:0/1835732650 conn(0x7f3dfc0773f0 msgr2=0x7f3dfc068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e02a75640 1 -- 172.21.15.23:0/1835732650 shutdown_connections 2024-03-28T05:26:02.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:02.725+0000 7f3e02a75640 1 -- 172.21.15.23:0/1835732650 wait complete. 2024-03-28T05:26:04.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:03 smithi098 ceph-mon[36087]: pgmap v77: 129 pgs: 129 active+clean; 60 MiB data, 582 MiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 2.4 MiB/s wr, 169 op/s 2024-03-28T05:26:04.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:03 smithi113 ceph-mon[37164]: pgmap v77: 129 pgs: 129 active+clean; 60 MiB data, 582 MiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 2.4 MiB/s wr, 169 op/s 2024-03-28T05:26:04.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:03 smithi023 ceph-mon[31954]: pgmap v77: 129 pgs: 129 active+clean; 60 MiB data, 582 MiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 2.4 MiB/s wr, 169 op/s 2024-03-28T05:26:06.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:05 smithi098 ceph-mon[36087]: pgmap v78: 129 pgs: 129 active+clean; 60 MiB data, 582 MiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 2.4 MiB/s wr, 169 op/s 2024-03-28T05:26:06.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:05 smithi113 ceph-mon[37164]: pgmap v78: 129 pgs: 129 active+clean; 60 MiB data, 582 MiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 2.4 MiB/s wr, 169 op/s 2024-03-28T05:26:06.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:05 smithi023 ceph-mon[31954]: pgmap v78: 129 pgs: 129 active+clean; 60 MiB data, 582 MiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 2.4 MiB/s wr, 169 op/s 2024-03-28T05:26:06.399 INFO:tasks.workunit.client.0.smithi023.stdout: 15 166708 181 117244 133 81038 333 2024-03-28T05:26:08.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:07 smithi098 ceph-mon[36087]: pgmap v79: 129 pgs: 129 active+clean; 54 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 532 op/s 2024-03-28T05:26:08.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:07 smithi113 ceph-mon[37164]: pgmap v79: 129 pgs: 129 active+clean; 54 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 532 op/s 2024-03-28T05:26:08.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:07 smithi023 ceph-mon[31954]: pgmap v79: 129 pgs: 129 active+clean; 54 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 532 op/s 2024-03-28T05:26:10.083 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:09 smithi023 ceph-mon[31954]: pgmap v80: 129 pgs: 129 active+clean; 54 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 2.6 MiB/s wr, 504 op/s 2024-03-28T05:26:10.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:09 smithi098 ceph-mon[36087]: pgmap v80: 129 pgs: 129 active+clean; 54 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 2.6 MiB/s wr, 504 op/s 2024-03-28T05:26:10.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:09 smithi113 ceph-mon[37164]: pgmap v80: 129 pgs: 129 active+clean; 54 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 2.6 MiB/s wr, 504 op/s 2024-03-28T05:26:12.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:11 smithi098 ceph-mon[36087]: pgmap v81: 129 pgs: 129 active+clean; 56 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 550 op/s 2024-03-28T05:26:12.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:11 smithi113 ceph-mon[37164]: pgmap v81: 129 pgs: 129 active+clean; 56 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 550 op/s 2024-03-28T05:26:12.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:11 smithi023 ceph-mon[31954]: pgmap v81: 129 pgs: 129 active+clean; 56 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 550 op/s 2024-03-28T05:26:14.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:13 smithi098 ceph-mon[36087]: pgmap v82: 129 pgs: 129 active+clean; 56 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 698 KiB/s wr, 409 op/s 2024-03-28T05:26:14.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:13 smithi113 ceph-mon[37164]: pgmap v82: 129 pgs: 129 active+clean; 56 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 698 KiB/s wr, 409 op/s 2024-03-28T05:26:14.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:13 smithi023 ceph-mon[31954]: pgmap v82: 129 pgs: 129 active+clean; 56 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 698 KiB/s wr, 409 op/s 2024-03-28T05:26:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.818+0000 7f30fd2a7640 1 -- 172.21.15.23:0/331340325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30f81011b0 msgr2=0x7f30f810d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.818+0000 7f30fd2a7640 1 --2- 172.21.15.23:0/331340325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30f81011b0 0x7f30f810d3d0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f30e0009900 tx=0x7f30e002bc30 comp rx=0 tx=0).stop 2024-03-28T05:26:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.818+0000 7f30fd2a7640 1 -- 172.21.15.23:0/331340325 shutdown_connections 2024-03-28T05:26:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.818+0000 7f30fd2a7640 1 --2- 172.21.15.23:0/331340325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30f81011b0 0x7f30f810d3d0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.818+0000 7f30fd2a7640 1 --2- 172.21.15.23:0/331340325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30f81007f0 0x7f30f8100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.818+0000 7f30fd2a7640 1 --2- 172.21.15.23:0/331340325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30f8105540 0x7f30f8105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.818+0000 7f30fd2a7640 1 -- 172.21.15.23:0/331340325 >> 172.21.15.23:0/331340325 conn(0x7f30f80773f0 msgr2=0x7f30f8077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.818+0000 7f30fd2a7640 1 -- 172.21.15.23:0/331340325 shutdown_connections 2024-03-28T05:26:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.818+0000 7f30fd2a7640 1 -- 172.21.15.23:0/331340325 wait complete. 2024-03-28T05:26:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.819+0000 7f30fd2a7640 1 Processor -- start 2024-03-28T05:26:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.819+0000 7f30fd2a7640 1 -- start start 2024-03-28T05:26:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.819+0000 7f30fd2a7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30f81007f0 0x7f30f8197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.819+0000 7f30fd2a7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30f81011b0 0x7f30f8198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.819+0000 7f30fd2a7640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30f8105540 0x7f30f819c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.819+0000 7f30fd2a7640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f8110ab0 con 0x7f30f81011b0 2024-03-28T05:26:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.819+0000 7f30fd2a7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f8110930 con 0x7f30f81007f0 2024-03-28T05:26:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.819+0000 7f30fd2a7640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f8110c30 con 0x7f30f8105540 2024-03-28T05:26:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.820+0000 7f30f67fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30f81011b0 0x7f30f8198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.820+0000 7f30f67fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30f81011b0 0x7f30f8198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52660/0 (socket says 172.21.15.23:52660) 2024-03-28T05:26:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.820+0000 7f30f67fc640 1 -- 172.21.15.23:0/3781753193 learned_addr learned my addr 172.21.15.23:0/3781753193 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.820+0000 7f30f6ffd640 1 --2- 172.21.15.23:0/3781753193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30f81007f0 0x7f30f8197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.820+0000 7f30f77fe640 1 --2- 172.21.15.23:0/3781753193 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30f8105540 0x7f30f819c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.820+0000 7f30f67fc640 1 -- 172.21.15.23:0/3781753193 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30f8105540 msgr2=0x7f30f819c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.820+0000 7f30f67fc640 1 --2- 172.21.15.23:0/3781753193 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30f8105540 0x7f30f819c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.820+0000 7f30f67fc640 1 -- 172.21.15.23:0/3781753193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30f81007f0 msgr2=0x7f30f8197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.820+0000 7f30f67fc640 1 --2- 172.21.15.23:0/3781753193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30f81007f0 0x7f30f8197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.820+0000 7f30f67fc640 1 -- 172.21.15.23:0/3781753193 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f819ced0 con 0x7f30f81011b0 2024-03-28T05:26:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.820+0000 7f30f6ffd640 1 --2- 172.21.15.23:0/3781753193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30f81007f0 0x7f30f8197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:26:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.820+0000 7f30f67fc640 1 --2- 172.21.15.23:0/3781753193 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30f81011b0 0x7f30f8198420 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f30e800e9f0 tx=0x7f30e800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.820+0000 7f30effff640 1 -- 172.21.15.23:0/3781753193 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e800cd80 con 0x7f30f81011b0 2024-03-28T05:26:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.820+0000 7f30effff640 1 -- 172.21.15.23:0/3781753193 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30e8010910 con 0x7f30f81011b0 2024-03-28T05:26:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.820+0000 7f30fd2a7640 1 -- 172.21.15.23:0/3781753193 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f819d1c0 con 0x7f30f81011b0 2024-03-28T05:26:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.820+0000 7f30effff640 1 -- 172.21.15.23:0/3781753193 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e8018990 con 0x7f30f81011b0 2024-03-28T05:26:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.820+0000 7f30fd2a7640 1 -- 172.21.15.23:0/3781753193 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f81a4a50 con 0x7f30f81011b0 2024-03-28T05:26:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.822+0000 7f30effff640 1 -- 172.21.15.23:0/3781753193 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f30e8010ac0 con 0x7f30f81011b0 2024-03-28T05:26:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.823+0000 7f30fff68740 1 Processor -- start 2024-03-28T05:26:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.823+0000 7f30fff68740 1 -- start start 2024-03-28T05:26:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.823+0000 7f30fff68740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0fbbe6710 0x55c0fbbe6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.823+0000 7f30fff68740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbe7030 0x55c0fbbef8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.823+0000 7f30fff68740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c0fbbefe00 0x55c0fbbf21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.823+0000 7f30fff68740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0fbb4ab10 con 0x55c0fbbe7030 2024-03-28T05:26:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.823+0000 7f30fff68740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0fbaf3900 con 0x55c0fbbe6710 2024-03-28T05:26:15.829 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.823+0000 7f30fff68740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0fbb46bb0 con 0x55c0fbbefe00 2024-03-28T05:26:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.823+0000 7f30effff640 1 --2- 172.21.15.23:0/3781753193 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f30e40a2a10 0x7f30e40a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.823+0000 7f30f6ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbe7030 0x55c0fbbef8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.823+0000 7f30f6ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbe7030 0x55c0fbbef8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52676/0 (socket says 172.21.15.23:52676) 2024-03-28T05:26:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.823+0000 7f30f6ffd640 1 -- 172.21.15.23:0/2335752255 learned_addr learned my addr 172.21.15.23:0/2335752255 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.823+0000 7f30f67fc640 1 --2- 172.21.15.23:0/2335752255 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c0fbbefe00 0x55c0fbbf21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.823+0000 7f30f77fe640 1 --2- 172.21.15.23:0/2335752255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0fbbe6710 0x55c0fbbe6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.823+0000 7f30f77fe640 1 --2- 172.21.15.23:0/3781753193 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f30e40a2a10 0x7f30e40a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.823+0000 7f30effff640 1 -- 172.21.15.23:0/3781753193 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f30e80cf490 con 0x7f30f81011b0 2024-03-28T05:26:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.824+0000 7f30f6ffd640 1 -- 172.21.15.23:0/2335752255 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c0fbbefe00 msgr2=0x55c0fbbf21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.824+0000 7f30f6ffd640 1 --2- 172.21.15.23:0/2335752255 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c0fbbefe00 0x55c0fbbf21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.824+0000 7f30f6ffd640 1 -- 172.21.15.23:0/2335752255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0fbbe6710 msgr2=0x55c0fbbe6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.824+0000 7f30f6ffd640 1 --2- 172.21.15.23:0/2335752255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0fbbe6710 0x55c0fbbe6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.824+0000 7f30f6ffd640 1 -- 172.21.15.23:0/2335752255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c0fb85f210 con 0x55c0fbbe7030 2024-03-28T05:26:15.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.824+0000 7f30f6ffd640 1 --2- 172.21.15.23:0/2335752255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbe7030 0x55c0fbbef8c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f30f0011860 tx=0x7f30f0011c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f9229464b64f7d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.824+0000 7f30f77fe640 1 --2- 172.21.15.23:0/3781753193 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f30e40a2a10 0x7f30e40a4ed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f30e0005ab0 tx=0x7f30e0005a20 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.824+0000 7f30ed7fa640 1 -- 172.21.15.23:0/2335752255 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f0011310 con 0x55c0fbbe7030 2024-03-28T05:26:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.824+0000 7f30ed7fa640 1 -- 172.21.15.23:0/2335752255 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30f0003e50 con 0x55c0fbbe7030 2024-03-28T05:26:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.824+0000 7f30ed7fa640 1 -- 172.21.15.23:0/2335752255 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f0004560 con 0x55c0fbbe7030 2024-03-28T05:26:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.825+0000 7f30fff68740 1 -- 172.21.15.23:0/2335752255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbe7030 msgr2=0x55c0fbbef8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.825+0000 7f30fff68740 1 --2- 172.21.15.23:0/2335752255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbe7030 0x55c0fbbef8c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f30f0011860 tx=0x7f30f0011c60 comp rx=0 tx=0).stop 2024-03-28T05:26:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.825+0000 7f30fff68740 1 -- 172.21.15.23:0/2335752255 shutdown_connections 2024-03-28T05:26:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.825+0000 7f30fff68740 1 --2- 172.21.15.23:0/2335752255 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c0fbbefe00 0x55c0fbbf21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.825+0000 7f30fff68740 1 --2- 172.21.15.23:0/2335752255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbe7030 0x55c0fbbef8c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.825+0000 7f30fff68740 1 --2- 172.21.15.23:0/2335752255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0fbbe6710 0x55c0fbbe6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.825+0000 7f30fff68740 1 -- 172.21.15.23:0/2335752255 >> 172.21.15.23:0/2335752255 conn(0x55c0fbbe1030 msgr2=0x55c0fbbe1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:15.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.825+0000 7f30fff68740 1 -- 172.21.15.23:0/2335752255 shutdown_connections 2024-03-28T05:26:15.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.825+0000 7f30fff68740 1 -- 172.21.15.23:0/2335752255 wait complete. 2024-03-28T05:26:15.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.825+0000 7f30fff68740 1 -- start start 2024-03-28T05:26:15.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.825+0000 7f30fff68740 1 Processor -- start 2024-03-28T05:26:15.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.825+0000 7f30fff68740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c0fbbe6710 0x55c0fbbecde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:15.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.825+0000 7f30fff68740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbe7030 0x55c0fbbed320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:15.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.825+0000 7f30fff68740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0fbbefe00 0x55c0fbbed860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:15.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.825+0000 7f30fff68740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0fbbf2a30 con 0x55c0fbbe7030 2024-03-28T05:26:15.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.825+0000 7f30fff68740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0fbb06d50 con 0x55c0fbbefe00 2024-03-28T05:26:15.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.825+0000 7f30fff68740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0fbbf2bb0 con 0x55c0fbbe6710 2024-03-28T05:26:15.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.825+0000 7f30fff68740 10 client.0 _ll_register_callbacks cb 0x55c0fbb06720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:26:15.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.825+0000 7f30f67fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbe7030 0x55c0fbbed320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:15.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.825+0000 7f30f67fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbe7030 0x55c0fbbed320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52678/0 (socket says 172.21.15.23:52678) 2024-03-28T05:26:15.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.825+0000 7f30f67fc640 1 -- 172.21.15.23:0/2019138106 learned_addr learned my addr 172.21.15.23:0/2019138106 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:15.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.826+0000 7f30f6ffd640 1 --2- 172.21.15.23:0/2019138106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c0fbbe6710 0x55c0fbbecde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:15.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.826+0000 7f30f77fe640 1 --2- 172.21.15.23:0/2019138106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0fbbefe00 0x55c0fbbed860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:15.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.826+0000 7f30f67fc640 1 -- 172.21.15.23:0/2019138106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c0fbbe6710 msgr2=0x55c0fbbecde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:15.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.826+0000 7f30f67fc640 1 --2- 172.21.15.23:0/2019138106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c0fbbe6710 0x55c0fbbecde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.826+0000 7f30f67fc640 1 -- 172.21.15.23:0/2019138106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0fbbefe00 msgr2=0x55c0fbbed860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:15.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.826+0000 7f30f67fc640 1 --2- 172.21.15.23:0/2019138106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0fbbefe00 0x55c0fbbed860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.826+0000 7f30f67fc640 1 -- 172.21.15.23:0/2019138106 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c0fb92b620 con 0x55c0fbbe7030 2024-03-28T05:26:15.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.826+0000 7f30f67fc640 1 --2- 172.21.15.23:0/2019138106 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbe7030 0x55c0fbbed320 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f30e8097d10 tx=0x7f30e806d5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31fb4bcd0a6e86d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:15.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.826+0000 7f30de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:26:15.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.826+0000 7f30de7fc640 1 -- 172.21.15.23:0/2019138106 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e80a3fa0 con 0x55c0fbbe7030 2024-03-28T05:26:15.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.826+0000 7f30de7fc640 1 -- 172.21.15.23:0/2019138106 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f30e8086070 con 0x55c0fbbe7030 2024-03-28T05:26:15.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.826+0000 7f30fff68740 1 -- 172.21.15.23:0/2019138106 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f30f0003e50 con 0x55c0fbbe7030 2024-03-28T05:26:15.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.826+0000 7f30de7fc640 1 -- 172.21.15.23:0/2019138106 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30e80cf120 con 0x55c0fbbe7030 2024-03-28T05:26:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.826+0000 7f30fff68740 10 client.15693 fetch_fsmap learned FSMap version 20 2024-03-28T05:26:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.826+0000 7f30fff68740 1 -- 172.21.15.23:0/2019138106 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55c0fbbfec90 con 0x55c0fbbe7030 2024-03-28T05:26:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.826+0000 7f30de7fc640 1 -- 172.21.15.23:0/2019138106 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30f0003e50 con 0x55c0fbbe7030 2024-03-28T05:26:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.827+0000 7f30de7fc640 1 -- 172.21.15.23:0/2019138106 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f30e8018b30 con 0x55c0fbbe7030 2024-03-28T05:26:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.827+0000 7f30fff68740 10 client.15693 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:26:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.827+0000 7f30fff68740 10 client.15693 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:26:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.827+0000 7f30fff68740 20 client.15693 populate_metadata read hostname 'smithi023' 2024-03-28T05:26:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.827+0000 7f30fff68740 1 --2- 172.21.15.23:0/2019138106 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55c0fbc00150 0x55c0fbc02540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.827+0000 7f30fff68740 4 client.15693 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:26:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.827+0000 7f30fff68740 1 -- 172.21.15.23:0/2019138106 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c0fbb58050 con 0x55c0fbc00150 2024-03-28T05:26:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.827+0000 7f30f6ffd640 1 --2- 172.21.15.23:0/2019138106 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55c0fbc00150 0x55c0fbc02540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.827+0000 7f30de7fc640 10 client.15693 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:26:15.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.827+0000 7f30f6ffd640 1 --2- 172.21.15.23:0/2019138106 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55c0fbc00150 0x55c0fbc02540 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c94b4ea2b9054170 server_cookie=d25aeb26d4a05689 in_seq=0 out_seq=0 2024-03-28T05:26:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.834+0000 7f30de7fc640 1 -- 172.21.15.23:0/2019138106 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c0fbb58050 con 0x55c0fbc00150 2024-03-28T05:26:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.834+0000 7f30de7fc640 10 client.15693 handle_command_reply: tid=0 2024-03-28T05:26:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.834+0000 7f30fff68740 1 client.15693 shutdown 2024-03-28T05:26:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.834+0000 7f30fff68740 1 -- 172.21.15.23:0/2019138106 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbe7030 msgr2=0x55c0fbbed320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.834+0000 7f30fff68740 1 --2- 172.21.15.23:0/2019138106 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbe7030 0x55c0fbbed320 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f30e8097d10 tx=0x7f30e806d5a0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.834+0000 7f30fff68740 1 -- 172.21.15.23:0/2019138106 shutdown_connections 2024-03-28T05:26:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.834+0000 7f30fff68740 1 --2- 172.21.15.23:0/2019138106 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55c0fbc00150 0x55c0fbc02540 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.834+0000 7f30fff68740 1 --2- 172.21.15.23:0/2019138106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0fbbefe00 0x55c0fbbed860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.834+0000 7f30de7fc640 0 client.15693 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:26:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.834+0000 7f30fff68740 1 --2- 172.21.15.23:0/2019138106 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbe7030 0x55c0fbbed320 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.834+0000 7f30fff68740 1 --2- 172.21.15.23:0/2019138106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c0fbbe6710 0x55c0fbbecde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.834+0000 7f30fff68740 1 -- 172.21.15.23:0/2019138106 >> 172.21.15.23:0/2019138106 conn(0x55c0fbbe1030 msgr2=0x55c0fbbe2d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.834+0000 7f30fff68740 1 -- 172.21.15.23:0/2019138106 shutdown_connections 2024-03-28T05:26:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.834+0000 7f30fff68740 1 -- 172.21.15.23:0/2019138106 wait complete. 2024-03-28T05:26:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.834+0000 7f30fff68740 20 client.15693 trim_cache size 0 max 16384 2024-03-28T05:26:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.848+0000 7f30fff68740 1 Processor -- start 2024-03-28T05:26:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.848+0000 7f30fff68740 1 -- start start 2024-03-28T05:26:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.848+0000 7f30fff68740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0fbbe6710 0x55c0fbbef510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.848+0000 7f30fff68740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c0fbbe7030 0x55c0fbbfb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.848+0000 7f30fff68740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbefe00 0x55c0fbbfb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.848+0000 7f30fff68740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f81ac9f0 con 0x55c0fbbefe00 2024-03-28T05:26:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.848+0000 7f30fff68740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f81aefb0 con 0x55c0fbbe6710 2024-03-28T05:26:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.848+0000 7f30fff68740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f81acff0 con 0x55c0fbbe7030 2024-03-28T05:26:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.848+0000 7f30f77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbefe00 0x55c0fbbfb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.848+0000 7f30f77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbefe00 0x55c0fbbfb800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52692/0 (socket says 172.21.15.23:52692) 2024-03-28T05:26:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.848+0000 7f30f77fe640 1 -- 172.21.15.23:0/3354207634 learned_addr learned my addr 172.21.15.23:0/3354207634 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.848+0000 7f30f67fc640 1 --2- 172.21.15.23:0/3354207634 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c0fbbe7030 0x55c0fbbfb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.848+0000 7f30f6ffd640 1 --2- 172.21.15.23:0/3354207634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0fbbe6710 0x55c0fbbef510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.848+0000 7f30f77fe640 1 -- 172.21.15.23:0/3354207634 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c0fbbe7030 msgr2=0x55c0fbbfb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.848+0000 7f30f77fe640 1 --2- 172.21.15.23:0/3354207634 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c0fbbe7030 0x55c0fbbfb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.848+0000 7f30f77fe640 1 -- 172.21.15.23:0/3354207634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0fbbe6710 msgr2=0x55c0fbbef510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.848+0000 7f30f77fe640 1 --2- 172.21.15.23:0/3354207634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0fbbe6710 0x55c0fbbef510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.848+0000 7f30f77fe640 1 -- 172.21.15.23:0/3354207634 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c0fbc02ad0 con 0x55c0fbbefe00 2024-03-28T05:26:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.848+0000 7f30f6ffd640 1 --2- 172.21.15.23:0/3354207634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0fbbe6710 0x55c0fbbef510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:26:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.849+0000 7f30f77fe640 1 --2- 172.21.15.23:0/3354207634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbefe00 0x55c0fbbfb800 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f30e0009900 tx=0x7f30e0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71d29322a82aec78 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.849+0000 7f30ecff9640 1 -- 172.21.15.23:0/3354207634 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e002cc30 con 0x55c0fbbefe00 2024-03-28T05:26:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.849+0000 7f30ecff9640 1 -- 172.21.15.23:0/3354207634 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30e0004ba0 con 0x55c0fbbefe00 2024-03-28T05:26:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.849+0000 7f30ecff9640 1 -- 172.21.15.23:0/3354207634 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e00035f0 con 0x55c0fbbefe00 2024-03-28T05:26:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.849+0000 7f30fff68740 1 -- 172.21.15.23:0/3354207634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbefe00 msgr2=0x55c0fbbfb800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.849+0000 7f30fff68740 1 --2- 172.21.15.23:0/3354207634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbefe00 0x55c0fbbfb800 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f30e0009900 tx=0x7f30e0005fd0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.849+0000 7f30fff68740 1 -- 172.21.15.23:0/3354207634 shutdown_connections 2024-03-28T05:26:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.849+0000 7f30fff68740 1 --2- 172.21.15.23:0/3354207634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbefe00 0x55c0fbbfb800 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.849+0000 7f30fff68740 1 --2- 172.21.15.23:0/3354207634 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c0fbbe7030 0x55c0fbbfb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.849+0000 7f30fff68740 1 --2- 172.21.15.23:0/3354207634 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0fbbe6710 0x55c0fbbef510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.849+0000 7f30fff68740 1 -- 172.21.15.23:0/3354207634 >> 172.21.15.23:0/3354207634 conn(0x55c0fbbe1030 msgr2=0x55c0fbbe7590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.850+0000 7f30fff68740 1 -- 172.21.15.23:0/3354207634 shutdown_connections 2024-03-28T05:26:15.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.850+0000 7f30fff68740 1 -- 172.21.15.23:0/3354207634 wait complete. 2024-03-28T05:26:15.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.850+0000 7f30fff68740 1 -- start start 2024-03-28T05:26:15.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.850+0000 7f30fff68740 1 Processor -- start 2024-03-28T05:26:15.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.850+0000 7f30fff68740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0fbbe6710 0x55c0fbbf33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:15.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.850+0000 7f30fff68740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c0fbbe7030 0x55c0fbbf3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:15.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.850+0000 7f30fff68740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbefe00 0x55c0fbbf3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:15.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.850+0000 7f30fff68740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f81ad630 con 0x55c0fbbefe00 2024-03-28T05:26:15.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.850+0000 7f30fff68740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f81ae950 con 0x55c0fbbe6710 2024-03-28T05:26:15.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.850+0000 7f30fff68740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f81adc90 con 0x55c0fbbe7030 2024-03-28T05:26:15.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.850+0000 7f30fff68740 10 client.0 _ll_register_callbacks cb 0x7f30e809b3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:26:15.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.850+0000 7f30f77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbefe00 0x55c0fbbf3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:15.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.850+0000 7f30f77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbefe00 0x55c0fbbf3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52708/0 (socket says 172.21.15.23:52708) 2024-03-28T05:26:15.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.850+0000 7f30f77fe640 1 -- 172.21.15.23:0/1521219083 learned_addr learned my addr 172.21.15.23:0/1521219083 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:15.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.850+0000 7f30f67fc640 1 --2- 172.21.15.23:0/1521219083 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c0fbbe7030 0x55c0fbbf3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:15.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.850+0000 7f30f6ffd640 1 --2- 172.21.15.23:0/1521219083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0fbbe6710 0x55c0fbbf33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:15.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.850+0000 7f30f77fe640 1 -- 172.21.15.23:0/1521219083 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c0fbbe7030 msgr2=0x55c0fbbf3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:15.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.850+0000 7f30f77fe640 1 --2- 172.21.15.23:0/1521219083 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c0fbbe7030 0x55c0fbbf3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.850+0000 7f30f77fe640 1 -- 172.21.15.23:0/1521219083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0fbbe6710 msgr2=0x55c0fbbf33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:15.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.850+0000 7f30f77fe640 1 --2- 172.21.15.23:0/1521219083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0fbbe6710 0x55c0fbbf33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:15.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.850+0000 7f30f77fe640 1 -- 172.21.15.23:0/1521219083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c0fbc5aad0 con 0x55c0fbbefe00 2024-03-28T05:26:15.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.850+0000 7f30f6ffd640 1 --2- 172.21.15.23:0/1521219083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0fbbe6710 0x55c0fbbf33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:26:15.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.851+0000 7f30f77fe640 1 --2- 172.21.15.23:0/1521219083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbefe00 0x55c0fbbf3e60 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f30e0005c40 tx=0x7f30e0005340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b178722bef4047f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:15.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.851+0000 7f30de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:26:15.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.851+0000 7f30de7fc640 1 -- 172.21.15.23:0/1521219083 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e00046c0 con 0x55c0fbbefe00 2024-03-28T05:26:15.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.851+0000 7f30de7fc640 1 -- 172.21.15.23:0/1521219083 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f30e0004860 con 0x55c0fbbefe00 2024-03-28T05:26:15.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.851+0000 7f30fff68740 1 -- 172.21.15.23:0/1521219083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f30e0004ba0 con 0x55c0fbbefe00 2024-03-28T05:26:15.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.851+0000 7f30de7fc640 1 -- 172.21.15.23:0/1521219083 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30e0067050 con 0x55c0fbbefe00 2024-03-28T05:26:15.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.851+0000 7f30fff68740 10 client.15699 fetch_fsmap learned FSMap version 20 2024-03-28T05:26:15.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.851+0000 7f30fff68740 1 -- 172.21.15.23:0/1521219083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55c0fbc5bd10 con 0x55c0fbbefe00 2024-03-28T05:26:15.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.851+0000 7f30de7fc640 1 -- 172.21.15.23:0/1521219083 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30e0004ba0 con 0x55c0fbbefe00 2024-03-28T05:26:15.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.851+0000 7f30de7fc640 1 -- 172.21.15.23:0/1521219083 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f30e00347d0 con 0x55c0fbbefe00 2024-03-28T05:26:15.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.851+0000 7f30fff68740 10 client.15699 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:26:15.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.851+0000 7f30fff68740 10 client.15699 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:26:15.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.851+0000 7f30fff68740 20 client.15699 populate_metadata read hostname 'smithi023' 2024-03-28T05:26:15.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.851+0000 7f30fff68740 1 --2- 172.21.15.23:0/1521219083 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55c0fbc5c400 0x55c0fbc5e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:15.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.851+0000 7f30fff68740 4 client.15699 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:26:15.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.851+0000 7f30fff68740 1 -- 172.21.15.23:0/1521219083 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c0fbb523a0 con 0x55c0fbc5c400 2024-03-28T05:26:15.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.852+0000 7f30f6ffd640 1 --2- 172.21.15.23:0/1521219083 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55c0fbc5c400 0x55c0fbc5e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:15.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.852+0000 7f30f6ffd640 1 --2- 172.21.15.23:0/1521219083 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55c0fbc5c400 0x55c0fbc5e820 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b840a72858607a9 server_cookie=6078f2b6472af5cf in_seq=0 out_seq=0 2024-03-28T05:26:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:15.852+0000 7f30de7fc640 10 client.15699 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:26:16.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:15 smithi098 ceph-mon[36087]: pgmap v83: 129 pgs: 129 active+clean; 56 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 698 KiB/s wr, 409 op/s 2024-03-28T05:26:16.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:15 smithi113 ceph-mon[37164]: pgmap v83: 129 pgs: 129 active+clean; 56 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 698 KiB/s wr, 409 op/s 2024-03-28T05:26:16.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:15 smithi023 ceph-mon[31954]: pgmap v83: 129 pgs: 129 active+clean; 56 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 698 KiB/s wr, 409 op/s 2024-03-28T05:26:16.399 INFO:tasks.workunit.client.0.smithi023.stdout: 19 85770 246 63186 169 36704 382 2024-03-28T05:26:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.422+0000 7f30de7fc640 1 -- 172.21.15.23:0/1521219083 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c0fbb523a0 con 0x55c0fbc5c400 2024-03-28T05:26:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.422+0000 7f30de7fc640 10 client.15699 handle_command_reply: tid=0 2024-03-28T05:26:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.422+0000 7f30fff68740 1 client.15699 shutdown 2024-03-28T05:26:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.422+0000 7f30fff68740 1 -- 172.21.15.23:0/1521219083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbefe00 msgr2=0x55c0fbbf3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.422+0000 7f30fff68740 1 --2- 172.21.15.23:0/1521219083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbefe00 0x55c0fbbf3e60 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f30e0005c40 tx=0x7f30e0005340 comp rx=0 tx=0).stop 2024-03-28T05:26:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.422+0000 7f30fff68740 1 -- 172.21.15.23:0/1521219083 shutdown_connections 2024-03-28T05:26:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.422+0000 7f30fff68740 1 --2- 172.21.15.23:0/1521219083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0fbbefe00 0x55c0fbbf3e60 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.422+0000 7f30fff68740 1 --2- 172.21.15.23:0/1521219083 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c0fbbe7030 0x55c0fbbf3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.422+0000 7f30fff68740 1 --2- 172.21.15.23:0/1521219083 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55c0fbc5c400 0x55c0fbc5e820 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.422+0000 7f30fff68740 1 --2- 172.21.15.23:0/1521219083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c0fbbe6710 0x55c0fbbf33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.422+0000 7f30de7fc640 0 client.15699 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:26:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.422+0000 7f30fff68740 1 -- 172.21.15.23:0/1521219083 >> 172.21.15.23:0/1521219083 conn(0x55c0fbbe1030 msgr2=0x55c0fbbecda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.422+0000 7f30fff68740 1 -- 172.21.15.23:0/1521219083 shutdown_connections 2024-03-28T05:26:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.422+0000 7f30fff68740 1 -- 172.21.15.23:0/1521219083 wait complete. 2024-03-28T05:26:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.422+0000 7f30fff68740 20 client.15699 trim_cache size 0 max 16384 2024-03-28T05:26:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:26:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:26:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:26:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:26:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.423+0000 7f30fd2a7640 1 -- 172.21.15.23:0/3781753193 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f30e40a2a10 msgr2=0x7f30e40a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.423+0000 7f30fd2a7640 1 --2- 172.21.15.23:0/3781753193 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f30e40a2a10 0x7f30e40a4ed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f30e0005ab0 tx=0x7f30e0005a20 comp rx=0 tx=0).stop 2024-03-28T05:26:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.423+0000 7f30fd2a7640 1 -- 172.21.15.23:0/3781753193 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30f81011b0 msgr2=0x7f30f8198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.423+0000 7f30fd2a7640 1 --2- 172.21.15.23:0/3781753193 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30f81011b0 0x7f30f8198420 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f30e800e9f0 tx=0x7f30e800eeb0 comp rx=0 tx=0).stop 2024-03-28T05:26:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.423+0000 7f30fd2a7640 1 -- 172.21.15.23:0/3781753193 shutdown_connections 2024-03-28T05:26:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.423+0000 7f30fd2a7640 1 --2- 172.21.15.23:0/3781753193 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30f8105540 0x7f30f819c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.423+0000 7f30fd2a7640 1 --2- 172.21.15.23:0/3781753193 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30f81011b0 0x7f30f8198420 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:16.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.423+0000 7f30fd2a7640 1 --2- 172.21.15.23:0/3781753193 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f30e40a2a10 0x7f30e40a4ed0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:16.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.423+0000 7f30fd2a7640 1 --2- 172.21.15.23:0/3781753193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30f81007f0 0x7f30f8197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:16.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.423+0000 7f30fd2a7640 1 -- 172.21.15.23:0/3781753193 >> 172.21.15.23:0/3781753193 conn(0x7f30f80773f0 msgr2=0x7f30f8068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:16.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.423+0000 7f30fd2a7640 1 -- 172.21.15.23:0/3781753193 shutdown_connections 2024-03-28T05:26:16.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:16.423+0000 7f30fd2a7640 1 -- 172.21.15.23:0/3781753193 wait complete. 2024-03-28T05:26:18.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:17 smithi098 ceph-mon[36087]: pgmap v84: 129 pgs: 129 active+clean; 60 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 460 op/s 2024-03-28T05:26:18.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:17 smithi113 ceph-mon[37164]: pgmap v84: 129 pgs: 129 active+clean; 60 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 460 op/s 2024-03-28T05:26:18.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:17 smithi023 ceph-mon[31954]: pgmap v84: 129 pgs: 129 active+clean; 60 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 460 op/s 2024-03-28T05:26:20.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:19 smithi098 ceph-mon[36087]: pgmap v85: 129 pgs: 129 active+clean; 60 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 614 KiB/s wr, 97 op/s 2024-03-28T05:26:20.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:19 smithi113 ceph-mon[37164]: pgmap v85: 129 pgs: 129 active+clean; 60 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 614 KiB/s wr, 97 op/s 2024-03-28T05:26:20.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:19 smithi023 ceph-mon[31954]: pgmap v85: 129 pgs: 129 active+clean; 60 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 614 KiB/s wr, 97 op/s 2024-03-28T05:26:22.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:21 smithi098 ceph-mon[36087]: pgmap v86: 129 pgs: 129 active+clean; 77 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 3.2 MiB/s wr, 595 op/s 2024-03-28T05:26:22.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:21 smithi113 ceph-mon[37164]: pgmap v86: 129 pgs: 129 active+clean; 77 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 3.2 MiB/s wr, 595 op/s 2024-03-28T05:26:22.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:21 smithi023 ceph-mon[31954]: pgmap v86: 129 pgs: 129 active+clean; 77 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 3.2 MiB/s wr, 595 op/s 2024-03-28T05:26:24.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:23 smithi098 ceph-mon[36087]: pgmap v87: 129 pgs: 129 active+clean; 77 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 2.9 MiB/s wr, 548 op/s 2024-03-28T05:26:24.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:23 smithi113 ceph-mon[37164]: pgmap v87: 129 pgs: 129 active+clean; 77 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 2.9 MiB/s wr, 548 op/s 2024-03-28T05:26:24.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:23 smithi023 ceph-mon[31954]: pgmap v87: 129 pgs: 129 active+clean; 77 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 2.9 MiB/s wr, 548 op/s 2024-03-28T05:26:26.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:25 smithi098 ceph-mon[36087]: pgmap v88: 129 pgs: 129 active+clean; 77 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 2.9 MiB/s wr, 548 op/s 2024-03-28T05:26:26.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:25 smithi113 ceph-mon[37164]: pgmap v88: 129 pgs: 129 active+clean; 77 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 2.9 MiB/s wr, 548 op/s 2024-03-28T05:26:26.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:25 smithi023 ceph-mon[31954]: pgmap v88: 129 pgs: 129 active+clean; 77 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 2.9 MiB/s wr, 548 op/s 2024-03-28T05:26:26.399 INFO:tasks.workunit.client.0.smithi023.stdout: 24 109943 189 81813 158 46692 359 2024-03-28T05:26:28.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:27 smithi098 ceph-mon[36087]: pgmap v89: 129 pgs: 129 active+clean; 81 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 634 op/s 2024-03-28T05:26:28.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:27 smithi113 ceph-mon[37164]: pgmap v89: 129 pgs: 129 active+clean; 81 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 634 op/s 2024-03-28T05:26:28.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:27 smithi023 ceph-mon[31954]: pgmap v89: 129 pgs: 129 active+clean; 81 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 634 op/s 2024-03-28T05:26:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.515+0000 7fe458eea640 1 -- 172.21.15.23:0/1129802696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4541007f0 msgr2=0x7fe454100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.515+0000 7fe458eea640 1 --2- 172.21.15.23:0/1129802696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4541007f0 0x7fe454100c70 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe43c009900 tx=0x7fe43c02bc30 comp rx=0 tx=0).stop 2024-03-28T05:26:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.515+0000 7fe458eea640 1 -- 172.21.15.23:0/1129802696 shutdown_connections 2024-03-28T05:26:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.515+0000 7fe458eea640 1 --2- 172.21.15.23:0/1129802696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4541011b0 0x7fe45410d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.517 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.515+0000 7fe458eea640 1 --2- 172.21.15.23:0/1129802696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4541007f0 0x7fe454100c70 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.515+0000 7fe458eea640 1 --2- 172.21.15.23:0/1129802696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe454105540 0x7fe454105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.515+0000 7fe458eea640 1 -- 172.21.15.23:0/1129802696 >> 172.21.15.23:0/1129802696 conn(0x7fe4540773f0 msgr2=0x7fe454077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.515+0000 7fe458eea640 1 -- 172.21.15.23:0/1129802696 shutdown_connections 2024-03-28T05:26:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.515+0000 7fe458eea640 1 -- 172.21.15.23:0/1129802696 wait complete. 2024-03-28T05:26:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.516+0000 7fe458eea640 1 Processor -- start 2024-03-28T05:26:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.516+0000 7fe458eea640 1 -- start start 2024-03-28T05:26:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.516+0000 7fe458eea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4541007f0 0x7fe454197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.516+0000 7fe458eea640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4541011b0 0x7fe4541984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.516+0000 7fe458eea640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe454105540 0x7fe45419c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.516+0000 7fe458eea640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe454111bb0 con 0x7fe454105540 2024-03-28T05:26:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.516+0000 7fe458eea640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe454111a30 con 0x7fe4541007f0 2024-03-28T05:26:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.516+0000 7fe458eea640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe454111d30 con 0x7fe4541011b0 2024-03-28T05:26:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.516+0000 7fe452d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe454105540 0x7fe45419c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.516+0000 7fe452d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe454105540 0x7fe45419c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57780/0 (socket says 172.21.15.23:57780) 2024-03-28T05:26:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.516+0000 7fe452d9d640 1 -- 172.21.15.23:0/3327359855 learned_addr learned my addr 172.21.15.23:0/3327359855 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.516+0000 7fe45259c640 1 --2- 172.21.15.23:0/3327359855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4541007f0 0x7fe454197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.516+0000 7fe44bfff640 1 --2- 172.21.15.23:0/3327359855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4541011b0 0x7fe4541984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.517+0000 7fe452d9d640 1 -- 172.21.15.23:0/3327359855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4541011b0 msgr2=0x7fe4541984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.517+0000 7fe452d9d640 1 --2- 172.21.15.23:0/3327359855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4541011b0 0x7fe4541984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.517+0000 7fe452d9d640 1 -- 172.21.15.23:0/3327359855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4541007f0 msgr2=0x7fe454197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.517+0000 7fe452d9d640 1 --2- 172.21.15.23:0/3327359855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4541007f0 0x7fe454197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.517+0000 7fe452d9d640 1 -- 172.21.15.23:0/3327359855 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45419cf10 con 0x7fe454105540 2024-03-28T05:26:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.517+0000 7fe452d9d640 1 --2- 172.21.15.23:0/3327359855 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe454105540 0x7fe45419c7e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00ea60 tx=0x7fe44c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.517+0000 7fe44b7fe640 1 -- 172.21.15.23:0/3327359855 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44c00ce40 con 0x7fe454105540 2024-03-28T05:26:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.517+0000 7fe44b7fe640 1 -- 172.21.15.23:0/3327359855 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe44c005020 con 0x7fe454105540 2024-03-28T05:26:29.519 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.517+0000 7fe458eea640 1 -- 172.21.15.23:0/3327359855 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe45419d200 con 0x7fe454105540 2024-03-28T05:26:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.517+0000 7fe44b7fe640 1 -- 172.21.15.23:0/3327359855 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44c010690 con 0x7fe454105540 2024-03-28T05:26:29.520 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.517+0000 7fe458eea640 1 -- 172.21.15.23:0/3327359855 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4541a4a90 con 0x7fe454105540 2024-03-28T05:26:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.519+0000 7fe44b7fe640 1 -- 172.21.15.23:0/3327359855 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fe44c005520 con 0x7fe454105540 2024-03-28T05:26:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.520+0000 7fe45bbb4740 1 Processor -- start 2024-03-28T05:26:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.520+0000 7fe45bbb4740 1 -- start start 2024-03-28T05:26:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.520+0000 7fe45bbb4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5572feb47710 0x5572feb47af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.520+0000 7fe45bbb4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb48030 0x5572feb508c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.520+0000 7fe45bbb4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572feb50e00 0x5572feb531f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.520+0000 7fe45bbb4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572feaabb10 con 0x5572feb48030 2024-03-28T05:26:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.520+0000 7fe45bbb4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572fea54900 con 0x5572feb50e00 2024-03-28T05:26:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.520+0000 7fe45bbb4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572feaa7bb0 con 0x5572feb47710 2024-03-28T05:26:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.520+0000 7fe45259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb48030 0x5572feb508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.520+0000 7fe45259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb48030 0x5572feb508c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57782/0 (socket says 172.21.15.23:57782) 2024-03-28T05:26:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.520+0000 7fe45259c640 1 -- 172.21.15.23:0/422494549 learned_addr learned my addr 172.21.15.23:0/422494549 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.520+0000 7fe44bfff640 1 --2- 172.21.15.23:0/422494549 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5572feb47710 0x5572feb47af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.520+0000 7fe44b7fe640 1 --2- 172.21.15.23:0/3327359855 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fe4400a2a10 0x7fe4400a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.520+0000 7fe44bfff640 1 --2- 172.21.15.23:0/422494549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572feb50e00 0x5572feb531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.520+0000 7fe45259c640 1 -- 172.21.15.23:0/422494549 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5572feb47710 msgr2=0x5572feb47af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.520+0000 7fe45259c640 1 --2- 172.21.15.23:0/422494549 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5572feb47710 0x5572feb47af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.520+0000 7fe45259c640 1 -- 172.21.15.23:0/422494549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572feb50e00 msgr2=0x5572feb531f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.520+0000 7fe45259c640 1 --2- 172.21.15.23:0/422494549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572feb50e00 0x5572feb531f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.520+0000 7fe45259c640 1 -- 172.21.15.23:0/422494549 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572fe7c0210 con 0x5572feb48030 2024-03-28T05:26:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.520+0000 7fe452d9d640 1 --2- 172.21.15.23:0/3327359855 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fe4400a2a10 0x7fe4400a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.520+0000 7fe44bfff640 1 --2- 172.21.15.23:0/422494549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572feb50e00 0x5572feb531f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:26:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.521+0000 7fe44b7fe640 1 -- 172.21.15.23:0/3327359855 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fe44c0cf6c0 con 0x7fe454105540 2024-03-28T05:26:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.521+0000 7fe45259c640 1 --2- 172.21.15.23:0/422494549 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb48030 0x5572feb508c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe44400bd10 tx=0x7fe4440107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d984513415d084a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.521+0000 7fe448ff9640 1 -- 172.21.15.23:0/422494549 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe444004030 con 0x5572feb48030 2024-03-28T05:26:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.521+0000 7fe448ff9640 1 -- 172.21.15.23:0/422494549 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4440041d0 con 0x5572feb48030 2024-03-28T05:26:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.521+0000 7fe448ff9640 1 -- 172.21.15.23:0/422494549 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe444011820 con 0x5572feb48030 2024-03-28T05:26:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.522+0000 7fe45bbb4740 1 -- 172.21.15.23:0/422494549 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb48030 msgr2=0x5572feb508c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.522+0000 7fe452d9d640 1 --2- 172.21.15.23:0/3327359855 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fe4400a2a10 0x7fe4400a4ed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe44c0ccb30 tx=0x7fe44c06d5c0 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.522+0000 7fe45bbb4740 1 --2- 172.21.15.23:0/422494549 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb48030 0x5572feb508c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe44400bd10 tx=0x7fe4440107a0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.522+0000 7fe45bbb4740 1 -- 172.21.15.23:0/422494549 shutdown_connections 2024-03-28T05:26:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.522+0000 7fe45bbb4740 1 --2- 172.21.15.23:0/422494549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572feb50e00 0x5572feb531f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.522+0000 7fe45bbb4740 1 --2- 172.21.15.23:0/422494549 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb48030 0x5572feb508c0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.522+0000 7fe45bbb4740 1 --2- 172.21.15.23:0/422494549 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5572feb47710 0x5572feb47af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.522+0000 7fe45bbb4740 1 -- 172.21.15.23:0/422494549 >> 172.21.15.23:0/422494549 conn(0x5572feb42030 msgr2=0x5572feb42410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.522+0000 7fe45bbb4740 1 -- 172.21.15.23:0/422494549 shutdown_connections 2024-03-28T05:26:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.522+0000 7fe45bbb4740 1 -- 172.21.15.23:0/422494549 wait complete. 2024-03-28T05:26:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.522+0000 7fe45bbb4740 1 -- start start 2024-03-28T05:26:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.522+0000 7fe45bbb4740 1 Processor -- start 2024-03-28T05:26:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.522+0000 7fe45bbb4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572feb47710 0x5572feb4de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.522+0000 7fe45bbb4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5572feb48030 0x5572feb4e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.522+0000 7fe45bbb4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb50e00 0x5572feb4e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.523+0000 7fe45bbb4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572feb53a30 con 0x5572feb50e00 2024-03-28T05:26:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.523+0000 7fe45bbb4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572fea67d50 con 0x5572feb47710 2024-03-28T05:26:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.523+0000 7fe45bbb4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572feb53bb0 con 0x5572feb48030 2024-03-28T05:26:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.523+0000 7fe45bbb4740 10 client.0 _ll_register_callbacks cb 0x5572fea67720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:26:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.523+0000 7fe44bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb50e00 0x5572feb4e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.523+0000 7fe44bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb50e00 0x5572feb4e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57796/0 (socket says 172.21.15.23:57796) 2024-03-28T05:26:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.523+0000 7fe44bfff640 1 -- 172.21.15.23:0/202139750 learned_addr learned my addr 172.21.15.23:0/202139750 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.523+0000 7fe45259c640 1 --2- 172.21.15.23:0/202139750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5572feb48030 0x5572feb4e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.523+0000 7fe44bfff640 1 --2- 172.21.15.23:0/202139750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572feb47710 0x5572feb4de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.523+0000 7fe45259c640 1 -- 172.21.15.23:0/202139750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572feb47710 msgr2=0x5572feb4de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.523+0000 7fe45259c640 1 --2- 172.21.15.23:0/202139750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572feb47710 0x5572feb4de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.523+0000 7fe45259c640 1 -- 172.21.15.23:0/202139750 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb50e00 msgr2=0x5572feb4e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.523+0000 7fe45259c640 1 --2- 172.21.15.23:0/202139750 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb50e00 0x5572feb4e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.523+0000 7fe45259c640 1 -- 172.21.15.23:0/202139750 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5572fe88c620 con 0x5572feb48030 2024-03-28T05:26:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.523+0000 7fe44bfff640 1 --2- 172.21.15.23:0/202139750 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb50e00 0x5572feb4e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:26:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.523+0000 7fe45259c640 1 --2- 172.21.15.23:0/202139750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5572feb48030 0x5572feb4e360 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe44400bd10 tx=0x7fe444003e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef80964288133e87 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.523+0000 7fe439ffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-28T05:26:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.523+0000 7fe439ffb640 1 -- 172.21.15.23:0/202139750 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe444029060 con 0x5572feb48030 2024-03-28T05:26:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.523+0000 7fe439ffb640 1 -- 172.21.15.23:0/202139750 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fe4440035f0 con 0x5572feb48030 2024-03-28T05:26:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.523+0000 7fe45bbb4740 1 -- 172.21.15.23:0/202139750 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe4440041d0 con 0x5572feb48030 2024-03-28T05:26:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.524+0000 7fe439ffb640 1 -- 172.21.15.23:0/202139750 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe444030050 con 0x5572feb48030 2024-03-28T05:26:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.524+0000 7fe45bbb4740 10 client.25546 fetch_fsmap learned FSMap version 20 2024-03-28T05:26:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.524+0000 7fe45bbb4740 1 -- 172.21.15.23:0/202139750 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5572feb5fbe0 con 0x5572feb48030 2024-03-28T05:26:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.524+0000 7fe439ffb640 1 -- 172.21.15.23:0/202139750 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4440041d0 con 0x5572feb48030 2024-03-28T05:26:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.524+0000 7fe439ffb640 1 -- 172.21.15.23:0/202139750 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fe4440046c0 con 0x5572feb48030 2024-03-28T05:26:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.524+0000 7fe45bbb4740 10 client.25546 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:26:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.524+0000 7fe45bbb4740 10 client.25546 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:26:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.524+0000 7fe45bbb4740 20 client.25546 populate_metadata read hostname 'smithi023' 2024-03-28T05:26:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.524+0000 7fe45bbb4740 1 --2- 172.21.15.23:0/202139750 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5572feb60fe0 0x5572feb633d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.524+0000 7fe45bbb4740 4 client.25546 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:26:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.524+0000 7fe45bbb4740 1 -- 172.21.15.23:0/202139750 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5572feab9050 con 0x5572feb60fe0 2024-03-28T05:26:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.524+0000 7fe44bfff640 1 --2- 172.21.15.23:0/202139750 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5572feb60fe0 0x5572feb633d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.525+0000 7fe439ffb640 10 client.25546 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:26:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.525+0000 7fe44bfff640 1 --2- 172.21.15.23:0/202139750 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5572feb60fe0 0x5572feb633d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94c8f5fcd7f9f64b server_cookie=5da35ebc2ab53ea3 in_seq=0 out_seq=0 2024-03-28T05:26:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.526+0000 7fe439ffb640 1 -- 172.21.15.23:0/202139750 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5572feab9050 con 0x5572feb60fe0 2024-03-28T05:26:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.526+0000 7fe439ffb640 10 client.25546 handle_command_reply: tid=0 2024-03-28T05:26:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.526+0000 7fe45bbb4740 1 client.25546 shutdown 2024-03-28T05:26:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.526+0000 7fe45bbb4740 1 -- 172.21.15.23:0/202139750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5572feb48030 msgr2=0x5572feb4e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.526+0000 7fe45bbb4740 1 --2- 172.21.15.23:0/202139750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5572feb48030 0x5572feb4e360 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe44400bd10 tx=0x7fe444003e30 comp rx=0 tx=0).stop 2024-03-28T05:26:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.526+0000 7fe45bbb4740 1 -- 172.21.15.23:0/202139750 shutdown_connections 2024-03-28T05:26:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.526+0000 7fe45bbb4740 1 --2- 172.21.15.23:0/202139750 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb50e00 0x5572feb4e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.532 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.526+0000 7fe45bbb4740 1 --2- 172.21.15.23:0/202139750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5572feb48030 0x5572feb4e360 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.532 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.526+0000 7fe45bbb4740 1 --2- 172.21.15.23:0/202139750 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5572feb60fe0 0x5572feb633d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.532 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.526+0000 7fe45bbb4740 1 --2- 172.21.15.23:0/202139750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572feb47710 0x5572feb4de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.532 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.526+0000 7fe45bbb4740 1 -- 172.21.15.23:0/202139750 >> 172.21.15.23:0/202139750 conn(0x5572feb42030 msgr2=0x5572feb43d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:29.532 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.526+0000 7fe439ffb640 0 client.25546 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:26:29.532 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.526+0000 7fe45bbb4740 1 -- 172.21.15.23:0/202139750 shutdown_connections 2024-03-28T05:26:29.532 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.526+0000 7fe45bbb4740 1 -- 172.21.15.23:0/202139750 wait complete. 2024-03-28T05:26:29.532 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.527+0000 7fe45bbb4740 20 client.25546 trim_cache size 0 max 16384 2024-03-28T05:26:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.541+0000 7fe45bbb4740 1 Processor -- start 2024-03-28T05:26:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.541+0000 7fe45bbb4740 1 -- start start 2024-03-28T05:26:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.541+0000 7fe45bbb4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572feb47710 0x5572feb50550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.541+0000 7fe45bbb4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb48030 0x5572feb5c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.541+0000 7fe45bbb4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5572feb50e00 0x5572feb4b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.541+0000 7fe45bbb4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4541ac9f0 con 0x5572feb48030 2024-03-28T05:26:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.541+0000 7fe45bbb4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4541aefb0 con 0x5572feb47710 2024-03-28T05:26:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.541+0000 7fe45bbb4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4541acff0 con 0x5572feb50e00 2024-03-28T05:26:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.541+0000 7fe45259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb48030 0x5572feb5c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.541+0000 7fe45259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb48030 0x5572feb5c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57804/0 (socket says 172.21.15.23:57804) 2024-03-28T05:26:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.541+0000 7fe45259c640 1 -- 172.21.15.23:0/748825321 learned_addr learned my addr 172.21.15.23:0/748825321 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.541+0000 7fe44bfff640 1 --2- 172.21.15.23:0/748825321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572feb47710 0x5572feb50550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.541+0000 7fe44bfff640 1 --2- 172.21.15.23:0/748825321 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5572feb50e00 0x5572feb4b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.541+0000 7fe45259c640 1 -- 172.21.15.23:0/748825321 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5572feb50e00 msgr2=0x5572feb4b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.541+0000 7fe45259c640 1 --2- 172.21.15.23:0/748825321 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5572feb50e00 0x5572feb4b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.541+0000 7fe45259c640 1 -- 172.21.15.23:0/748825321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572feb47710 msgr2=0x5572feb50550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.541+0000 7fe45259c640 1 --2- 172.21.15.23:0/748825321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572feb47710 0x5572feb50550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.541+0000 7fe45259c640 1 -- 172.21.15.23:0/748825321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572feb531f0 con 0x5572feb48030 2024-03-28T05:26:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.541+0000 7fe44bfff640 1 --2- 172.21.15.23:0/748825321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572feb47710 0x5572feb50550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:26:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.541+0000 7fe45259c640 1 --2- 172.21.15.23:0/748825321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb48030 0x5572feb5c510 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fe444003f50 tx=0x7fe444010dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f68fac7ccacfa22e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.541+0000 7fe43bfff640 1 -- 172.21.15.23:0/748825321 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe444029070 con 0x5572feb48030 2024-03-28T05:26:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.542+0000 7fe43bfff640 1 -- 172.21.15.23:0/748825321 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4440040c0 con 0x5572feb48030 2024-03-28T05:26:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.542+0000 7fe43bfff640 1 -- 172.21.15.23:0/748825321 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe444004f70 con 0x5572feb48030 2024-03-28T05:26:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.542+0000 7fe45bbb4740 1 -- 172.21.15.23:0/748825321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb48030 msgr2=0x5572feb5c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.542+0000 7fe45bbb4740 1 --2- 172.21.15.23:0/748825321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb48030 0x5572feb5c510 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fe444003f50 tx=0x7fe444010dd0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.542+0000 7fe45bbb4740 1 -- 172.21.15.23:0/748825321 shutdown_connections 2024-03-28T05:26:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.542+0000 7fe45bbb4740 1 --2- 172.21.15.23:0/748825321 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5572feb50e00 0x5572feb4b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.542+0000 7fe45bbb4740 1 --2- 172.21.15.23:0/748825321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb48030 0x5572feb5c510 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.542+0000 7fe45bbb4740 1 --2- 172.21.15.23:0/748825321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572feb47710 0x5572feb50550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.542+0000 7fe45bbb4740 1 -- 172.21.15.23:0/748825321 >> 172.21.15.23:0/748825321 conn(0x5572feb42030 msgr2=0x5572feb5be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.542+0000 7fe45bbb4740 1 -- 172.21.15.23:0/748825321 shutdown_connections 2024-03-28T05:26:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.542+0000 7fe45bbb4740 1 -- 172.21.15.23:0/748825321 wait complete. 2024-03-28T05:26:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.542+0000 7fe45bbb4740 1 -- start start 2024-03-28T05:26:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.542+0000 7fe45bbb4740 1 Processor -- start 2024-03-28T05:26:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe45bbb4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572feb47710 0x5572feb63eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe45bbb4740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb48030 0x5572feb643f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe45bbb4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5572feb50e00 0x5572febbb510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe45bbb4740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4541ad630 con 0x5572feb48030 2024-03-28T05:26:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe45bbb4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4541ae950 con 0x5572feb47710 2024-03-28T05:26:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe45bbb4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4541adc90 con 0x5572feb50e00 2024-03-28T05:26:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe45bbb4740 10 client.0 _ll_register_callbacks cb 0x7fe444003d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:26:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe45259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb48030 0x5572feb643f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe45259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb48030 0x5572feb643f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57814/0 (socket says 172.21.15.23:57814) 2024-03-28T05:26:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe45259c640 1 -- 172.21.15.23:0/255840450 learned_addr learned my addr 172.21.15.23:0/255840450 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe44bfff640 1 --2- 172.21.15.23:0/255840450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5572feb50e00 0x5572febbb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe44bfff640 1 --2- 172.21.15.23:0/255840450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572feb47710 0x5572feb63eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe45259c640 1 -- 172.21.15.23:0/255840450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5572feb50e00 msgr2=0x5572febbb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe45259c640 1 --2- 172.21.15.23:0/255840450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5572feb50e00 0x5572febbb510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe45259c640 1 -- 172.21.15.23:0/255840450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572feb47710 msgr2=0x5572feb63eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe45259c640 1 --2- 172.21.15.23:0/255840450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572feb47710 0x5572feb63eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe45259c640 1 -- 172.21.15.23:0/255840450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5572febbba50 con 0x5572feb48030 2024-03-28T05:26:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe44bfff640 1 --2- 172.21.15.23:0/255840450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5572feb50e00 0x5572febbb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:26:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe45259c640 1 --2- 172.21.15.23:0/255840450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb48030 0x5572feb643f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fe444000c00 tx=0x7fe44400ba00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=803188d6e759a08a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe439ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:26:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe439ffb640 1 -- 172.21.15.23:0/255840450 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe444029070 con 0x5572feb48030 2024-03-28T05:26:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe439ffb640 1 -- 172.21.15.23:0/255840450 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fe4440047a0 con 0x5572feb48030 2024-03-28T05:26:29.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.543+0000 7fe45bbb4740 1 -- 172.21.15.23:0/255840450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe4440040c0 con 0x5572feb48030 2024-03-28T05:26:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.544+0000 7fe439ffb640 1 -- 172.21.15.23:0/255840450 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe444046050 con 0x5572feb48030 2024-03-28T05:26:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.544+0000 7fe45bbb4740 10 client.15717 fetch_fsmap learned FSMap version 20 2024-03-28T05:26:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.544+0000 7fe45bbb4740 1 -- 172.21.15.23:0/255840450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5572febbccd0 con 0x5572feb48030 2024-03-28T05:26:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.544+0000 7fe439ffb640 1 -- 172.21.15.23:0/255840450 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4440040c0 con 0x5572feb48030 2024-03-28T05:26:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.544+0000 7fe439ffb640 1 -- 172.21.15.23:0/255840450 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fe44403b020 con 0x5572feb48030 2024-03-28T05:26:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.544+0000 7fe45bbb4740 10 client.15717 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:26:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.544+0000 7fe45bbb4740 10 client.15717 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:26:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.544+0000 7fe45bbb4740 20 client.15717 populate_metadata read hostname 'smithi023' 2024-03-28T05:26:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.544+0000 7fe45bbb4740 1 --2- 172.21.15.23:0/255840450 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5572febbd280 0x5572febbf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.544+0000 7fe45bbb4740 4 client.15717 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:26:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.544+0000 7fe45bbb4740 1 -- 172.21.15.23:0/255840450 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5572feab33a0 con 0x5572febbd280 2024-03-28T05:26:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.544+0000 7fe44bfff640 1 --2- 172.21.15.23:0/255840450 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5572febbd280 0x5572febbf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.545+0000 7fe439ffb640 10 client.15717 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:26:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:29.545+0000 7fe44bfff640 1 --2- 172.21.15.23:0/255840450 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5572febbd280 0x5572febbf670 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e689524ce8e7322 server_cookie=5411b28adb7c1a03 in_seq=0 out_seq=0 2024-03-28T05:26:30.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.034+0000 7fe439ffb640 1 -- 172.21.15.23:0/255840450 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5572feab33a0 con 0x5572febbd280 2024-03-28T05:26:30.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.034+0000 7fe439ffb640 10 client.15717 handle_command_reply: tid=0 2024-03-28T05:26:30.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.034+0000 7fe45bbb4740 1 client.15717 shutdown 2024-03-28T05:26:30.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.034+0000 7fe45bbb4740 1 -- 172.21.15.23:0/255840450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb48030 msgr2=0x5572feb643f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:30.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.034+0000 7fe45bbb4740 1 --2- 172.21.15.23:0/255840450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb48030 0x5572feb643f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fe444000c00 tx=0x7fe44400ba00 comp rx=0 tx=0).stop 2024-03-28T05:26:30.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.034+0000 7fe45bbb4740 1 -- 172.21.15.23:0/255840450 shutdown_connections 2024-03-28T05:26:30.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.034+0000 7fe45bbb4740 1 --2- 172.21.15.23:0/255840450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5572feb50e00 0x5572febbb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:30.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.034+0000 7fe45bbb4740 1 --2- 172.21.15.23:0/255840450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572feb48030 0x5572feb643f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:30.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.034+0000 7fe45bbb4740 1 --2- 172.21.15.23:0/255840450 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5572febbd280 0x5572febbf670 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:30.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.034+0000 7fe45bbb4740 1 --2- 172.21.15.23:0/255840450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5572feb47710 0x5572feb63eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:30.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.034+0000 7fe439ffb640 0 client.15717 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:26:30.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.034+0000 7fe45bbb4740 1 -- 172.21.15.23:0/255840450 >> 172.21.15.23:0/255840450 conn(0x5572feb42030 msgr2=0x5572feb41020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:30.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.034+0000 7fe45bbb4740 1 -- 172.21.15.23:0/255840450 shutdown_connections 2024-03-28T05:26:30.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.034+0000 7fe45bbb4740 1 -- 172.21.15.23:0/255840450 wait complete. 2024-03-28T05:26:30.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.034+0000 7fe45bbb4740 20 client.15717 trim_cache size 0 max 16384 2024-03-28T05:26:30.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:26:30.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:26:30.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:26:30.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:26:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.035+0000 7fe458eea640 1 -- 172.21.15.23:0/3327359855 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fe4400a2a10 msgr2=0x7fe4400a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.035+0000 7fe458eea640 1 --2- 172.21.15.23:0/3327359855 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fe4400a2a10 0x7fe4400a4ed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe44c0ccb30 tx=0x7fe44c06d5c0 comp rx=0 tx=0).stop 2024-03-28T05:26:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.035+0000 7fe458eea640 1 -- 172.21.15.23:0/3327359855 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe454105540 msgr2=0x7fe45419c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.035+0000 7fe458eea640 1 --2- 172.21.15.23:0/3327359855 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe454105540 0x7fe45419c7e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00ea60 tx=0x7fe44c00ef20 comp rx=0 tx=0).stop 2024-03-28T05:26:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.035+0000 7fe458eea640 1 -- 172.21.15.23:0/3327359855 shutdown_connections 2024-03-28T05:26:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.035+0000 7fe458eea640 1 --2- 172.21.15.23:0/3327359855 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe454105540 0x7fe45419c7e0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.035+0000 7fe458eea640 1 --2- 172.21.15.23:0/3327359855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe4541011b0 0x7fe4541984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.035+0000 7fe458eea640 1 --2- 172.21.15.23:0/3327359855 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fe4400a2a10 0x7fe4400a4ed0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.035+0000 7fe458eea640 1 --2- 172.21.15.23:0/3327359855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4541007f0 0x7fe454197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.035+0000 7fe458eea640 1 -- 172.21.15.23:0/3327359855 >> 172.21.15.23:0/3327359855 conn(0x7fe4540773f0 msgr2=0x7fe4540ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.035+0000 7fe458eea640 1 -- 172.21.15.23:0/3327359855 shutdown_connections 2024-03-28T05:26:30.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:30.035+0000 7fe458eea640 1 -- 172.21.15.23:0/3327359855 wait complete. 2024-03-28T05:26:30.115 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:29 smithi098 ceph-mon[36087]: pgmap v90: 129 pgs: 129 active+clean; 81 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 583 op/s 2024-03-28T05:26:30.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:29 smithi113 ceph-mon[37164]: pgmap v90: 129 pgs: 129 active+clean; 81 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 583 op/s 2024-03-28T05:26:30.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:29 smithi023 ceph-mon[31954]: pgmap v90: 129 pgs: 129 active+clean; 81 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 583 op/s 2024-03-28T05:26:32.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:31 smithi098 ceph-mon[36087]: pgmap v91: 129 pgs: 129 active+clean; 82 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.3 MiB/s wr, 876 op/s 2024-03-28T05:26:32.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:31 smithi113 ceph-mon[37164]: pgmap v91: 129 pgs: 129 active+clean; 82 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.3 MiB/s wr, 876 op/s 2024-03-28T05:26:32.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:31 smithi023 ceph-mon[31954]: pgmap v91: 129 pgs: 129 active+clean; 82 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.3 MiB/s wr, 876 op/s 2024-03-28T05:26:34.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:33 smithi098 ceph-mon[36087]: pgmap v92: 129 pgs: 129 active+clean; 82 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 749 KiB/s wr, 378 op/s 2024-03-28T05:26:34.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:33 smithi113 ceph-mon[37164]: pgmap v92: 129 pgs: 129 active+clean; 82 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 749 KiB/s wr, 378 op/s 2024-03-28T05:26:34.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:33 smithi023 ceph-mon[31954]: pgmap v92: 129 pgs: 129 active+clean; 82 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 749 KiB/s wr, 378 op/s 2024-03-28T05:26:36.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:35 smithi098 ceph-mon[36087]: pgmap v93: 129 pgs: 129 active+clean; 82 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 749 KiB/s wr, 378 op/s 2024-03-28T05:26:36.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:35 smithi113 ceph-mon[37164]: pgmap v93: 129 pgs: 129 active+clean; 82 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 749 KiB/s wr, 378 op/s 2024-03-28T05:26:36.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:35 smithi023 ceph-mon[31954]: pgmap v93: 129 pgs: 129 active+clean; 82 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 749 KiB/s wr, 378 op/s 2024-03-28T05:26:36.399 INFO:tasks.workunit.client.0.smithi023.stdout: 24 128750 167 94415 110 57097 415 2024-03-28T05:26:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:37 smithi098 ceph-mon[36087]: pgmap v94: 129 pgs: 129 active+clean; 78 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 424 op/s 2024-03-28T05:26:38.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:37 smithi113 ceph-mon[37164]: pgmap v94: 129 pgs: 129 active+clean; 78 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 424 op/s 2024-03-28T05:26:38.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:37 smithi023 ceph-mon[31954]: pgmap v94: 129 pgs: 129 active+clean; 78 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 424 op/s 2024-03-28T05:26:40.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:39 smithi098 ceph-mon[36087]: pgmap v95: 129 pgs: 129 active+clean; 78 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 643 KiB/s wr, 338 op/s 2024-03-28T05:26:40.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:39 smithi113 ceph-mon[37164]: pgmap v95: 129 pgs: 129 active+clean; 78 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 643 KiB/s wr, 338 op/s 2024-03-28T05:26:40.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:39 smithi023 ceph-mon[31954]: pgmap v95: 129 pgs: 129 active+clean; 78 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 643 KiB/s wr, 338 op/s 2024-03-28T05:26:42.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:41 smithi098 ceph-mon[36087]: pgmap v96: 129 pgs: 129 active+clean; 80 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 888 KiB/s wr, 393 op/s 2024-03-28T05:26:42.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:41 smithi113 ceph-mon[37164]: pgmap v96: 129 pgs: 129 active+clean; 80 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 888 KiB/s wr, 393 op/s 2024-03-28T05:26:42.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:41 smithi023 ceph-mon[31954]: pgmap v96: 129 pgs: 129 active+clean; 80 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 888 KiB/s wr, 393 op/s 2024-03-28T05:26:43.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:42 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:26:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.132+0000 7f127deea640 1 -- 172.21.15.23:0/1706119284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1278105540 msgr2=0x7f1278105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.132+0000 7f127deea640 1 --2- 172.21.15.23:0/1706119284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1278105540 0x7f1278105920 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1268009900 tx=0x7f126802bc30 comp rx=0 tx=0).stop 2024-03-28T05:26:43.133 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.132+0000 7f127deea640 1 -- 172.21.15.23:0/1706119284 shutdown_connections 2024-03-28T05:26:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.132+0000 7f127deea640 1 --2- 172.21.15.23:0/1706119284 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12781011b0 0x7f127810d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.132+0000 7f127deea640 1 --2- 172.21.15.23:0/1706119284 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12781007f0 0x7f1278100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.132+0000 7f127deea640 1 --2- 172.21.15.23:0/1706119284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1278105540 0x7f1278105920 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.132+0000 7f127deea640 1 -- 172.21.15.23:0/1706119284 >> 172.21.15.23:0/1706119284 conn(0x7f12780773f0 msgr2=0x7f1278077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.132+0000 7f127deea640 1 -- 172.21.15.23:0/1706119284 shutdown_connections 2024-03-28T05:26:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.132+0000 7f127deea640 1 -- 172.21.15.23:0/1706119284 wait complete. 2024-03-28T05:26:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.133+0000 7f127deea640 1 Processor -- start 2024-03-28T05:26:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.133+0000 7f127deea640 1 -- start start 2024-03-28T05:26:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.133+0000 7f127deea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12781007f0 0x7f1278198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.133+0000 7f127deea640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12781011b0 0x7f1278199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.133+0000 7f127deea640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1278105540 0x7f12781a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.133+0000 7f127deea640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1278110a20 con 0x7f12781011b0 2024-03-28T05:26:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.133+0000 7f127deea640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12781108a0 con 0x7f12781007f0 2024-03-28T05:26:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.133+0000 7f127deea640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1278110ba0 con 0x7f1278105540 2024-03-28T05:26:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.133+0000 7f1276ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12781011b0 0x7f1278199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.133+0000 7f12777fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12781007f0 0x7f1278198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.133+0000 7f1276ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12781011b0 0x7f1278199300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60748/0 (socket says 172.21.15.23:60748) 2024-03-28T05:26:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.133+0000 7f1276ffd640 1 -- 172.21.15.23:0/2100928050 learned_addr learned my addr 172.21.15.23:0/2100928050 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.133+0000 7f1277fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1278105540 0x7f12781a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.134+0000 7f1276ffd640 1 -- 172.21.15.23:0/2100928050 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1278105540 msgr2=0x7f12781a3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.134+0000 7f1276ffd640 1 --2- 172.21.15.23:0/2100928050 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1278105540 0x7f12781a3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.134+0000 7f1276ffd640 1 -- 172.21.15.23:0/2100928050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12781007f0 msgr2=0x7f1278198dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.134+0000 7f1276ffd640 1 --2- 172.21.15.23:0/2100928050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12781007f0 0x7f1278198dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.134+0000 7f1276ffd640 1 -- 172.21.15.23:0/2100928050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12781a4380 con 0x7f12781011b0 2024-03-28T05:26:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.134+0000 7f1276ffd640 1 --2- 172.21.15.23:0/2100928050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12781011b0 0x7f1278199300 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f126c00e9f0 tx=0x7f126c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.134+0000 7f1274ff9640 1 -- 172.21.15.23:0/2100928050 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126c00cd80 con 0x7f12781011b0 2024-03-28T05:26:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.134+0000 7f1274ff9640 1 -- 172.21.15.23:0/2100928050 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f126c010910 con 0x7f12781011b0 2024-03-28T05:26:43.135 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.134+0000 7f127deea640 1 -- 172.21.15.23:0/2100928050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12781a4550 con 0x7f12781011b0 2024-03-28T05:26:43.136 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.134+0000 7f1274ff9640 1 -- 172.21.15.23:0/2100928050 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126c018990 con 0x7f12781011b0 2024-03-28T05:26:43.136 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.134+0000 7f127deea640 1 -- 172.21.15.23:0/2100928050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12781a4aa0 con 0x7f12781011b0 2024-03-28T05:26:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.136+0000 7f1280bb1740 1 Processor -- start 2024-03-28T05:26:43.137 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.137+0000 7f1280bb1740 1 -- start start 2024-03-28T05:26:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.137+0000 7f1274ff9640 1 -- 172.21.15.23:0/2100928050 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f126c018b30 con 0x7f12781011b0 2024-03-28T05:26:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.137+0000 7f1280bb1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b0963160 0x55e3b0963540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.137+0000 7f1280bb1740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e3b0963b10 0x55e3b0963f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.137+0000 7f1280bb1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3b09754a0 0x55e3b0977890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.137+0000 7f1280bb1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3b08c7b10 con 0x55e3b0963160 2024-03-28T05:26:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.137+0000 7f1280bb1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3b0870900 con 0x55e3b09754a0 2024-03-28T05:26:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.137+0000 7f1280bb1740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3b08c3bb0 con 0x55e3b0963b10 2024-03-28T05:26:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.137+0000 7f1277fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b0963160 0x55e3b0963540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.137+0000 7f1277fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b0963160 0x55e3b0963540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60756/0 (socket says 172.21.15.23:60756) 2024-03-28T05:26:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.137+0000 7f1277fff640 1 -- 172.21.15.23:0/1561405177 learned_addr learned my addr 172.21.15.23:0/1561405177 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.137+0000 7f1276ffd640 1 --2- 172.21.15.23:0/1561405177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3b09754a0 0x55e3b0977890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.138+0000 7f1277fff640 1 -- 172.21.15.23:0/1561405177 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e3b0963b10 msgr2=0x55e3b0963f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.138+0000 7f1277fff640 1 --2- 172.21.15.23:0/1561405177 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e3b0963b10 0x55e3b0963f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.138+0000 7f1277fff640 1 -- 172.21.15.23:0/1561405177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3b09754a0 msgr2=0x55e3b0977890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.138+0000 7f1277fff640 1 --2- 172.21.15.23:0/1561405177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3b09754a0 0x55e3b0977890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.138+0000 7f1277fff640 1 -- 172.21.15.23:0/1561405177 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3b05dc210 con 0x55e3b0963160 2024-03-28T05:26:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.138+0000 7f1274ff9640 1 --2- 172.21.15.23:0/2100928050 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f12600a2a10 0x7f12600a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.138+0000 7f1274ff9640 1 -- 172.21.15.23:0/2100928050 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f126c0cf4a0 con 0x7f12781011b0 2024-03-28T05:26:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.138+0000 7f12777fe640 1 --2- 172.21.15.23:0/2100928050 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f12600a2a10 0x7f12600a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.138+0000 7f1277fff640 1 --2- 172.21.15.23:0/1561405177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b0963160 0x55e3b0963540 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f1270009f80 tx=0x7f1270010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8b449b225d66fc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.138+0000 7f125dffb640 1 -- 172.21.15.23:0/1561405177 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1270011700 con 0x55e3b0963160 2024-03-28T05:26:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.138+0000 7f125dffb640 1 -- 172.21.15.23:0/1561405177 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1270011d80 con 0x55e3b0963160 2024-03-28T05:26:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.138+0000 7f125dffb640 1 -- 172.21.15.23:0/1561405177 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1270018380 con 0x55e3b0963160 2024-03-28T05:26:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.139+0000 7f1280bb1740 1 -- 172.21.15.23:0/1561405177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b0963160 msgr2=0x55e3b0963540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.139+0000 7f12777fe640 1 --2- 172.21.15.23:0/2100928050 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f12600a2a10 0x7f12600a4ed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1268009230 tx=0x7f1268005dd0 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.139+0000 7f1280bb1740 1 --2- 172.21.15.23:0/1561405177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b0963160 0x55e3b0963540 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f1270009f80 tx=0x7f1270010860 comp rx=0 tx=0).stop 2024-03-28T05:26:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.139+0000 7f1280bb1740 1 -- 172.21.15.23:0/1561405177 shutdown_connections 2024-03-28T05:26:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.139+0000 7f1280bb1740 1 --2- 172.21.15.23:0/1561405177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3b09754a0 0x55e3b0977890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.139+0000 7f1280bb1740 1 --2- 172.21.15.23:0/1561405177 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e3b0963b10 0x55e3b0963f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.139+0000 7f1280bb1740 1 --2- 172.21.15.23:0/1561405177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b0963160 0x55e3b0963540 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.140+0000 7f1280bb1740 1 -- 172.21.15.23:0/1561405177 >> 172.21.15.23:0/1561405177 conn(0x55e3b095e030 msgr2=0x55e3b095e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.140+0000 7f1280bb1740 1 -- 172.21.15.23:0/1561405177 shutdown_connections 2024-03-28T05:26:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.140+0000 7f1280bb1740 1 -- 172.21.15.23:0/1561405177 wait complete. 2024-03-28T05:26:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.140+0000 7f1280bb1740 1 -- start start 2024-03-28T05:26:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.140+0000 7f1280bb1740 1 Processor -- start 2024-03-28T05:26:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.141+0000 7f1280bb1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3b0963160 0x55e3b09658b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.141+0000 7f1280bb1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b0963b10 0x55e3b0965df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.141+0000 7f1280bb1740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e3b09754a0 0x55e3b0966330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.141+0000 7f1280bb1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3b09781d0 con 0x55e3b0963b10 2024-03-28T05:26:43.143 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.141+0000 7f1280bb1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3b0883d50 con 0x55e3b0963160 2024-03-28T05:26:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.141+0000 7f1280bb1740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3b0978350 con 0x55e3b09754a0 2024-03-28T05:26:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.141+0000 7f1280bb1740 10 client.0 _ll_register_callbacks cb 0x55e3b0883720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:26:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.141+0000 7f1276ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b0963b10 0x55e3b0965df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.141+0000 7f1276ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b0963b10 0x55e3b0965df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60768/0 (socket says 172.21.15.23:60768) 2024-03-28T05:26:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.141+0000 7f1277fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e3b09754a0 0x55e3b0966330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.141+0000 7f1276ffd640 1 -- 172.21.15.23:0/2766011910 learned_addr learned my addr 172.21.15.23:0/2766011910 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.141+0000 7f12777fe640 1 --2- 172.21.15.23:0/2766011910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3b0963160 0x55e3b09658b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.141+0000 7f1276ffd640 1 -- 172.21.15.23:0/2766011910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e3b09754a0 msgr2=0x55e3b0966330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.141+0000 7f1276ffd640 1 --2- 172.21.15.23:0/2766011910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e3b09754a0 0x55e3b0966330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.141+0000 7f1276ffd640 1 -- 172.21.15.23:0/2766011910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3b0963160 msgr2=0x55e3b09658b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.141+0000 7f1276ffd640 1 --2- 172.21.15.23:0/2766011910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3b0963160 0x55e3b09658b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.141+0000 7f1276ffd640 1 -- 172.21.15.23:0/2766011910 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e3b06a8620 con 0x55e3b0963b10 2024-03-28T05:26:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.141+0000 7f1277fff640 1 --2- 172.21.15.23:0/2766011910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e3b09754a0 0x55e3b0966330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:26:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.141+0000 7f12777fe640 1 --2- 172.21.15.23:0/2766011910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3b0963160 0x55e3b09658b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:26:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.142+0000 7f1276ffd640 1 --2- 172.21.15.23:0/2766011910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b0963b10 0x55e3b0965df0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f126c00e9c0 tx=0x7f126c0d2c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7c3962214608bb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.142+0000 7f1256ffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:26:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.142+0000 7f1256ffd640 1 -- 172.21.15.23:0/2766011910 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126c09b220 con 0x55e3b0963b10 2024-03-28T05:26:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.142+0000 7f1256ffd640 1 -- 172.21.15.23:0/2766011910 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f126c097f30 con 0x55e3b0963b10 2024-03-28T05:26:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.142+0000 7f1280bb1740 1 -- 172.21.15.23:0/2766011910 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1270011d80 con 0x55e3b0963b10 2024-03-28T05:26:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.142+0000 7f1256ffd640 1 -- 172.21.15.23:0/2766011910 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f126c09b8c0 con 0x55e3b0963b10 2024-03-28T05:26:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.142+0000 7f1280bb1740 10 client.15729 fetch_fsmap learned FSMap version 20 2024-03-28T05:26:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.142+0000 7f1280bb1740 1 -- 172.21.15.23:0/2766011910 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55e3b0984460 con 0x55e3b0963b10 2024-03-28T05:26:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.142+0000 7f1256ffd640 1 -- 172.21.15.23:0/2766011910 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1270011d80 con 0x55e3b0963b10 2024-03-28T05:26:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.142+0000 7f1256ffd640 1 -- 172.21.15.23:0/2766011910 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f126c0860e0 con 0x55e3b0963b10 2024-03-28T05:26:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.142+0000 7f1280bb1740 10 client.15729 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:26:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.142+0000 7f1280bb1740 10 client.15729 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:26:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.142+0000 7f1280bb1740 20 client.15729 populate_metadata read hostname 'smithi023' 2024-03-28T05:26:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.142+0000 7f1280bb1740 1 --2- 172.21.15.23:0/2766011910 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55e3b0985920 0x55e3b0987d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.142+0000 7f1280bb1740 4 client.15729 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:26:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.142+0000 7f1280bb1740 1 -- 172.21.15.23:0/2766011910 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e3b08d5050 con 0x55e3b0985920 2024-03-28T05:26:43.145 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.143+0000 7f12777fe640 1 --2- 172.21.15.23:0/2766011910 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55e3b0985920 0x55e3b0987d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.143+0000 7f1256ffd640 10 client.15729 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:26:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.143+0000 7f12777fe640 1 --2- 172.21.15.23:0/2766011910 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55e3b0985920 0x55e3b0987d10 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11d459945139365e server_cookie=2947fe41c5ebb113 in_seq=0 out_seq=0 2024-03-28T05:26:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.145+0000 7f1256ffd640 1 -- 172.21.15.23:0/2766011910 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e3b08d5050 con 0x55e3b0985920 2024-03-28T05:26:43.147 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.145+0000 7f1256ffd640 10 client.15729 handle_command_reply: tid=0 2024-03-28T05:26:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.145+0000 7f1280bb1740 1 client.15729 shutdown 2024-03-28T05:26:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.145+0000 7f1280bb1740 1 -- 172.21.15.23:0/2766011910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b0963b10 msgr2=0x55e3b0965df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.145+0000 7f1280bb1740 1 --2- 172.21.15.23:0/2766011910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b0963b10 0x55e3b0965df0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f126c00e9c0 tx=0x7f126c0d2c70 comp rx=0 tx=0).stop 2024-03-28T05:26:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.145+0000 7f1280bb1740 1 -- 172.21.15.23:0/2766011910 shutdown_connections 2024-03-28T05:26:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.145+0000 7f1280bb1740 1 --2- 172.21.15.23:0/2766011910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e3b09754a0 0x55e3b0966330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.145+0000 7f1280bb1740 1 --2- 172.21.15.23:0/2766011910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b0963b10 0x55e3b0965df0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.145+0000 7f1280bb1740 1 --2- 172.21.15.23:0/2766011910 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55e3b0985920 0x55e3b0987d10 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.145+0000 7f1280bb1740 1 --2- 172.21.15.23:0/2766011910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3b0963160 0x55e3b09658b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.145+0000 7f1280bb1740 1 -- 172.21.15.23:0/2766011910 >> 172.21.15.23:0/2766011910 conn(0x55e3b095e030 msgr2=0x55e3b095fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.145+0000 7f1256ffd640 0 client.15729 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:26:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.145+0000 7f1280bb1740 1 -- 172.21.15.23:0/2766011910 shutdown_connections 2024-03-28T05:26:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.145+0000 7f1280bb1740 1 -- 172.21.15.23:0/2766011910 wait complete. 2024-03-28T05:26:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.146+0000 7f1280bb1740 20 client.15729 trim_cache size 0 max 16384 2024-03-28T05:26:43.171 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.164+0000 7f1280bb1740 1 Processor -- start 2024-03-28T05:26:43.171 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.164+0000 7f1280bb1740 1 -- start start 2024-03-28T05:26:43.171 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.165+0000 7f1280bb1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3b0963160 0x55e3b0980aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:43.171 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.165+0000 7f1280bb1740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e3b0963b10 0x55e3b0980fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:43.172 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.165+0000 7f1280bb1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b09754a0 0x55e3b0965870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:43.172 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.165+0000 7f1280bb1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12781aa990 con 0x55e3b09754a0 2024-03-28T05:26:43.172 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.165+0000 7f1280bb1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12781acf50 con 0x55e3b0963160 2024-03-28T05:26:43.172 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.165+0000 7f1280bb1740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12781aaf90 con 0x55e3b0963b10 2024-03-28T05:26:43.172 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.165+0000 7f1277fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b09754a0 0x55e3b0965870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:43.172 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.165+0000 7f1277fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b09754a0 0x55e3b0965870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60774/0 (socket says 172.21.15.23:60774) 2024-03-28T05:26:43.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.165+0000 7f1277fff640 1 -- 172.21.15.23:0/1921559839 learned_addr learned my addr 172.21.15.23:0/1921559839 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:43.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.165+0000 7f1276ffd640 1 --2- 172.21.15.23:0/1921559839 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e3b0963b10 0x55e3b0980fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:43.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.165+0000 7f12777fe640 1 --2- 172.21.15.23:0/1921559839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3b0963160 0x55e3b0980aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:43.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.165+0000 7f1277fff640 1 -- 172.21.15.23:0/1921559839 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e3b0963b10 msgr2=0x55e3b0980fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:43.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.165+0000 7f1277fff640 1 --2- 172.21.15.23:0/1921559839 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e3b0963b10 0x55e3b0980fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.165+0000 7f1277fff640 1 -- 172.21.15.23:0/1921559839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3b0963160 msgr2=0x55e3b0980aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:43.173 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.165+0000 7f1277fff640 1 --2- 172.21.15.23:0/1921559839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3b0963160 0x55e3b0980aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.165+0000 7f1277fff640 1 -- 172.21.15.23:0/1921559839 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3b0962b40 con 0x55e3b09754a0 2024-03-28T05:26:43.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.165+0000 7f1277fff640 1 --2- 172.21.15.23:0/1921559839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b09754a0 0x55e3b0965870 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1270010d60 tx=0x7f1270005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=756b583e5f313e41 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:43.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.165+0000 7f125d7fa640 1 -- 172.21.15.23:0/1921559839 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1270028020 con 0x55e3b09754a0 2024-03-28T05:26:43.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.166+0000 7f125d7fa640 1 -- 172.21.15.23:0/1921559839 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1270019b90 con 0x55e3b09754a0 2024-03-28T05:26:43.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.166+0000 7f125d7fa640 1 -- 172.21.15.23:0/1921559839 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1270011500 con 0x55e3b09754a0 2024-03-28T05:26:43.174 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.166+0000 7f1280bb1740 1 -- 172.21.15.23:0/1921559839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b09754a0 msgr2=0x55e3b0965870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:43.175 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.166+0000 7f1280bb1740 1 --2- 172.21.15.23:0/1921559839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b09754a0 0x55e3b0965870 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1270010d60 tx=0x7f1270005bd0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.175 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.166+0000 7f1280bb1740 1 -- 172.21.15.23:0/1921559839 shutdown_connections 2024-03-28T05:26:43.175 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.166+0000 7f1280bb1740 1 --2- 172.21.15.23:0/1921559839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b09754a0 0x55e3b0965870 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.175 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.166+0000 7f1280bb1740 1 --2- 172.21.15.23:0/1921559839 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e3b0963b10 0x55e3b0980fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.175 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.166+0000 7f1280bb1740 1 --2- 172.21.15.23:0/1921559839 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3b0963160 0x55e3b0980aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.175 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.166+0000 7f1280bb1740 1 -- 172.21.15.23:0/1921559839 >> 172.21.15.23:0/1921559839 conn(0x55e3b095e030 msgr2=0x55e3b0987930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:43.175 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.166+0000 7f1280bb1740 1 -- 172.21.15.23:0/1921559839 shutdown_connections 2024-03-28T05:26:43.175 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.166+0000 7f1280bb1740 1 -- 172.21.15.23:0/1921559839 wait complete. 2024-03-28T05:26:43.175 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.167+0000 7f1280bb1740 1 -- start start 2024-03-28T05:26:43.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.167+0000 7f1280bb1740 1 Processor -- start 2024-03-28T05:26:43.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.167+0000 7f1280bb1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3b0963160 0x55e3b0976450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:43.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.167+0000 7f1280bb1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b0963b10 0x55e3b0976990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:43.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.167+0000 7f1280bb1740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e3b09754a0 0x55e3b097c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:43.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.167+0000 7f1280bb1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12781ab5d0 con 0x55e3b0963b10 2024-03-28T05:26:43.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.167+0000 7f1280bb1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12781ac8f0 con 0x55e3b0963160 2024-03-28T05:26:43.176 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.167+0000 7f1280bb1740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12781abc30 con 0x55e3b09754a0 2024-03-28T05:26:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.167+0000 7f1280bb1740 10 client.0 _ll_register_callbacks cb 0x7f126c00cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:26:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.167+0000 7f1276ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b0963b10 0x55e3b0976990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.168+0000 7f1276ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b0963b10 0x55e3b0976990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60782/0 (socket says 172.21.15.23:60782) 2024-03-28T05:26:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.168+0000 7f1276ffd640 1 -- 172.21.15.23:0/2894595267 learned_addr learned my addr 172.21.15.23:0/2894595267 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.168+0000 7f1277fff640 1 --2- 172.21.15.23:0/2894595267 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e3b09754a0 0x55e3b097c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:43.177 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.168+0000 7f12777fe640 1 --2- 172.21.15.23:0/2894595267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3b0963160 0x55e3b0976450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.168+0000 7f1276ffd640 1 -- 172.21.15.23:0/2894595267 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e3b09754a0 msgr2=0x55e3b097c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.168+0000 7f1276ffd640 1 --2- 172.21.15.23:0/2894595267 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e3b09754a0 0x55e3b097c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.168+0000 7f1276ffd640 1 -- 172.21.15.23:0/2894595267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3b0963160 msgr2=0x55e3b0976450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.168+0000 7f1276ffd640 1 --2- 172.21.15.23:0/2894595267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3b0963160 0x55e3b0976450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.168+0000 7f1276ffd640 1 -- 172.21.15.23:0/2894595267 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e3b097cb20 con 0x55e3b0963b10 2024-03-28T05:26:43.178 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.168+0000 7f1276ffd640 1 --2- 172.21.15.23:0/2894595267 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b0963b10 0x55e3b0976990 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f126c00c6a0 tx=0x7f126c0cf410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0f64d2a70e3f2d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.168+0000 7f1256ffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:26:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.168+0000 7f1256ffd640 1 -- 172.21.15.23:0/2894595267 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126c09b220 con 0x55e3b0963b10 2024-03-28T05:26:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.168+0000 7f1256ffd640 1 -- 172.21.15.23:0/2894595267 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f126c0b45a0 con 0x55e3b0963b10 2024-03-28T05:26:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.168+0000 7f1280bb1740 1 -- 172.21.15.23:0/2894595267 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1270019b90 con 0x55e3b0963b10 2024-03-28T05:26:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.168+0000 7f1256ffd640 1 -- 172.21.15.23:0/2894595267 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f126c09b690 con 0x55e3b0963b10 2024-03-28T05:26:43.179 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.168+0000 7f1280bb1740 10 client.15735 fetch_fsmap learned FSMap version 20 2024-03-28T05:26:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.168+0000 7f1280bb1740 1 -- 172.21.15.23:0/2894595267 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55e3b09e1320 con 0x55e3b0963b10 2024-03-28T05:26:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.168+0000 7f1256ffd640 1 -- 172.21.15.23:0/2894595267 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1270019b90 con 0x55e3b0963b10 2024-03-28T05:26:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.169+0000 7f1256ffd640 1 -- 172.21.15.23:0/2894595267 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f126c0b4160 con 0x55e3b0963b10 2024-03-28T05:26:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.169+0000 7f1280bb1740 10 client.15735 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:26:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.169+0000 7f1280bb1740 10 client.15735 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:26:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.169+0000 7f1280bb1740 20 client.15735 populate_metadata read hostname 'smithi023' 2024-03-28T05:26:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.169+0000 7f1280bb1740 1 --2- 172.21.15.23:0/2894595267 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55e3b09e1880 0x55e3b09e3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.169+0000 7f1280bb1740 4 client.15735 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:26:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.169+0000 7f1280bb1740 1 -- 172.21.15.23:0/2894595267 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e3b08cf3a0 con 0x55e3b09e1880 2024-03-28T05:26:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.169+0000 7f12777fe640 1 --2- 172.21.15.23:0/2894595267 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55e3b09e1880 0x55e3b09e3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.169+0000 7f12777fe640 1 --2- 172.21.15.23:0/2894595267 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55e3b09e1880 0x55e3b09e3c70 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=899158e22481affb server_cookie=ad95efb76704f0ca in_seq=0 out_seq=0 2024-03-28T05:26:43.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.169+0000 7f1256ffd640 10 client.15735 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:26:43.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:42 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:26:43.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:42 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:26:43.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.706+0000 7f1256ffd640 1 -- 172.21.15.23:0/2894595267 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e3b08cf3a0 con 0x55e3b09e1880 2024-03-28T05:26:43.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.706+0000 7f1256ffd640 10 client.15735 handle_command_reply: tid=0 2024-03-28T05:26:43.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.706+0000 7f1280bb1740 1 client.15735 shutdown 2024-03-28T05:26:43.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.706+0000 7f1280bb1740 1 -- 172.21.15.23:0/2894595267 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b0963b10 msgr2=0x55e3b0976990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:43.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.706+0000 7f1280bb1740 1 --2- 172.21.15.23:0/2894595267 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b0963b10 0x55e3b0976990 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f126c00c6a0 tx=0x7f126c0cf410 comp rx=0 tx=0).stop 2024-03-28T05:26:43.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.706+0000 7f1280bb1740 1 -- 172.21.15.23:0/2894595267 shutdown_connections 2024-03-28T05:26:43.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.706+0000 7f1280bb1740 1 --2- 172.21.15.23:0/2894595267 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e3b09754a0 0x55e3b097c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.706+0000 7f1280bb1740 1 --2- 172.21.15.23:0/2894595267 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e3b0963b10 0x55e3b0976990 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.706+0000 7f1280bb1740 1 --2- 172.21.15.23:0/2894595267 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55e3b09e1880 0x55e3b09e3c70 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.706+0000 7f1280bb1740 1 --2- 172.21.15.23:0/2894595267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e3b0963160 0x55e3b0976450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.706+0000 7f1256ffd640 0 client.15735 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:26:43.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.706+0000 7f1280bb1740 1 -- 172.21.15.23:0/2894595267 >> 172.21.15.23:0/2894595267 conn(0x55e3b095e030 msgr2=0x55e3b0966100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:43.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.706+0000 7f1280bb1740 1 -- 172.21.15.23:0/2894595267 shutdown_connections 2024-03-28T05:26:43.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.707+0000 7f1280bb1740 1 -- 172.21.15.23:0/2894595267 wait complete. 2024-03-28T05:26:43.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.707+0000 7f1280bb1740 20 client.15735 trim_cache size 0 max 16384 2024-03-28T05:26:43.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:26:43.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:26:43.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:26:43.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:26:43.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.707+0000 7f127deea640 1 -- 172.21.15.23:0/2100928050 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f12600a2a10 msgr2=0x7f12600a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:43.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.707+0000 7f127deea640 1 --2- 172.21.15.23:0/2100928050 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f12600a2a10 0x7f12600a4ed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1268009230 tx=0x7f1268005dd0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.707+0000 7f127deea640 1 -- 172.21.15.23:0/2100928050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12781011b0 msgr2=0x7f1278199300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:43.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.707+0000 7f127deea640 1 --2- 172.21.15.23:0/2100928050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12781011b0 0x7f1278199300 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f126c00e9f0 tx=0x7f126c00eeb0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.707+0000 7f127deea640 1 -- 172.21.15.23:0/2100928050 shutdown_connections 2024-03-28T05:26:43.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.707+0000 7f127deea640 1 --2- 172.21.15.23:0/2100928050 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1278105540 0x7f12781a3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.707+0000 7f127deea640 1 --2- 172.21.15.23:0/2100928050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12781011b0 0x7f1278199300 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.707+0000 7f127deea640 1 --2- 172.21.15.23:0/2100928050 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f12600a2a10 0x7f12600a4ed0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.707+0000 7f127deea640 1 --2- 172.21.15.23:0/2100928050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12781007f0 0x7f1278198dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:43.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.707+0000 7f127deea640 1 -- 172.21.15.23:0/2100928050 >> 172.21.15.23:0/2100928050 conn(0x7f12780773f0 msgr2=0x7f12780fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:43.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.707+0000 7f127deea640 1 -- 172.21.15.23:0/2100928050 shutdown_connections 2024-03-28T05:26:43.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:43.707+0000 7f127deea640 1 -- 172.21.15.23:0/2100928050 wait complete. 2024-03-28T05:26:44.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:43 smithi098 ceph-mon[36087]: pgmap v97: 129 pgs: 129 active+clean; 80 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 599 KiB/s wr, 99 op/s 2024-03-28T05:26:44.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:43 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:26:44.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:43 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:26:44.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:43 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:26:44.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:43 smithi113 ceph-mon[37164]: pgmap v97: 129 pgs: 129 active+clean; 80 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 599 KiB/s wr, 99 op/s 2024-03-28T05:26:44.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:43 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:26:44.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:43 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:26:44.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:43 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:26:44.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:43 smithi023 ceph-mon[31954]: pgmap v97: 129 pgs: 129 active+clean; 80 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 599 KiB/s wr, 99 op/s 2024-03-28T05:26:44.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:43 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:26:44.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:43 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:26:44.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:43 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:26:46.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:45 smithi113 ceph-mon[37164]: pgmap v98: 129 pgs: 129 active+clean; 80 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 599 KiB/s wr, 99 op/s 2024-03-28T05:26:46.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:45 smithi023 ceph-mon[31954]: pgmap v98: 129 pgs: 129 active+clean; 80 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 599 KiB/s wr, 99 op/s 2024-03-28T05:26:46.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:45 smithi098 ceph-mon[36087]: pgmap v98: 129 pgs: 129 active+clean; 80 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 599 KiB/s wr, 99 op/s 2024-03-28T05:26:46.399 INFO:tasks.workunit.client.0.smithi023.stdout: 26 126552 91 88263 120 54202 208 2024-03-28T05:26:48.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:47 smithi113 ceph-mon[37164]: pgmap v99: 129 pgs: 129 active+clean; 85 MiB data, 804 MiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 1.9 MiB/s wr, 509 op/s 2024-03-28T05:26:48.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:47 smithi023 ceph-mon[31954]: pgmap v99: 129 pgs: 129 active+clean; 85 MiB data, 804 MiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 1.9 MiB/s wr, 509 op/s 2024-03-28T05:26:48.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:47 smithi098 ceph-mon[36087]: pgmap v99: 129 pgs: 129 active+clean; 85 MiB data, 804 MiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 1.9 MiB/s wr, 509 op/s 2024-03-28T05:26:50.168 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:49 smithi098 ceph-mon[36087]: pgmap v100: 129 pgs: 129 active+clean; 85 MiB data, 804 MiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 1.5 MiB/s wr, 463 op/s 2024-03-28T05:26:50.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:49 smithi113 ceph-mon[37164]: pgmap v100: 129 pgs: 129 active+clean; 85 MiB data, 804 MiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 1.5 MiB/s wr, 463 op/s 2024-03-28T05:26:50.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:49 smithi023 ceph-mon[31954]: pgmap v100: 129 pgs: 129 active+clean; 85 MiB data, 804 MiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 1.5 MiB/s wr, 463 op/s 2024-03-28T05:26:52.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:51 smithi113 ceph-mon[37164]: pgmap v101: 129 pgs: 129 active+clean; 86 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 1.6 MiB/s wr, 480 op/s 2024-03-28T05:26:52.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:51 smithi023 ceph-mon[31954]: pgmap v101: 129 pgs: 129 active+clean; 86 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 1.6 MiB/s wr, 480 op/s 2024-03-28T05:26:52.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:51 smithi098 ceph-mon[36087]: pgmap v101: 129 pgs: 129 active+clean; 86 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 1.6 MiB/s wr, 480 op/s 2024-03-28T05:26:54.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:53 smithi113 ceph-mon[37164]: pgmap v102: 129 pgs: 129 active+clean; 98 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 2.5 MiB/s wr, 525 op/s 2024-03-28T05:26:54.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:53 smithi023 ceph-mon[31954]: pgmap v102: 129 pgs: 129 active+clean; 98 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 2.5 MiB/s wr, 525 op/s 2024-03-28T05:26:54.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:53 smithi098 ceph-mon[36087]: pgmap v102: 129 pgs: 129 active+clean; 98 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 2.5 MiB/s wr, 525 op/s 2024-03-28T05:26:56.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:55 smithi113 ceph-mon[37164]: pgmap v103: 129 pgs: 129 active+clean; 98 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 2.5 MiB/s wr, 525 op/s 2024-03-28T05:26:56.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:55 smithi023 ceph-mon[31954]: pgmap v103: 129 pgs: 129 active+clean; 98 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 2.5 MiB/s wr, 525 op/s 2024-03-28T05:26:56.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:55 smithi098 ceph-mon[36087]: pgmap v103: 129 pgs: 129 active+clean; 98 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 2.5 MiB/s wr, 525 op/s 2024-03-28T05:26:56.399 INFO:tasks.workunit.client.0.smithi023.stdout: 30 385883 205 276762 148 164571 364 2024-03-28T05:26:56.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.800+0000 7f9dffbd4640 1 -- 172.21.15.23:0/2977347372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9df81007f0 msgr2=0x7f9df8100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:56.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.800+0000 7f9dffbd4640 1 --2- 172.21.15.23:0/2977347372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9df81007f0 0x7f9df8100c70 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009900 tx=0x7f9de802bc30 comp rx=0 tx=0).stop 2024-03-28T05:26:56.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.800+0000 7f9dffbd4640 1 -- 172.21.15.23:0/2977347372 shutdown_connections 2024-03-28T05:26:56.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.800+0000 7f9dffbd4640 1 --2- 172.21.15.23:0/2977347372 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9df81011b0 0x7f9df810d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.800+0000 7f9dffbd4640 1 --2- 172.21.15.23:0/2977347372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9df81007f0 0x7f9df8100c70 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.800+0000 7f9dffbd4640 1 --2- 172.21.15.23:0/2977347372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9df8105540 0x7f9df8105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.800+0000 7f9dffbd4640 1 -- 172.21.15.23:0/2977347372 >> 172.21.15.23:0/2977347372 conn(0x7f9df80773f0 msgr2=0x7f9df8077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:56.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.801+0000 7f9dffbd4640 1 -- 172.21.15.23:0/2977347372 shutdown_connections 2024-03-28T05:26:56.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.801+0000 7f9dffbd4640 1 -- 172.21.15.23:0/2977347372 wait complete. 2024-03-28T05:26:56.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.801+0000 7f9dffbd4640 1 Processor -- start 2024-03-28T05:26:56.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.801+0000 7f9dffbd4640 1 -- start start 2024-03-28T05:26:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.801+0000 7f9dffbd4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9df81007f0 0x7f9df8198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.801+0000 7f9dffbd4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9df81011b0 0x7f9df8198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.801+0000 7f9dffbd4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9df8105540 0x7f9df819c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.801+0000 7f9dffbd4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df8110a60 con 0x7f9df81011b0 2024-03-28T05:26:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.801+0000 7f9dffbd4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df81108e0 con 0x7f9df8105540 2024-03-28T05:26:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.801+0000 7f9dffbd4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df8110be0 con 0x7f9df81007f0 2024-03-28T05:26:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.801+0000 7f9dfd16f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9df81011b0 0x7f9df8198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.802+0000 7f9dfd16f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9df81011b0 0x7f9df8198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45744/0 (socket says 172.21.15.23:45744) 2024-03-28T05:26:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.802+0000 7f9dfd16f640 1 -- 172.21.15.23:0/37276450 learned_addr learned my addr 172.21.15.23:0/37276450 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.802+0000 7f9dfd970640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9df81007f0 0x7f9df8198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.802+0000 7f9dfe171640 1 --2- 172.21.15.23:0/37276450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9df8105540 0x7f9df819c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:56.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.802+0000 7f9dfd16f640 1 -- 172.21.15.23:0/37276450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9df81007f0 msgr2=0x7f9df8198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.802+0000 7f9dfd16f640 1 --2- 172.21.15.23:0/37276450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9df81007f0 0x7f9df8198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.802+0000 7f9dfd16f640 1 -- 172.21.15.23:0/37276450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9df8105540 msgr2=0x7f9df819c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.802+0000 7f9dfd16f640 1 --2- 172.21.15.23:0/37276450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9df8105540 0x7f9df819c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.802+0000 7f9dfd16f640 1 -- 172.21.15.23:0/37276450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df819cfa0 con 0x7f9df81011b0 2024-03-28T05:26:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.802+0000 7f9dfd16f640 1 --2- 172.21.15.23:0/37276450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9df81011b0 0x7f9df8198540 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9de8005940 tx=0x7f9de80043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.802+0000 7f9deeffd640 1 -- 172.21.15.23:0/37276450 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de8041070 con 0x7f9df81011b0 2024-03-28T05:26:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.802+0000 7f9deeffd640 1 -- 172.21.15.23:0/37276450 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9de802ccc0 con 0x7f9df81011b0 2024-03-28T05:26:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.802+0000 7f9dffbd4640 1 -- 172.21.15.23:0/37276450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df819d230 con 0x7f9df81011b0 2024-03-28T05:26:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.802+0000 7f9deeffd640 1 -- 172.21.15.23:0/37276450 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de8003db0 con 0x7f9df81011b0 2024-03-28T05:26:56.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.802+0000 7f9dffbd4640 1 -- 172.21.15.23:0/37276450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df81a4b20 con 0x7f9df81011b0 2024-03-28T05:26:56.814 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.805+0000 7f9deeffd640 1 -- 172.21.15.23:0/37276450 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f9de8044070 con 0x7f9df81011b0 2024-03-28T05:26:56.814 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.805+0000 7f9e027c2740 1 Processor -- start 2024-03-28T05:26:56.814 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.805+0000 7f9e027c2740 1 -- start start 2024-03-28T05:26:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.805+0000 7f9e027c2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8a3710 0x5560fa8a3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.805+0000 7f9e027c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560fa8a4030 0x5560fa8ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.805+0000 7f9e027c2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5560fa8ace00 0x5560fa8af1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.805+0000 7f9e027c2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560fa807b10 con 0x5560fa8a3710 2024-03-28T05:26:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.805+0000 7f9e027c2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560fa7b0900 con 0x5560fa8a4030 2024-03-28T05:26:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.805+0000 7f9e027c2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560fa803bb0 con 0x5560fa8ace00 2024-03-28T05:26:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.806+0000 7f9dfe171640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8a3710 0x5560fa8a3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.806+0000 7f9dfe171640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8a3710 0x5560fa8a3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45756/0 (socket says 172.21.15.23:45756) 2024-03-28T05:26:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.806+0000 7f9dfe171640 1 -- 172.21.15.23:0/2692552982 learned_addr learned my addr 172.21.15.23:0/2692552982 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.806+0000 7f9dfd16f640 1 --2- 172.21.15.23:0/2692552982 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5560fa8ace00 0x5560fa8af1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.806+0000 7f9dfd970640 1 --2- 172.21.15.23:0/2692552982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560fa8a4030 0x5560fa8ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.806+0000 7f9deeffd640 1 --2- 172.21.15.23:0/37276450 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f9de40a2a10 0x7f9de40a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.806+0000 7f9dfd16f640 1 -- 172.21.15.23:0/2692552982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560fa8a4030 msgr2=0x5560fa8ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.806+0000 7f9dfd16f640 1 --2- 172.21.15.23:0/2692552982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560fa8a4030 0x5560fa8ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.806+0000 7f9dfd16f640 1 -- 172.21.15.23:0/2692552982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8a3710 msgr2=0x5560fa8a3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.806+0000 7f9dfd16f640 1 --2- 172.21.15.23:0/2692552982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8a3710 0x5560fa8a3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.806+0000 7f9dfd16f640 1 -- 172.21.15.23:0/2692552982 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560fa51c210 con 0x5560fa8ace00 2024-03-28T05:26:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.806+0000 7f9dfe171640 1 --2- 172.21.15.23:0/2692552982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8a3710 0x5560fa8a3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:26:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.806+0000 7f9dfe171640 1 --2- 172.21.15.23:0/37276450 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f9de40a2a10 0x7f9de40a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.806+0000 7f9deeffd640 1 -- 172.21.15.23:0/37276450 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f9de80f0f50 con 0x7f9df81011b0 2024-03-28T05:26:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.806+0000 7f9dfd16f640 1 --2- 172.21.15.23:0/2692552982 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5560fa8ace00 0x5560fa8af1f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9de80ee420 tx=0x7f9de804e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de56521170f6221c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.806+0000 7f9dfe171640 1 --2- 172.21.15.23:0/37276450 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f9de40a2a10 0x7f9de40a4ed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9df0005f40 tx=0x7f9df0004a40 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.806+0000 7f9de3fff640 1 -- 172.21.15.23:0/2692552982 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de80bce70 con 0x5560fa8ace00 2024-03-28T05:26:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.806+0000 7f9de3fff640 1 -- 172.21.15.23:0/2692552982 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9de8004c80 con 0x5560fa8ace00 2024-03-28T05:26:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.807+0000 7f9de3fff640 1 -- 172.21.15.23:0/2692552982 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de80bcc90 con 0x5560fa8ace00 2024-03-28T05:26:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.807+0000 7f9e027c2740 1 -- 172.21.15.23:0/2692552982 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5560fa8ace00 msgr2=0x5560fa8af1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.807+0000 7f9e027c2740 1 --2- 172.21.15.23:0/2692552982 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5560fa8ace00 0x5560fa8af1f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9de80ee420 tx=0x7f9de804e000 comp rx=0 tx=0).stop 2024-03-28T05:26:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.807+0000 7f9e027c2740 1 -- 172.21.15.23:0/2692552982 shutdown_connections 2024-03-28T05:26:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.807+0000 7f9e027c2740 1 --2- 172.21.15.23:0/2692552982 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5560fa8ace00 0x5560fa8af1f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.807+0000 7f9e027c2740 1 --2- 172.21.15.23:0/2692552982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560fa8a4030 0x5560fa8ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.807+0000 7f9e027c2740 1 --2- 172.21.15.23:0/2692552982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8a3710 0x5560fa8a3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.807+0000 7f9e027c2740 1 -- 172.21.15.23:0/2692552982 >> 172.21.15.23:0/2692552982 conn(0x5560fa89e030 msgr2=0x5560fa89e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.807+0000 7f9e027c2740 1 -- 172.21.15.23:0/2692552982 shutdown_connections 2024-03-28T05:26:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.807+0000 7f9e027c2740 1 -- 172.21.15.23:0/2692552982 wait complete. 2024-03-28T05:26:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.808+0000 7f9e027c2740 1 -- start start 2024-03-28T05:26:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.808+0000 7f9e027c2740 1 Processor -- start 2024-03-28T05:26:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.808+0000 7f9e027c2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5560fa8a3710 0x5560fa8a9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.808+0000 7f9e027c2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8a4030 0x5560fa8aa380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.808+0000 7f9e027c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560fa8ace00 0x5560fa8aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.808+0000 7f9e027c2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560fa803bb0 con 0x5560fa8a4030 2024-03-28T05:26:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.808+0000 7f9e027c2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560fa7c3d50 con 0x5560fa8ace00 2024-03-28T05:26:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.808+0000 7f9e027c2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560fa8af7f0 con 0x5560fa8a3710 2024-03-28T05:26:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.808+0000 7f9e027c2740 10 client.0 _ll_register_callbacks cb 0x5560fa7c3720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:26:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.808+0000 7f9dfd16f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8a4030 0x5560fa8aa380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.808+0000 7f9dfd16f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8a4030 0x5560fa8aa380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45762/0 (socket says 172.21.15.23:45762) 2024-03-28T05:26:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.808+0000 7f9dfd16f640 1 -- 172.21.15.23:0/2312125934 learned_addr learned my addr 172.21.15.23:0/2312125934 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.808+0000 7f9dfd970640 1 --2- 172.21.15.23:0/2312125934 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5560fa8a3710 0x5560fa8a9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.808+0000 7f9dfe171640 1 --2- 172.21.15.23:0/2312125934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560fa8ace00 0x5560fa8aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.809+0000 7f9dfd16f640 1 -- 172.21.15.23:0/2312125934 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5560fa8a3710 msgr2=0x5560fa8a9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.809+0000 7f9dfd16f640 1 --2- 172.21.15.23:0/2312125934 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5560fa8a3710 0x5560fa8a9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.809+0000 7f9dfd16f640 1 -- 172.21.15.23:0/2312125934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560fa8ace00 msgr2=0x5560fa8aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.809+0000 7f9dfd16f640 1 --2- 172.21.15.23:0/2312125934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560fa8ace00 0x5560fa8aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.809+0000 7f9dfd16f640 1 -- 172.21.15.23:0/2312125934 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5560fa5e8620 con 0x5560fa8a4030 2024-03-28T05:26:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.809+0000 7f9dfd16f640 1 --2- 172.21.15.23:0/2312125934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8a4030 0x5560fa8aa380 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f9de80ee4d0 tx=0x7f9de80bd200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88d149b2a53903a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.809+0000 7f9de17fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:26:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.809+0000 7f9de17fa640 1 -- 172.21.15.23:0/2312125934 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de80bcc90 con 0x5560fa8a4030 2024-03-28T05:26:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.809+0000 7f9de17fa640 1 -- 172.21.15.23:0/2312125934 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f9de80035f0 con 0x5560fa8a4030 2024-03-28T05:26:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.809+0000 7f9e027c2740 1 -- 172.21.15.23:0/2312125934 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9de8004c80 con 0x5560fa8a4030 2024-03-28T05:26:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.809+0000 7f9de17fa640 1 -- 172.21.15.23:0/2312125934 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9de804b020 con 0x5560fa8a4030 2024-03-28T05:26:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.809+0000 7f9e027c2740 10 client.15747 fetch_fsmap learned FSMap version 20 2024-03-28T05:26:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.809+0000 7f9e027c2740 1 -- 172.21.15.23:0/2312125934 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5560fa8bb8f0 con 0x5560fa8a4030 2024-03-28T05:26:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.809+0000 7f9de17fa640 1 -- 172.21.15.23:0/2312125934 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9de8004c80 con 0x5560fa8a4030 2024-03-28T05:26:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.809+0000 7f9de17fa640 1 -- 172.21.15.23:0/2312125934 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f9de80b99a0 con 0x5560fa8a4030 2024-03-28T05:26:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.810+0000 7f9e027c2740 10 client.15747 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:26:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.810+0000 7f9e027c2740 10 client.15747 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:26:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.810+0000 7f9e027c2740 20 client.15747 populate_metadata read hostname 'smithi023' 2024-03-28T05:26:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.810+0000 7f9e027c2740 1 --2- 172.21.15.23:0/2312125934 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5560fa8bca70 0x5560fa8bee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.810+0000 7f9e027c2740 4 client.15747 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:26:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.810+0000 7f9e027c2740 1 -- 172.21.15.23:0/2312125934 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5560fa815050 con 0x5560fa8bca70 2024-03-28T05:26:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.810+0000 7f9dfd970640 1 --2- 172.21.15.23:0/2312125934 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5560fa8bca70 0x5560fa8bee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.810+0000 7f9de17fa640 10 client.15747 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:26:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.810+0000 7f9dfd970640 1 --2- 172.21.15.23:0/2312125934 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5560fa8bca70 0x5560fa8bee60 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=864ddcc02a7a64ce server_cookie=535c3a7eca9941d1 in_seq=0 out_seq=0 2024-03-28T05:26:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.812+0000 7f9de17fa640 1 -- 172.21.15.23:0/2312125934 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5560fa815050 con 0x5560fa8bca70 2024-03-28T05:26:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.812+0000 7f9de17fa640 10 client.15747 handle_command_reply: tid=0 2024-03-28T05:26:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.812+0000 7f9e027c2740 1 client.15747 shutdown 2024-03-28T05:26:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.812+0000 7f9e027c2740 1 -- 172.21.15.23:0/2312125934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8a4030 msgr2=0x5560fa8aa380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.812+0000 7f9e027c2740 1 --2- 172.21.15.23:0/2312125934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8a4030 0x5560fa8aa380 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f9de80ee4d0 tx=0x7f9de80bd200 comp rx=0 tx=0).stop 2024-03-28T05:26:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.812+0000 7f9e027c2740 1 -- 172.21.15.23:0/2312125934 shutdown_connections 2024-03-28T05:26:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.812+0000 7f9e027c2740 1 --2- 172.21.15.23:0/2312125934 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5560fa8bca70 0x5560fa8bee60 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.812+0000 7f9e027c2740 1 --2- 172.21.15.23:0/2312125934 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560fa8ace00 0x5560fa8aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.812+0000 7f9e027c2740 1 --2- 172.21.15.23:0/2312125934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8a4030 0x5560fa8aa380 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.812+0000 7f9e027c2740 1 --2- 172.21.15.23:0/2312125934 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5560fa8a3710 0x5560fa8a9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.812+0000 7f9de17fa640 0 client.15747 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:26:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.812+0000 7f9e027c2740 1 -- 172.21.15.23:0/2312125934 >> 172.21.15.23:0/2312125934 conn(0x5560fa89e030 msgr2=0x5560fa89fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.812+0000 7f9e027c2740 1 -- 172.21.15.23:0/2312125934 shutdown_connections 2024-03-28T05:26:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.812+0000 7f9e027c2740 1 -- 172.21.15.23:0/2312125934 wait complete. 2024-03-28T05:26:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.812+0000 7f9e027c2740 20 client.15747 trim_cache size 0 max 16384 2024-03-28T05:26:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.826+0000 7f9e027c2740 1 Processor -- start 2024-03-28T05:26:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.826+0000 7f9e027c2740 1 -- start start 2024-03-28T05:26:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.826+0000 7f9e027c2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5560fa8a3710 0x5560fa8ac570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.826+0000 7f9e027c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560fa8a4030 0x5560fa8b7f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.826+0000 7f9e027c2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8ace00 0x5560fa8b8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.826+0000 7f9e027c2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df81ac9f0 con 0x5560fa8ace00 2024-03-28T05:26:56.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.826+0000 7f9e027c2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df81aefb0 con 0x5560fa8a4030 2024-03-28T05:26:56.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.826+0000 7f9e027c2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df81acff0 con 0x5560fa8a3710 2024-03-28T05:26:56.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.827+0000 7f9dfe171640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8ace00 0x5560fa8b8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:56.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.827+0000 7f9dfe171640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8ace00 0x5560fa8b8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45764/0 (socket says 172.21.15.23:45764) 2024-03-28T05:26:56.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.827+0000 7f9dfe171640 1 -- 172.21.15.23:0/56532253 learned_addr learned my addr 172.21.15.23:0/56532253 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:56.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.827+0000 7f9dfd970640 1 --2- 172.21.15.23:0/56532253 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5560fa8a3710 0x5560fa8ac570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:56.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.827+0000 7f9dfd16f640 1 --2- 172.21.15.23:0/56532253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560fa8a4030 0x5560fa8b7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:56.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.827+0000 7f9dfe171640 1 -- 172.21.15.23:0/56532253 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5560fa8a3710 msgr2=0x5560fa8ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:56.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.827+0000 7f9dfe171640 1 --2- 172.21.15.23:0/56532253 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5560fa8a3710 0x5560fa8ac570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.827+0000 7f9dfe171640 1 -- 172.21.15.23:0/56532253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560fa8a4030 msgr2=0x5560fa8b7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:56.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.827+0000 7f9dfe171640 1 --2- 172.21.15.23:0/56532253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560fa8a4030 0x5560fa8b7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.827+0000 7f9dfe171640 1 -- 172.21.15.23:0/56532253 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560fa8bf1d0 con 0x5560fa8ace00 2024-03-28T05:26:56.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.827+0000 7f9dfd970640 1 --2- 172.21.15.23:0/56532253 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5560fa8a3710 0x5560fa8ac570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:26:56.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.827+0000 7f9dfe171640 1 --2- 172.21.15.23:0/56532253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8ace00 0x5560fa8b8440 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f9df0011540 tx=0x7f9df0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8ee50a5a82bb0fe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:56.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.827+0000 7f9de37fe640 1 -- 172.21.15.23:0/56532253 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df0004b20 con 0x5560fa8ace00 2024-03-28T05:26:56.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.827+0000 7f9de37fe640 1 -- 172.21.15.23:0/56532253 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9df0004cc0 con 0x5560fa8ace00 2024-03-28T05:26:56.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.827+0000 7f9de37fe640 1 -- 172.21.15.23:0/56532253 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df00054c0 con 0x5560fa8ace00 2024-03-28T05:26:56.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.828+0000 7f9e027c2740 1 -- 172.21.15.23:0/56532253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8ace00 msgr2=0x5560fa8b8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:56.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.828+0000 7f9e027c2740 1 --2- 172.21.15.23:0/56532253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8ace00 0x5560fa8b8440 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f9df0011540 tx=0x7f9df0016000 comp rx=0 tx=0).stop 2024-03-28T05:26:56.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.828+0000 7f9e027c2740 1 -- 172.21.15.23:0/56532253 shutdown_connections 2024-03-28T05:26:56.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.828+0000 7f9e027c2740 1 --2- 172.21.15.23:0/56532253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8ace00 0x5560fa8b8440 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.828+0000 7f9e027c2740 1 --2- 172.21.15.23:0/56532253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560fa8a4030 0x5560fa8b7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.828+0000 7f9e027c2740 1 --2- 172.21.15.23:0/56532253 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5560fa8a3710 0x5560fa8ac570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.828+0000 7f9e027c2740 1 -- 172.21.15.23:0/56532253 >> 172.21.15.23:0/56532253 conn(0x5560fa89e030 msgr2=0x5560fa8a5180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:56.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.828+0000 7f9e027c2740 1 -- 172.21.15.23:0/56532253 shutdown_connections 2024-03-28T05:26:56.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.828+0000 7f9e027c2740 1 -- 172.21.15.23:0/56532253 wait complete. 2024-03-28T05:26:56.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.829+0000 7f9e027c2740 1 -- start start 2024-03-28T05:26:56.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.829+0000 7f9e027c2740 1 Processor -- start 2024-03-28T05:26:56.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.829+0000 7f9e027c2740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560fa8a3710 0x5560fa8b0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.829+0000 7f9e027c2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8a4030 0x5560fa8b0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.829+0000 7f9e027c2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5560fa8ace00 0x5560fa917250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.829+0000 7f9e027c2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df81ad630 con 0x5560fa8a4030 2024-03-28T05:26:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.829+0000 7f9e027c2740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df81ae950 con 0x5560fa8a3710 2024-03-28T05:26:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.829+0000 7f9e027c2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df81adc90 con 0x5560fa8ace00 2024-03-28T05:26:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.829+0000 7f9e027c2740 10 client.0 _ll_register_callbacks cb 0x7f9de80b9660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:26:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.829+0000 7f9dfd16f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8a4030 0x5560fa8b0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.829+0000 7f9dfd16f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8a4030 0x5560fa8b0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45778/0 (socket says 172.21.15.23:45778) 2024-03-28T05:26:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.829+0000 7f9dfd16f640 1 -- 172.21.15.23:0/1711017256 learned_addr learned my addr 172.21.15.23:0/1711017256 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:26:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.829+0000 7f9dfe171640 1 --2- 172.21.15.23:0/1711017256 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5560fa8ace00 0x5560fa917250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:56.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.829+0000 7f9dfd970640 1 --2- 172.21.15.23:0/1711017256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560fa8a3710 0x5560fa8b0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:56.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.829+0000 7f9dfd16f640 1 -- 172.21.15.23:0/1711017256 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5560fa8ace00 msgr2=0x5560fa917250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:56.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.829+0000 7f9dfd16f640 1 --2- 172.21.15.23:0/1711017256 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5560fa8ace00 0x5560fa917250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.829+0000 7f9dfd16f640 1 -- 172.21.15.23:0/1711017256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560fa8a3710 msgr2=0x5560fa8b0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:26:56.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.829+0000 7f9dfd16f640 1 --2- 172.21.15.23:0/1711017256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560fa8a3710 0x5560fa8b0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:56.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.830+0000 7f9dfd16f640 1 -- 172.21.15.23:0/1711017256 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5560fa917790 con 0x5560fa8a4030 2024-03-28T05:26:56.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.830+0000 7f9dfd16f640 1 --2- 172.21.15.23:0/1711017256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8a4030 0x5560fa8b0a40 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f9de8000c00 tx=0x7f9de8005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=239b80b25dcef146 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:26:56.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.830+0000 7f9de17fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:26:56.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.830+0000 7f9de17fa640 1 -- 172.21.15.23:0/1711017256 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de80d5030 con 0x5560fa8a4030 2024-03-28T05:26:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.830+0000 7f9de17fa640 1 -- 172.21.15.23:0/1711017256 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f9de80dd0b0 con 0x5560fa8a4030 2024-03-28T05:26:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.830+0000 7f9e027c2740 1 -- 172.21.15.23:0/1711017256 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9df0004cc0 con 0x5560fa8a4030 2024-03-28T05:26:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.830+0000 7f9de17fa640 1 -- 172.21.15.23:0/1711017256 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9de80bce70 con 0x5560fa8a4030 2024-03-28T05:26:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.830+0000 7f9e027c2740 10 client.15753 fetch_fsmap learned FSMap version 20 2024-03-28T05:26:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.830+0000 7f9e027c2740 1 -- 172.21.15.23:0/1711017256 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5560fa917790 con 0x5560fa8a4030 2024-03-28T05:26:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.830+0000 7f9de17fa640 1 -- 172.21.15.23:0/1711017256 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9df0004cc0 con 0x5560fa8a4030 2024-03-28T05:26:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.830+0000 7f9de17fa640 1 -- 172.21.15.23:0/1711017256 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f9de80c5640 con 0x5560fa8a4030 2024-03-28T05:26:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.830+0000 7f9e027c2740 10 client.15753 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:26:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.830+0000 7f9e027c2740 10 client.15753 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:26:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.830+0000 7f9e027c2740 20 client.15753 populate_metadata read hostname 'smithi023' 2024-03-28T05:26:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.830+0000 7f9e027c2740 1 --2- 172.21.15.23:0/1711017256 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5560fa918c50 0x5560fa91b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:26:56.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.830+0000 7f9e027c2740 4 client.15753 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:26:56.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.830+0000 7f9e027c2740 1 -- 172.21.15.23:0/1711017256 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5560fa80f3a0 con 0x5560fa918c50 2024-03-28T05:26:56.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.831+0000 7f9dfd970640 1 --2- 172.21.15.23:0/1711017256 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5560fa918c50 0x5560fa91b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:26:56.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.831+0000 7f9de17fa640 10 client.15753 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:26:56.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:56.831+0000 7f9dfd970640 1 --2- 172.21.15.23:0/1711017256 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5560fa918c50 0x5560fa91b040 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b6a566316f4b0c server_cookie=91b00da2b3e07974 in_seq=0 out_seq=0 2024-03-28T05:26:57.312 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.310+0000 7f9de17fa640 1 -- 172.21.15.23:0/1711017256 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5560fa80f3a0 con 0x5560fa918c50 2024-03-28T05:26:57.312 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.310+0000 7f9de17fa640 10 client.15753 handle_command_reply: tid=0 2024-03-28T05:26:57.312 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.310+0000 7f9e027c2740 1 client.15753 shutdown 2024-03-28T05:26:57.312 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.310+0000 7f9e027c2740 1 -- 172.21.15.23:0/1711017256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8a4030 msgr2=0x5560fa8b0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:57.312 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.310+0000 7f9e027c2740 1 --2- 172.21.15.23:0/1711017256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8a4030 0x5560fa8b0a40 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f9de8000c00 tx=0x7f9de8005b50 comp rx=0 tx=0).stop 2024-03-28T05:26:57.312 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.311+0000 7f9e027c2740 1 -- 172.21.15.23:0/1711017256 shutdown_connections 2024-03-28T05:26:57.312 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.311+0000 7f9e027c2740 1 --2- 172.21.15.23:0/1711017256 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5560fa8ace00 0x5560fa917250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:57.312 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.311+0000 7f9e027c2740 1 --2- 172.21.15.23:0/1711017256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5560fa8a4030 0x5560fa8b0a40 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:57.312 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.311+0000 7f9e027c2740 1 --2- 172.21.15.23:0/1711017256 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5560fa918c50 0x5560fa91b040 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:57.312 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.311+0000 7f9e027c2740 1 --2- 172.21.15.23:0/1711017256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5560fa8a3710 0x5560fa8b0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.311+0000 7f9e027c2740 1 -- 172.21.15.23:0/1711017256 >> 172.21.15.23:0/1711017256 conn(0x5560fa89e030 msgr2=0x5560fa89c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.311+0000 7f9de17fa640 0 client.15753 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:26:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.311+0000 7f9e027c2740 1 -- 172.21.15.23:0/1711017256 shutdown_connections 2024-03-28T05:26:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.311+0000 7f9e027c2740 1 -- 172.21.15.23:0/1711017256 wait complete. 2024-03-28T05:26:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.311+0000 7f9e027c2740 20 client.15753 trim_cache size 0 max 16384 2024-03-28T05:26:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:26:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:26:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:26:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:26:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.311+0000 7f9dffbd4640 1 -- 172.21.15.23:0/37276450 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f9de40a2a10 msgr2=0x7f9de40a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.311+0000 7f9dffbd4640 1 --2- 172.21.15.23:0/37276450 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f9de40a2a10 0x7f9de40a4ed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9df0005f40 tx=0x7f9df0004a40 comp rx=0 tx=0).stop 2024-03-28T05:26:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.311+0000 7f9dffbd4640 1 -- 172.21.15.23:0/37276450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9df81011b0 msgr2=0x7f9df8198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:26:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.311+0000 7f9dffbd4640 1 --2- 172.21.15.23:0/37276450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9df81011b0 0x7f9df8198540 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9de8005940 tx=0x7f9de80043b0 comp rx=0 tx=0).stop 2024-03-28T05:26:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.311+0000 7f9dffbd4640 1 -- 172.21.15.23:0/37276450 shutdown_connections 2024-03-28T05:26:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.311+0000 7f9dffbd4640 1 --2- 172.21.15.23:0/37276450 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f9de40a2a10 0x7f9de40a4ed0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.312+0000 7f9dffbd4640 1 --2- 172.21.15.23:0/37276450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9df8105540 0x7f9df819c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.312+0000 7f9dffbd4640 1 --2- 172.21.15.23:0/37276450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9df81011b0 0x7f9df8198540 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.312+0000 7f9dffbd4640 1 --2- 172.21.15.23:0/37276450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9df81007f0 0x7f9df8198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:26:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.312+0000 7f9dffbd4640 1 -- 172.21.15.23:0/37276450 >> 172.21.15.23:0/37276450 conn(0x7f9df80773f0 msgr2=0x7f9df8068fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:26:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.312+0000 7f9dffbd4640 1 -- 172.21.15.23:0/37276450 shutdown_connections 2024-03-28T05:26:57.316 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:26:57.312+0000 7f9dffbd4640 1 -- 172.21.15.23:0/37276450 wait complete. 2024-03-28T05:26:58.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:57 smithi113 ceph-mon[37164]: pgmap v104: 129 pgs: 129 active+clean; 100 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 2.8 MiB/s wr, 576 op/s 2024-03-28T05:26:58.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:57 smithi023 ceph-mon[31954]: pgmap v104: 129 pgs: 129 active+clean; 100 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 2.8 MiB/s wr, 576 op/s 2024-03-28T05:26:58.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:57 smithi098 ceph-mon[36087]: pgmap v104: 129 pgs: 129 active+clean; 100 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 2.8 MiB/s wr, 576 op/s 2024-03-28T05:27:00.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:26:59 smithi113 ceph-mon[37164]: pgmap v105: 129 pgs: 129 active+clean; 100 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 1.5 MiB/s wr, 166 op/s 2024-03-28T05:27:00.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:26:59 smithi023 ceph-mon[31954]: pgmap v105: 129 pgs: 129 active+clean; 100 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 1.5 MiB/s wr, 166 op/s 2024-03-28T05:27:00.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:26:59 smithi098 ceph-mon[36087]: pgmap v105: 129 pgs: 129 active+clean; 100 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 1.5 MiB/s wr, 166 op/s 2024-03-28T05:27:02.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:01 smithi113 ceph-mon[37164]: pgmap v106: 129 pgs: 129 active+clean; 100 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 1.6 MiB/s wr, 237 op/s 2024-03-28T05:27:02.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:01 smithi023 ceph-mon[31954]: pgmap v106: 129 pgs: 129 active+clean; 100 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 1.6 MiB/s wr, 237 op/s 2024-03-28T05:27:02.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:01 smithi098 ceph-mon[36087]: pgmap v106: 129 pgs: 129 active+clean; 100 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 1.6 MiB/s wr, 237 op/s 2024-03-28T05:27:04.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:03 smithi113 ceph-mon[37164]: pgmap v107: 129 pgs: 129 active+clean; 90 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 1.7 MiB/s wr, 435 op/s 2024-03-28T05:27:04.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:03 smithi023 ceph-mon[31954]: pgmap v107: 129 pgs: 129 active+clean; 90 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 1.7 MiB/s wr, 435 op/s 2024-03-28T05:27:04.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:03 smithi098 ceph-mon[36087]: pgmap v107: 129 pgs: 129 active+clean; 90 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 1.7 MiB/s wr, 435 op/s 2024-03-28T05:27:06.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:05 smithi113 ceph-mon[37164]: pgmap v108: 129 pgs: 129 active+clean; 90 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 584 KiB/s wr, 337 op/s 2024-03-28T05:27:06.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:05 smithi023 ceph-mon[31954]: pgmap v108: 129 pgs: 129 active+clean; 90 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 584 KiB/s wr, 337 op/s 2024-03-28T05:27:06.365 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:05 smithi098 ceph-mon[36087]: pgmap v108: 129 pgs: 129 active+clean; 90 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 584 KiB/s wr, 337 op/s 2024-03-28T05:27:06.399 INFO:tasks.workunit.client.0.smithi023.stdout: 31 77311 46 50918 48 32487 83 2024-03-28T05:27:08.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:07 smithi113 ceph-mon[37164]: pgmap v109: 129 pgs: 129 active+clean; 90 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 620 KiB/s wr, 353 op/s 2024-03-28T05:27:08.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:07 smithi023 ceph-mon[31954]: pgmap v109: 129 pgs: 129 active+clean; 90 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 620 KiB/s wr, 353 op/s 2024-03-28T05:27:08.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:07 smithi098 ceph-mon[36087]: pgmap v109: 129 pgs: 129 active+clean; 90 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 620 KiB/s wr, 353 op/s 2024-03-28T05:27:10.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:09 smithi113 ceph-mon[37164]: pgmap v110: 129 pgs: 129 active+clean; 90 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 317 KiB/s wr, 302 op/s 2024-03-28T05:27:10.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:09 smithi023 ceph-mon[31954]: pgmap v110: 129 pgs: 129 active+clean; 90 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 317 KiB/s wr, 302 op/s 2024-03-28T05:27:10.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:09 smithi098 ceph-mon[36087]: pgmap v110: 129 pgs: 129 active+clean; 90 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 317 KiB/s wr, 302 op/s 2024-03-28T05:27:10.403 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.403+0000 7f91f0916640 1 -- 172.21.15.23:0/2736449506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91ec105540 msgr2=0x7f91ec105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:10.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.403+0000 7f91f0916640 1 --2- 172.21.15.23:0/2736449506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91ec105540 0x7f91ec105920 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f91dc009900 tx=0x7f91dc02bc30 comp rx=0 tx=0).stop 2024-03-28T05:27:10.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.403+0000 7f91f0916640 1 -- 172.21.15.23:0/2736449506 shutdown_connections 2024-03-28T05:27:10.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.403+0000 7f91f0916640 1 --2- 172.21.15.23:0/2736449506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91ec1011b0 0x7f91ec10d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.403+0000 7f91f0916640 1 --2- 172.21.15.23:0/2736449506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91ec1007f0 0x7f91ec100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.403+0000 7f91f0916640 1 --2- 172.21.15.23:0/2736449506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91ec105540 0x7f91ec105920 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.403+0000 7f91f0916640 1 -- 172.21.15.23:0/2736449506 >> 172.21.15.23:0/2736449506 conn(0x7f91ec0773f0 msgr2=0x7f91ec077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:10.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.403+0000 7f91f0916640 1 -- 172.21.15.23:0/2736449506 shutdown_connections 2024-03-28T05:27:10.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.403+0000 7f91f0916640 1 -- 172.21.15.23:0/2736449506 wait complete. 2024-03-28T05:27:10.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.404+0000 7f91f0916640 1 Processor -- start 2024-03-28T05:27:10.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.404+0000 7f91f0916640 1 -- start start 2024-03-28T05:27:10.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.404+0000 7f91f0916640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91ec1007f0 0x7f91ec102210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:10.406 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.404+0000 7f91f0916640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91ec1011b0 0x7f91ec102750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:10.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.404+0000 7f91f0916640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91ec105540 0x7f91ec102c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:10.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.404+0000 7f91f0916640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec110a60 con 0x7f91ec105540 2024-03-28T05:27:10.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.404+0000 7f91f0916640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec1108e0 con 0x7f91ec1011b0 2024-03-28T05:27:10.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.404+0000 7f91f0916640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec110be0 con 0x7f91ec1007f0 2024-03-28T05:27:10.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.404+0000 7f91ead9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91ec105540 0x7f91ec102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:10.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.404+0000 7f91ea59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91ec1007f0 0x7f91ec102210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:10.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.404+0000 7f91ea59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91ec1007f0 0x7f91ec102210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:39638/0 (socket says 172.21.15.23:39638) 2024-03-28T05:27:10.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.404+0000 7f91ea59c640 1 -- 172.21.15.23:0/2214514069 learned_addr learned my addr 172.21.15.23:0/2214514069 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:27:10.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.405+0000 7f91e9d9b640 1 --2- 172.21.15.23:0/2214514069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91ec1011b0 0x7f91ec102750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:10.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.405+0000 7f91ea59c640 1 -- 172.21.15.23:0/2214514069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91ec1011b0 msgr2=0x7f91ec102750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:10.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.405+0000 7f91ea59c640 1 --2- 172.21.15.23:0/2214514069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91ec1011b0 0x7f91ec102750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.405+0000 7f91ea59c640 1 -- 172.21.15.23:0/2214514069 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91ec105540 msgr2=0x7f91ec102c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:10.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.405+0000 7f91ea59c640 1 --2- 172.21.15.23:0/2214514069 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91ec105540 0x7f91ec102c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.407 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.405+0000 7f91ea59c640 1 -- 172.21.15.23:0/2214514069 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ec1a63e0 con 0x7f91ec1007f0 2024-03-28T05:27:10.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.405+0000 7f91e9d9b640 1 --2- 172.21.15.23:0/2214514069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91ec1011b0 0x7f91ec102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:27:10.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.405+0000 7f91ead9d640 1 --2- 172.21.15.23:0/2214514069 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91ec105540 0x7f91ec102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:27:10.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.405+0000 7f91ea59c640 1 --2- 172.21.15.23:0/2214514069 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91ec1007f0 0x7f91ec102210 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f91dc005940 tx=0x7f91dc0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:10.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.405+0000 7f91db7fe640 1 -- 172.21.15.23:0/2214514069 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc041070 con 0x7f91ec1007f0 2024-03-28T05:27:10.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.405+0000 7f91db7fe640 1 -- 172.21.15.23:0/2214514069 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f91dc02ccc0 con 0x7f91ec1007f0 2024-03-28T05:27:10.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.405+0000 7f91f0916640 1 -- 172.21.15.23:0/2214514069 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91ec1a6670 con 0x7f91ec1007f0 2024-03-28T05:27:10.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.405+0000 7f91db7fe640 1 -- 172.21.15.23:0/2214514069 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc003d80 con 0x7f91ec1007f0 2024-03-28T05:27:10.408 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.405+0000 7f91f0916640 1 -- 172.21.15.23:0/2214514069 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91ec1a6b60 con 0x7f91ec1007f0 2024-03-28T05:27:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.408+0000 7f91db7fe640 1 -- 172.21.15.23:0/2214514069 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f91dc044070 con 0x7f91ec1007f0 2024-03-28T05:27:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.408+0000 7f91f35d6740 1 Processor -- start 2024-03-28T05:27:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.408+0000 7f91f35d6740 1 -- start start 2024-03-28T05:27:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.408+0000 7f91f35d6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd963710 0x5594fd963af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.408+0000 7f91f35d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594fd964030 0x5594fd96c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.408+0000 7f91f35d6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd96ce00 0x5594fd96f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.409+0000 7f91f35d6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594fd8c7b10 con 0x5594fd96ce00 2024-03-28T05:27:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.409+0000 7f91f35d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594fd870900 con 0x5594fd964030 2024-03-28T05:27:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.409+0000 7f91f35d6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594fd8c3bb0 con 0x5594fd963710 2024-03-28T05:27:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.409+0000 7f91db7fe640 1 --2- 172.21.15.23:0/2214514069 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f91d40a2a10 0x7f91d40a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.409+0000 7f91ead9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd963710 0x5594fd963af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.409+0000 7f91ead9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd963710 0x5594fd963af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:39646/0 (socket says 172.21.15.23:39646) 2024-03-28T05:27:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.409+0000 7f91ead9d640 1 -- 172.21.15.23:0/1464143506 learned_addr learned my addr 172.21.15.23:0/1464143506 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:27:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.409+0000 7f91e9d9b640 1 --2- 172.21.15.23:0/1464143506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd96ce00 0x5594fd96f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.409+0000 7f91ead9d640 1 --2- 172.21.15.23:0/2214514069 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f91d40a2a10 0x7f91d40a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.409+0000 7f91db7fe640 1 -- 172.21.15.23:0/2214514069 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f91dc0f0c70 con 0x7f91ec1007f0 2024-03-28T05:27:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.409+0000 7f91ea59c640 1 --2- 172.21.15.23:0/1464143506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594fd964030 0x5594fd96c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.409+0000 7f91ead9d640 1 -- 172.21.15.23:0/1464143506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594fd964030 msgr2=0x5594fd96c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.409+0000 7f91ead9d640 1 --2- 172.21.15.23:0/1464143506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594fd964030 0x5594fd96c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.409+0000 7f91ead9d640 1 -- 172.21.15.23:0/1464143506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd96ce00 msgr2=0x5594fd96f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.409+0000 7f91ead9d640 1 --2- 172.21.15.23:0/1464143506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd96ce00 0x5594fd96f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.409+0000 7f91ead9d640 1 -- 172.21.15.23:0/1464143506 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594fd5dc210 con 0x5594fd963710 2024-03-28T05:27:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.409+0000 7f91ead9d640 1 --2- 172.21.15.23:0/1464143506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd963710 0x5594fd963af0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f91e4013d80 tx=0x7f91e40117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f5295baf07eace5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.409+0000 7f91ead9d640 1 --2- 172.21.15.23:0/2214514069 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f91d40a2a10 0x7f91d40a4ed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f91e400a000 tx=0x7f91e401d000 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.409+0000 7f91d8ff9640 1 -- 172.21.15.23:0/1464143506 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e401caf0 con 0x5594fd963710 2024-03-28T05:27:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.409+0000 7f91d8ff9640 1 -- 172.21.15.23:0/1464143506 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f91e401cc90 con 0x5594fd963710 2024-03-28T05:27:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.410+0000 7f91d8ff9640 1 -- 172.21.15.23:0/1464143506 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e400e3b0 con 0x5594fd963710 2024-03-28T05:27:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.410+0000 7f91f35d6740 1 -- 172.21.15.23:0/1464143506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd963710 msgr2=0x5594fd963af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.410+0000 7f91f35d6740 1 --2- 172.21.15.23:0/1464143506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd963710 0x5594fd963af0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f91e4013d80 tx=0x7f91e40117b0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.410+0000 7f91f35d6740 1 -- 172.21.15.23:0/1464143506 shutdown_connections 2024-03-28T05:27:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.410+0000 7f91f35d6740 1 --2- 172.21.15.23:0/1464143506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd96ce00 0x5594fd96f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.410+0000 7f91f35d6740 1 --2- 172.21.15.23:0/1464143506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594fd964030 0x5594fd96c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.410+0000 7f91f35d6740 1 --2- 172.21.15.23:0/1464143506 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd963710 0x5594fd963af0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.410+0000 7f91f35d6740 1 -- 172.21.15.23:0/1464143506 >> 172.21.15.23:0/1464143506 conn(0x5594fd95e030 msgr2=0x5594fd95e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.410+0000 7f91f35d6740 1 -- 172.21.15.23:0/1464143506 shutdown_connections 2024-03-28T05:27:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.410+0000 7f91f35d6740 1 -- 172.21.15.23:0/1464143506 wait complete. 2024-03-28T05:27:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.411+0000 7f91f35d6740 1 -- start start 2024-03-28T05:27:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.411+0000 7f91f35d6740 1 Processor -- start 2024-03-28T05:27:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.411+0000 7f91f35d6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd963710 0x5594fd969ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.411+0000 7f91f35d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594fd964030 0x5594fd96a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.411+0000 7f91f35d6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd96ce00 0x5594fd96a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.411+0000 7f91f35d6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594fd96f9d0 con 0x5594fd96ce00 2024-03-28T05:27:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.411+0000 7f91f35d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594fd883d50 con 0x5594fd964030 2024-03-28T05:27:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.411+0000 7f91f35d6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594fd96fb50 con 0x5594fd963710 2024-03-28T05:27:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.411+0000 7f91f35d6740 10 client.0 _ll_register_callbacks cb 0x5594fd883720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:27:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.411+0000 7f91ead9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd96ce00 0x5594fd96a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.411+0000 7f91ea59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd963710 0x5594fd969ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.411+0000 7f91e9d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594fd964030 0x5594fd96a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.411+0000 7f91ead9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd96ce00 0x5594fd96a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40524/0 (socket says 172.21.15.23:40524) 2024-03-28T05:27:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.411+0000 7f91ea59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd963710 0x5594fd969ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:39654/0 (socket says 172.21.15.23:39654) 2024-03-28T05:27:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.411+0000 7f91ead9d640 1 -- 172.21.15.23:0/599656946 learned_addr learned my addr 172.21.15.23:0/599656946 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:27:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.412+0000 7f91ea59c640 1 -- 172.21.15.23:0/599656946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594fd964030 msgr2=0x5594fd96a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.412+0000 7f91ea59c640 1 --2- 172.21.15.23:0/599656946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594fd964030 0x5594fd96a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.412+0000 7f91ea59c640 1 -- 172.21.15.23:0/599656946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd96ce00 msgr2=0x5594fd96a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.412+0000 7f91ea59c640 1 --2- 172.21.15.23:0/599656946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd96ce00 0x5594fd96a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.412+0000 7f91ea59c640 1 -- 172.21.15.23:0/599656946 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5594fd6a8620 con 0x5594fd963710 2024-03-28T05:27:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.412+0000 7f91ead9d640 1 --2- 172.21.15.23:0/599656946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd96ce00 0x5594fd96a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:27:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.412+0000 7f91e9d9b640 1 --2- 172.21.15.23:0/599656946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594fd964030 0x5594fd96a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:27:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.412+0000 7f91ea59c640 1 --2- 172.21.15.23:0/599656946 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd963710 0x5594fd969ea0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0ee140 tx=0x7f91dc0b9780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88b8a77283125662 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.412+0000 7f91d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-28T05:27:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.412+0000 7f91d1ffb640 1 -- 172.21.15.23:0/599656946 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc04b040 con 0x5594fd963710 2024-03-28T05:27:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.412+0000 7f91f35d6740 1 -- 172.21.15.23:0/599656946 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f91e401cc90 con 0x5594fd963710 2024-03-28T05:27:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.412+0000 7f91d1ffb640 1 -- 172.21.15.23:0/599656946 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f91dc0c5360 con 0x5594fd963710 2024-03-28T05:27:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.412+0000 7f91d1ffb640 1 -- 172.21.15.23:0/599656946 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f91dc0bc9b0 con 0x5594fd963710 2024-03-28T05:27:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.412+0000 7f91f35d6740 10 client.25597 fetch_fsmap learned FSMap version 20 2024-03-28T05:27:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.412+0000 7f91f35d6740 1 -- 172.21.15.23:0/599656946 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5594fd96f9d0 con 0x5594fd963710 2024-03-28T05:27:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.412+0000 7f91d1ffb640 1 -- 172.21.15.23:0/599656946 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f91e401cc90 con 0x5594fd963710 2024-03-28T05:27:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.412+0000 7f91d1ffb640 1 -- 172.21.15.23:0/599656946 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f91dc0b7bc0 con 0x5594fd963710 2024-03-28T05:27:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.413+0000 7f91f35d6740 10 client.25597 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:27:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.413+0000 7f91f35d6740 10 client.25597 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:27:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.413+0000 7f91f35d6740 20 client.25597 populate_metadata read hostname 'smithi023' 2024-03-28T05:27:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.413+0000 7f91f35d6740 1 --2- 172.21.15.23:0/599656946 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5594fd985830 0x5594fd987c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.413+0000 7f91f35d6740 4 client.25597 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:27:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.413+0000 7f91f35d6740 1 -- 172.21.15.23:0/599656946 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5594fd8d5050 con 0x5594fd985830 2024-03-28T05:27:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.413+0000 7f91e9d9b640 1 --2- 172.21.15.23:0/599656946 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5594fd985830 0x5594fd987c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.413+0000 7f91d1ffb640 10 client.25597 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:27:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.413+0000 7f91e9d9b640 1 --2- 172.21.15.23:0/599656946 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5594fd985830 0x5594fd987c70 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd8e068b186a689 server_cookie=b726325cfcd2eef0 in_seq=0 out_seq=0 2024-03-28T05:27:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.414+0000 7f91d1ffb640 1 -- 172.21.15.23:0/599656946 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5594fd8d5050 con 0x5594fd985830 2024-03-28T05:27:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.414+0000 7f91d1ffb640 10 client.25597 handle_command_reply: tid=0 2024-03-28T05:27:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.414+0000 7f91f35d6740 1 client.25597 shutdown 2024-03-28T05:27:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.414+0000 7f91f35d6740 1 -- 172.21.15.23:0/599656946 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd963710 msgr2=0x5594fd969ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.414+0000 7f91f35d6740 1 --2- 172.21.15.23:0/599656946 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd963710 0x5594fd969ea0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0ee140 tx=0x7f91dc0b9780 comp rx=0 tx=0).stop 2024-03-28T05:27:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.414+0000 7f91f35d6740 1 -- 172.21.15.23:0/599656946 shutdown_connections 2024-03-28T05:27:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.414+0000 7f91f35d6740 1 --2- 172.21.15.23:0/599656946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd96ce00 0x5594fd96a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.414+0000 7f91f35d6740 1 --2- 172.21.15.23:0/599656946 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5594fd985830 0x5594fd987c70 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.414+0000 7f91f35d6740 1 --2- 172.21.15.23:0/599656946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594fd964030 0x5594fd96a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.414+0000 7f91f35d6740 1 --2- 172.21.15.23:0/599656946 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd963710 0x5594fd969ea0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.414+0000 7f91d1ffb640 0 client.25597 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:27:10.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.414+0000 7f91f35d6740 1 -- 172.21.15.23:0/599656946 >> 172.21.15.23:0/599656946 conn(0x5594fd95e030 msgr2=0x5594fd95fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:10.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.414+0000 7f91f35d6740 1 -- 172.21.15.23:0/599656946 shutdown_connections 2024-03-28T05:27:10.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.414+0000 7f91f35d6740 1 -- 172.21.15.23:0/599656946 wait complete. 2024-03-28T05:27:10.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.415+0000 7f91f35d6740 20 client.25597 trim_cache size 0 max 16384 2024-03-28T05:27:10.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.431+0000 7f91f35d6740 1 Processor -- start 2024-03-28T05:27:10.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.431+0000 7f91f35d6740 1 -- start start 2024-03-28T05:27:10.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.432+0000 7f91f35d6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd963710 0x5594fd96c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.432+0000 7f91f35d6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd964030 0x5594fd980ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.432+0000 7f91f35d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594fd96ce00 0x5594fd980ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.432+0000 7f91f35d6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec1aa7b0 con 0x5594fd963710 2024-03-28T05:27:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.432+0000 7f91f35d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec1acd50 con 0x5594fd96ce00 2024-03-28T05:27:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.432+0000 7f91f35d6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec1aadb0 con 0x5594fd964030 2024-03-28T05:27:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.432+0000 7f91ea59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd963710 0x5594fd96c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.432+0000 7f91ea59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd963710 0x5594fd96c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40530/0 (socket says 172.21.15.23:40530) 2024-03-28T05:27:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.432+0000 7f91ea59c640 1 -- 172.21.15.23:0/2862094105 learned_addr learned my addr 172.21.15.23:0/2862094105 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:27:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.432+0000 7f91e9d9b640 1 --2- 172.21.15.23:0/2862094105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd964030 0x5594fd980ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.432+0000 7f91ead9d640 1 --2- 172.21.15.23:0/2862094105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594fd96ce00 0x5594fd980ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.432+0000 7f91ea59c640 1 -- 172.21.15.23:0/2862094105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd964030 msgr2=0x5594fd980ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.432+0000 7f91ea59c640 1 --2- 172.21.15.23:0/2862094105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd964030 0x5594fd980ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.432+0000 7f91ea59c640 1 -- 172.21.15.23:0/2862094105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594fd96ce00 msgr2=0x5594fd980ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.432+0000 7f91ea59c640 1 --2- 172.21.15.23:0/2862094105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594fd96ce00 0x5594fd980ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.432+0000 7f91ea59c640 1 -- 172.21.15.23:0/2862094105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594fd9882f0 con 0x5594fd963710 2024-03-28T05:27:10.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.432+0000 7f91ea59c640 1 --2- 172.21.15.23:0/2862094105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd963710 0x5594fd96c5d0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0b94b0 tx=0x7f91dc02c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a38b1358711c5461 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:10.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.432+0000 7f91d3fff640 1 -- 172.21.15.23:0/2862094105 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc04b070 con 0x5594fd963710 2024-03-28T05:27:10.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.432+0000 7f91d3fff640 1 -- 172.21.15.23:0/2862094105 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f91dc004c80 con 0x5594fd963710 2024-03-28T05:27:10.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.432+0000 7f91d3fff640 1 -- 172.21.15.23:0/2862094105 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc0b7bc0 con 0x5594fd963710 2024-03-28T05:27:10.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.433+0000 7f91f35d6740 1 -- 172.21.15.23:0/2862094105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd963710 msgr2=0x5594fd96c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:10.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.433+0000 7f91f35d6740 1 --2- 172.21.15.23:0/2862094105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd963710 0x5594fd96c5d0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0b94b0 tx=0x7f91dc02c450 comp rx=0 tx=0).stop 2024-03-28T05:27:10.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.433+0000 7f91f35d6740 1 -- 172.21.15.23:0/2862094105 shutdown_connections 2024-03-28T05:27:10.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.433+0000 7f91f35d6740 1 --2- 172.21.15.23:0/2862094105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594fd96ce00 0x5594fd980ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.433+0000 7f91f35d6740 1 --2- 172.21.15.23:0/2862094105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd964030 0x5594fd980ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.433+0000 7f91f35d6740 1 --2- 172.21.15.23:0/2862094105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd963710 0x5594fd96c5d0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.433+0000 7f91f35d6740 1 -- 172.21.15.23:0/2862094105 >> 172.21.15.23:0/2862094105 conn(0x5594fd95e030 msgr2=0x5594fd95e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:10.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.433+0000 7f91f35d6740 1 -- 172.21.15.23:0/2862094105 shutdown_connections 2024-03-28T05:27:10.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.433+0000 7f91f35d6740 1 -- 172.21.15.23:0/2862094105 wait complete. 2024-03-28T05:27:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.433+0000 7f91f35d6740 1 -- start start 2024-03-28T05:27:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.434+0000 7f91f35d6740 1 Processor -- start 2024-03-28T05:27:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.434+0000 7f91f35d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594fd963710 0x5594fd978bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.434+0000 7f91f35d6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd964030 0x5594fd9790f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.434+0000 7f91f35d6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd96ce00 0x5594fd979630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.434+0000 7f91f35d6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec1ab3d0 con 0x5594fd964030 2024-03-28T05:27:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.434+0000 7f91f35d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec1ac6f0 con 0x5594fd963710 2024-03-28T05:27:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.434+0000 7f91f35d6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec1aba30 con 0x5594fd96ce00 2024-03-28T05:27:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.434+0000 7f91f35d6740 10 client.0 _ll_register_callbacks cb 0x7f91dc0dd090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:27:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.434+0000 7f91e9d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd964030 0x5594fd9790f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.434+0000 7f91ead9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd96ce00 0x5594fd979630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.434+0000 7f91e9d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd964030 0x5594fd9790f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40540/0 (socket says 172.21.15.23:40540) 2024-03-28T05:27:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.434+0000 7f91e9d9b640 1 -- 172.21.15.23:0/1680525684 learned_addr learned my addr 172.21.15.23:0/1680525684 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:27:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.434+0000 7f91ea59c640 1 --2- 172.21.15.23:0/1680525684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594fd963710 0x5594fd978bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.434+0000 7f91e9d9b640 1 -- 172.21.15.23:0/1680525684 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd96ce00 msgr2=0x5594fd979630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.434+0000 7f91e9d9b640 1 --2- 172.21.15.23:0/1680525684 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd96ce00 0x5594fd979630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.434+0000 7f91e9d9b640 1 -- 172.21.15.23:0/1680525684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594fd963710 msgr2=0x5594fd978bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.434+0000 7f91e9d9b640 1 --2- 172.21.15.23:0/1680525684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594fd963710 0x5594fd978bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.434+0000 7f91e9d9b640 1 -- 172.21.15.23:0/1680525684 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5594fd9e0340 con 0x5594fd964030 2024-03-28T05:27:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.434+0000 7f91ea59c640 1 --2- 172.21.15.23:0/1680525684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594fd963710 0x5594fd978bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:27:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.434+0000 7f91ead9d640 1 --2- 172.21.15.23:0/1680525684 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd96ce00 0x5594fd979630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:27:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.435+0000 7f91e9d9b640 1 --2- 172.21.15.23:0/1680525684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd964030 0x5594fd9790f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x5594fd95f1c0 tx=0x7f91e00056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c05b21b3bc99ee2f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.435+0000 7f91d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:27:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.435+0000 7f91d1ffb640 1 -- 172.21.15.23:0/1680525684 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e0020690 con 0x5594fd964030 2024-03-28T05:27:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.435+0000 7f91d1ffb640 1 -- 172.21.15.23:0/1680525684 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f91e0020830 con 0x5594fd964030 2024-03-28T05:27:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.435+0000 7f91f35d6740 1 -- 172.21.15.23:0/1680525684 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f91dc004c80 con 0x5594fd964030 2024-03-28T05:27:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.435+0000 7f91d1ffb640 1 -- 172.21.15.23:0/1680525684 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f91e0020a30 con 0x5594fd964030 2024-03-28T05:27:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.435+0000 7f91f35d6740 10 client.15771 fetch_fsmap learned FSMap version 20 2024-03-28T05:27:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.435+0000 7f91f35d6740 1 -- 172.21.15.23:0/1680525684 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5594fd9e16a0 con 0x5594fd964030 2024-03-28T05:27:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.435+0000 7f91d1ffb640 1 -- 172.21.15.23:0/1680525684 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f91dc004c80 con 0x5594fd964030 2024-03-28T05:27:10.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.435+0000 7f91d1ffb640 1 -- 172.21.15.23:0/1680525684 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f91e0029480 con 0x5594fd964030 2024-03-28T05:27:10.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.435+0000 7f91f35d6740 10 client.15771 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:27:10.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.435+0000 7f91f35d6740 10 client.15771 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:27:10.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.435+0000 7f91f35d6740 20 client.15771 populate_metadata read hostname 'smithi023' 2024-03-28T05:27:10.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.435+0000 7f91f35d6740 1 --2- 172.21.15.23:0/1680525684 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5594fd9e1dd0 0x5594fd9e41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:10.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.435+0000 7f91f35d6740 4 client.15771 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:27:10.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.435+0000 7f91f35d6740 1 -- 172.21.15.23:0/1680525684 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5594fd8cf3a0 con 0x5594fd9e1dd0 2024-03-28T05:27:10.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.435+0000 7f91ea59c640 1 --2- 172.21.15.23:0/1680525684 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5594fd9e1dd0 0x5594fd9e41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:10.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.436+0000 7f91ea59c640 1 --2- 172.21.15.23:0/1680525684 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5594fd9e1dd0 0x5594fd9e41f0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f33400ae6965498b server_cookie=56f455edaa9b0097 in_seq=0 out_seq=0 2024-03-28T05:27:10.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.436+0000 7f91d1ffb640 10 client.15771 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:27:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.743+0000 7f91d1ffb640 1 -- 172.21.15.23:0/1680525684 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5594fd8cf3a0 con 0x5594fd9e1dd0 2024-03-28T05:27:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.743+0000 7f91d1ffb640 10 client.15771 handle_command_reply: tid=0 2024-03-28T05:27:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.743+0000 7f91f35d6740 1 client.15771 shutdown 2024-03-28T05:27:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.743+0000 7f91f35d6740 1 -- 172.21.15.23:0/1680525684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd964030 msgr2=0x5594fd9790f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.743+0000 7f91f35d6740 1 --2- 172.21.15.23:0/1680525684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd964030 0x5594fd9790f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x5594fd95f1c0 tx=0x7f91e00056e0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.744 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.743+0000 7f91f35d6740 1 -- 172.21.15.23:0/1680525684 shutdown_connections 2024-03-28T05:27:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.743+0000 7f91f35d6740 1 --2- 172.21.15.23:0/1680525684 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5594fd96ce00 0x5594fd979630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.743+0000 7f91f35d6740 1 --2- 172.21.15.23:0/1680525684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd964030 0x5594fd9790f0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.743+0000 7f91f35d6740 1 --2- 172.21.15.23:0/1680525684 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5594fd9e1dd0 0x5594fd9e41f0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.743+0000 7f91f35d6740 1 --2- 172.21.15.23:0/1680525684 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594fd963710 0x5594fd978bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.743+0000 7f91d1ffb640 0 client.15771 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:27:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.743+0000 7f91f35d6740 1 -- 172.21.15.23:0/1680525684 >> 172.21.15.23:0/1680525684 conn(0x5594fd95e030 msgr2=0x5594fd95f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.744+0000 7f91f35d6740 1 -- 172.21.15.23:0/1680525684 shutdown_connections 2024-03-28T05:27:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.744+0000 7f91f35d6740 1 -- 172.21.15.23:0/1680525684 wait complete. 2024-03-28T05:27:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.744+0000 7f91f35d6740 20 client.15771 trim_cache size 0 max 16384 2024-03-28T05:27:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:27:10.745 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:27:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:27:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:27:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.744+0000 7f91f0916640 1 -- 172.21.15.23:0/2214514069 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f91d40a2a10 msgr2=0x7f91d40a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.744+0000 7f91f0916640 1 --2- 172.21.15.23:0/2214514069 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f91d40a2a10 0x7f91d40a4ed0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f91e400a000 tx=0x7f91e401d000 comp rx=0 tx=0).stop 2024-03-28T05:27:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.744+0000 7f91f0916640 1 -- 172.21.15.23:0/2214514069 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91ec1007f0 msgr2=0x7f91ec102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.744+0000 7f91f0916640 1 --2- 172.21.15.23:0/2214514069 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91ec1007f0 0x7f91ec102210 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f91dc005940 tx=0x7f91dc0042f0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.744+0000 7f91f0916640 1 -- 172.21.15.23:0/2214514069 shutdown_connections 2024-03-28T05:27:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.744+0000 7f91f0916640 1 --2- 172.21.15.23:0/2214514069 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91ec105540 0x7f91ec102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.744+0000 7f91f0916640 1 --2- 172.21.15.23:0/2214514069 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f91d40a2a10 0x7f91d40a4ed0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.744+0000 7f91f0916640 1 --2- 172.21.15.23:0/2214514069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91ec1011b0 0x7f91ec102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.744+0000 7f91f0916640 1 --2- 172.21.15.23:0/2214514069 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f91ec1007f0 0x7f91ec102210 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.744+0000 7f91f0916640 1 -- 172.21.15.23:0/2214514069 >> 172.21.15.23:0/2214514069 conn(0x7f91ec0773f0 msgr2=0x7f91ec068f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.744+0000 7f91f0916640 1 -- 172.21.15.23:0/2214514069 shutdown_connections 2024-03-28T05:27:10.747 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:10.745+0000 7f91f0916640 1 -- 172.21.15.23:0/2214514069 wait complete. 2024-03-28T05:27:12.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:11 smithi113 ceph-mon[37164]: pgmap v111: 129 pgs: 129 active+clean; 95 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 765 KiB/s wr, 330 op/s 2024-03-28T05:27:12.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:11 smithi023 ceph-mon[31954]: pgmap v111: 129 pgs: 129 active+clean; 95 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 765 KiB/s wr, 330 op/s 2024-03-28T05:27:12.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:11 smithi098 ceph-mon[36087]: pgmap v111: 129 pgs: 129 active+clean; 95 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 765 KiB/s wr, 330 op/s 2024-03-28T05:27:14.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:13 smithi113 ceph-mon[37164]: pgmap v112: 129 pgs: 129 active+clean; 108 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 1.9 MiB/s wr, 354 op/s 2024-03-28T05:27:14.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:13 smithi023 ceph-mon[31954]: pgmap v112: 129 pgs: 129 active+clean; 108 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 1.9 MiB/s wr, 354 op/s 2024-03-28T05:27:14.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:13 smithi098 ceph-mon[36087]: pgmap v112: 129 pgs: 129 active+clean; 108 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 1.9 MiB/s wr, 354 op/s 2024-03-28T05:27:16.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:15 smithi113 ceph-mon[37164]: pgmap v113: 129 pgs: 129 active+clean; 108 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 1.6 MiB/s wr, 139 op/s 2024-03-28T05:27:16.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:15 smithi023 ceph-mon[31954]: pgmap v113: 129 pgs: 129 active+clean; 108 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 1.6 MiB/s wr, 139 op/s 2024-03-28T05:27:16.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:15 smithi098 ceph-mon[36087]: pgmap v113: 129 pgs: 129 active+clean; 108 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 1.6 MiB/s wr, 139 op/s 2024-03-28T05:27:16.399 INFO:tasks.workunit.client.0.smithi023.stdout: 33 199198 134 139589 86 80385 160 2024-03-28T05:27:18.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:17 smithi113 ceph-mon[37164]: pgmap v114: 129 pgs: 129 active+clean; 105 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 1.8 MiB/s wr, 314 op/s 2024-03-28T05:27:18.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:17 smithi023 ceph-mon[31954]: pgmap v114: 129 pgs: 129 active+clean; 105 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 1.8 MiB/s wr, 314 op/s 2024-03-28T05:27:18.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:17 smithi098 ceph-mon[36087]: pgmap v114: 129 pgs: 129 active+clean; 105 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 1.8 MiB/s wr, 314 op/s 2024-03-28T05:27:20.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:19 smithi113 ceph-mon[37164]: pgmap v115: 129 pgs: 129 active+clean; 105 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 1.8 MiB/s wr, 297 op/s 2024-03-28T05:27:20.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:19 smithi023 ceph-mon[31954]: pgmap v115: 129 pgs: 129 active+clean; 105 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 1.8 MiB/s wr, 297 op/s 2024-03-28T05:27:20.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:19 smithi098 ceph-mon[36087]: pgmap v115: 129 pgs: 129 active+clean; 105 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 1.8 MiB/s wr, 297 op/s 2024-03-28T05:27:22.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:21 smithi113 ceph-mon[37164]: pgmap v116: 129 pgs: 129 active+clean; 108 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 2.0 MiB/s wr, 314 op/s 2024-03-28T05:27:22.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:21 smithi023 ceph-mon[31954]: pgmap v116: 129 pgs: 129 active+clean; 108 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 2.0 MiB/s wr, 314 op/s 2024-03-28T05:27:22.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:21 smithi098 ceph-mon[36087]: pgmap v116: 129 pgs: 129 active+clean; 108 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 2.0 MiB/s wr, 314 op/s 2024-03-28T05:27:23.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:22 smithi113 ceph-mon[37164]: pgmap v117: 129 pgs: 129 active+clean; 108 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 1.6 MiB/s wr, 293 op/s 2024-03-28T05:27:23.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:22 smithi023 ceph-mon[31954]: pgmap v117: 129 pgs: 129 active+clean; 108 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 1.6 MiB/s wr, 293 op/s 2024-03-28T05:27:23.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:22 smithi098 ceph-mon[36087]: pgmap v117: 129 pgs: 129 active+clean; 108 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 1.6 MiB/s wr, 293 op/s 2024-03-28T05:27:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.836+0000 7f530437e640 1 -- 172.21.15.23:0/4019212883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52fc1011b0 msgr2=0x7f52fc10d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.836+0000 7f530437e640 1 --2- 172.21.15.23:0/4019212883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52fc1011b0 0x7f52fc10d3d0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f52f4009ca0 tx=0x7f52f402c0f0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.837+0000 7f530437e640 1 -- 172.21.15.23:0/4019212883 shutdown_connections 2024-03-28T05:27:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.837+0000 7f530437e640 1 --2- 172.21.15.23:0/4019212883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52fc1011b0 0x7f52fc10d3d0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.837+0000 7f530437e640 1 --2- 172.21.15.23:0/4019212883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52fc1007f0 0x7f52fc100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.837+0000 7f530437e640 1 --2- 172.21.15.23:0/4019212883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52fc105540 0x7f52fc105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.837+0000 7f530437e640 1 -- 172.21.15.23:0/4019212883 >> 172.21.15.23:0/4019212883 conn(0x7f52fc0773f0 msgr2=0x7f52fc077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.837+0000 7f530437e640 1 -- 172.21.15.23:0/4019212883 shutdown_connections 2024-03-28T05:27:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.837+0000 7f530437e640 1 -- 172.21.15.23:0/4019212883 wait complete. 2024-03-28T05:27:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.837+0000 7f530437e640 1 Processor -- start 2024-03-28T05:27:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.837+0000 7f530437e640 1 -- start start 2024-03-28T05:27:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.837+0000 7f530437e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52fc1007f0 0x7f52fc197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.837+0000 7f530437e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52fc1011b0 0x7f52fc198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.837+0000 7f530437e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52fc105540 0x7f52fc19c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.837+0000 7f530437e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc110a60 con 0x7f52fc1007f0 2024-03-28T05:27:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.837+0000 7f530437e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc1108e0 con 0x7f52fc105540 2024-03-28T05:27:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.837+0000 7f530437e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc110be0 con 0x7f52fc1011b0 2024-03-28T05:27:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.838+0000 7f530211a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52fc1007f0 0x7f52fc197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.838+0000 7f5301919640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52fc1011b0 0x7f52fc198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.838+0000 7f530211a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52fc1007f0 0x7f52fc197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51324/0 (socket says 172.21.15.23:51324) 2024-03-28T05:27:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.838+0000 7f530211a640 1 -- 172.21.15.23:0/349669228 learned_addr learned my addr 172.21.15.23:0/349669228 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:27:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.838+0000 7f5301919640 1 --2- 172.21.15.23:0/349669228 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52fc1011b0 0x7f52fc198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:49010/0 (socket says 172.21.15.23:49010) 2024-03-28T05:27:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.838+0000 7f530291b640 1 --2- 172.21.15.23:0/349669228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52fc105540 0x7f52fc19c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.838+0000 7f530211a640 1 -- 172.21.15.23:0/349669228 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52fc1011b0 msgr2=0x7f52fc198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.838+0000 7f530211a640 1 --2- 172.21.15.23:0/349669228 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52fc1011b0 0x7f52fc198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.838+0000 7f530211a640 1 -- 172.21.15.23:0/349669228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52fc105540 msgr2=0x7f52fc19c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.838+0000 7f530211a640 1 --2- 172.21.15.23:0/349669228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52fc105540 0x7f52fc19c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.838+0000 7f530211a640 1 -- 172.21.15.23:0/349669228 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52fc19cee0 con 0x7f52fc1007f0 2024-03-28T05:27:23.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.838+0000 7f530211a640 1 --2- 172.21.15.23:0/349669228 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52fc1007f0 0x7f52fc197ef0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f52f800e9f0 tx=0x7f52f800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:23.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.838+0000 7f52f37fe640 1 -- 172.21.15.23:0/349669228 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f800cd80 con 0x7f52fc1007f0 2024-03-28T05:27:23.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.838+0000 7f530437e640 1 -- 172.21.15.23:0/349669228 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52fc19d1d0 con 0x7f52fc1007f0 2024-03-28T05:27:23.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.838+0000 7f52f37fe640 1 -- 172.21.15.23:0/349669228 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52f8005500 con 0x7f52fc1007f0 2024-03-28T05:27:23.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.838+0000 7f52f37fe640 1 -- 172.21.15.23:0/349669228 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f8010690 con 0x7f52fc1007f0 2024-03-28T05:27:23.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.838+0000 7f530437e640 1 -- 172.21.15.23:0/349669228 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52fc1a4a60 con 0x7f52fc1007f0 2024-03-28T05:27:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.841+0000 7f52f37fe640 1 -- 172.21.15.23:0/349669228 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f52f80046c0 con 0x7f52fc1007f0 2024-03-28T05:27:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.841+0000 7f5306f69740 1 Processor -- start 2024-03-28T05:27:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.841+0000 7f5306f69740 1 -- start start 2024-03-28T05:27:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.841+0000 7f52f37fe640 1 --2- 172.21.15.23:0/349669228 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f52e80a2a10 0x7f52e80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f5306f69740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af75710 0x55957af75af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f5306f69740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55957af76030 0x55957af7e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f5306f69740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55957af7ee00 0x55957af811f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f5306f69740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55957aed9b10 con 0x55957af75710 2024-03-28T05:27:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f5306f69740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55957ae82900 con 0x55957af76030 2024-03-28T05:27:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f5306f69740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55957aed5bb0 con 0x55957af7ee00 2024-03-28T05:27:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f530291b640 1 --2- 172.21.15.23:0/349669228 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f52e80a2a10 0x7f52e80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f52f37fe640 1 -- 172.21.15.23:0/349669228 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f52f80cf5e0 con 0x7f52fc1007f0 2024-03-28T05:27:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f530211a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af75710 0x55957af75af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f530291b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55957af7ee00 0x55957af811f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f530211a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af75710 0x55957af75af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51336/0 (socket says 172.21.15.23:51336) 2024-03-28T05:27:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f530211a640 1 -- 172.21.15.23:0/402192986 learned_addr learned my addr 172.21.15.23:0/402192986 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:27:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f5301919640 1 --2- 172.21.15.23:0/402192986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55957af76030 0x55957af7e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f530211a640 1 -- 172.21.15.23:0/402192986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55957af7ee00 msgr2=0x55957af811f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f530211a640 1 --2- 172.21.15.23:0/402192986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55957af7ee00 0x55957af811f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f530211a640 1 -- 172.21.15.23:0/402192986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55957af76030 msgr2=0x55957af7e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f530211a640 1 --2- 172.21.15.23:0/402192986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55957af76030 0x55957af7e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f530211a640 1 -- 172.21.15.23:0/402192986 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55957abee210 con 0x55957af75710 2024-03-28T05:27:23.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f530291b640 1 --2- 172.21.15.23:0/349669228 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f52e80a2a10 0x7f52e80a4ed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f52f4004af0 tx=0x7f52f40323c0 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:23.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f530211a640 1 --2- 172.21.15.23:0/402192986 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af75710 0x55957af75af0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f52f8098280 tx=0x7f52f8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0b68ba25e73a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:23.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f52f0bfe640 1 -- 172.21.15.23:0/402192986 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f809bfb0 con 0x55957af75710 2024-03-28T05:27:23.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f52f0bfe640 1 -- 172.21.15.23:0/402192986 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52f80120f0 con 0x55957af75710 2024-03-28T05:27:23.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.842+0000 7f52f0bfe640 1 -- 172.21.15.23:0/402192986 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f80a4050 con 0x55957af75710 2024-03-28T05:27:23.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.843+0000 7f5306f69740 1 -- 172.21.15.23:0/402192986 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af75710 msgr2=0x55957af75af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:23.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.843+0000 7f5306f69740 1 --2- 172.21.15.23:0/402192986 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af75710 0x55957af75af0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f52f8098280 tx=0x7f52f8017000 comp rx=0 tx=0).stop 2024-03-28T05:27:23.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.843+0000 7f5306f69740 1 -- 172.21.15.23:0/402192986 shutdown_connections 2024-03-28T05:27:23.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.843+0000 7f5306f69740 1 --2- 172.21.15.23:0/402192986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55957af7ee00 0x55957af811f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.843+0000 7f5306f69740 1 --2- 172.21.15.23:0/402192986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55957af76030 0x55957af7e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.843+0000 7f5306f69740 1 --2- 172.21.15.23:0/402192986 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af75710 0x55957af75af0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.843+0000 7f5306f69740 1 -- 172.21.15.23:0/402192986 >> 172.21.15.23:0/402192986 conn(0x55957af70030 msgr2=0x55957af70410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:23.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.843+0000 7f5306f69740 1 -- 172.21.15.23:0/402192986 shutdown_connections 2024-03-28T05:27:23.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.843+0000 7f5306f69740 1 -- 172.21.15.23:0/402192986 wait complete. 2024-03-28T05:27:23.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.844+0000 7f5306f69740 1 -- start start 2024-03-28T05:27:23.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.844+0000 7f5306f69740 1 Processor -- start 2024-03-28T05:27:23.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.844+0000 7f5306f69740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55957af75710 0x55957af7bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:23.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.844+0000 7f5306f69740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55957af76030 0x55957af7c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:23.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.844+0000 7f5306f69740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af7ee00 0x55957af7c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:23.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.844+0000 7f5306f69740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55957af81a30 con 0x55957af7ee00 2024-03-28T05:27:23.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.844+0000 7f5306f69740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55957ae95d50 con 0x55957af76030 2024-03-28T05:27:23.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.844+0000 7f5306f69740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55957af81bb0 con 0x55957af75710 2024-03-28T05:27:23.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.844+0000 7f530291b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af7ee00 0x55957af7c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:23.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.844+0000 7f5306f69740 10 client.0 _ll_register_callbacks cb 0x55957ae95720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:27:23.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.844+0000 7f530291b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af7ee00 0x55957af7c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51344/0 (socket says 172.21.15.23:51344) 2024-03-28T05:27:23.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.844+0000 7f530291b640 1 -- 172.21.15.23:0/558134969 learned_addr learned my addr 172.21.15.23:0/558134969 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:27:23.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.844+0000 7f5301919640 1 --2- 172.21.15.23:0/558134969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55957af76030 0x55957af7c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:23.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.844+0000 7f530211a640 1 --2- 172.21.15.23:0/558134969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55957af75710 0x55957af7bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.844+0000 7f530291b640 1 -- 172.21.15.23:0/558134969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55957af75710 msgr2=0x55957af7bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.844+0000 7f530291b640 1 --2- 172.21.15.23:0/558134969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55957af75710 0x55957af7bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.844+0000 7f530291b640 1 -- 172.21.15.23:0/558134969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55957af76030 msgr2=0x55957af7c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.844+0000 7f530291b640 1 --2- 172.21.15.23:0/558134969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55957af76030 0x55957af7c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.844+0000 7f530291b640 1 -- 172.21.15.23:0/558134969 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55957acba620 con 0x55957af7ee00 2024-03-28T05:27:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.844+0000 7f530291b640 1 --2- 172.21.15.23:0/558134969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af7ee00 0x55957af7c940 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f52f4009ca0 tx=0x7f52f402c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60865677a515d04c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.844+0000 7f52e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:27:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.845+0000 7f52e5ffb640 1 -- 172.21.15.23:0/558134969 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f4039040 con 0x55957af7ee00 2024-03-28T05:27:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.845+0000 7f52e5ffb640 1 -- 172.21.15.23:0/558134969 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f52f4003a30 con 0x55957af7ee00 2024-03-28T05:27:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.845+0000 7f5306f69740 1 -- 172.21.15.23:0/558134969 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f52f80120f0 con 0x55957af7ee00 2024-03-28T05:27:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.845+0000 7f52e5ffb640 1 -- 172.21.15.23:0/558134969 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52f405b050 con 0x55957af7ee00 2024-03-28T05:27:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.845+0000 7f5306f69740 10 client.15783 fetch_fsmap learned FSMap version 20 2024-03-28T05:27:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.845+0000 7f5306f69740 1 -- 172.21.15.23:0/558134969 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55957af8dd20 con 0x55957af7ee00 2024-03-28T05:27:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.845+0000 7f52e5ffb640 1 -- 172.21.15.23:0/558134969 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52f80120f0 con 0x55957af7ee00 2024-03-28T05:27:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.845+0000 7f52e5ffb640 1 -- 172.21.15.23:0/558134969 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f52f4034440 con 0x55957af7ee00 2024-03-28T05:27:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.845+0000 7f5306f69740 10 client.15783 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:27:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.845+0000 7f5306f69740 10 client.15783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:27:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.845+0000 7f5306f69740 20 client.15783 populate_metadata read hostname 'smithi023' 2024-03-28T05:27:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.845+0000 7f5306f69740 1 --2- 172.21.15.23:0/558134969 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55957af8f230 0x55957af91620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.845+0000 7f5306f69740 4 client.15783 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:27:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.845+0000 7f5306f69740 1 -- 172.21.15.23:0/558134969 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55957aee7050 con 0x55957af8f230 2024-03-28T05:27:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.845+0000 7f530211a640 1 --2- 172.21.15.23:0/558134969 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55957af8f230 0x55957af91620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.846+0000 7f530211a640 1 --2- 172.21.15.23:0/558134969 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55957af8f230 0x55957af91620 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74ab926b8587157e server_cookie=4f1e1fa048e59da6 in_seq=0 out_seq=0 2024-03-28T05:27:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.846+0000 7f52e5ffb640 10 client.15783 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:27:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.850+0000 7f52e5ffb640 1 -- 172.21.15.23:0/558134969 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55957aee7050 con 0x55957af8f230 2024-03-28T05:27:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.850+0000 7f52e5ffb640 10 client.15783 handle_command_reply: tid=0 2024-03-28T05:27:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.850+0000 7f5306f69740 1 client.15783 shutdown 2024-03-28T05:27:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.851+0000 7f5306f69740 1 -- 172.21.15.23:0/558134969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af7ee00 msgr2=0x55957af7c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.851+0000 7f5306f69740 1 --2- 172.21.15.23:0/558134969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af7ee00 0x55957af7c940 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f52f4009ca0 tx=0x7f52f402c0f0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.851+0000 7f5306f69740 1 -- 172.21.15.23:0/558134969 shutdown_connections 2024-03-28T05:27:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.851+0000 7f5306f69740 1 --2- 172.21.15.23:0/558134969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af7ee00 0x55957af7c940 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.851+0000 7f5306f69740 1 --2- 172.21.15.23:0/558134969 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55957af8f230 0x55957af91620 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.851+0000 7f5306f69740 1 --2- 172.21.15.23:0/558134969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55957af76030 0x55957af7c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.851+0000 7f5306f69740 1 --2- 172.21.15.23:0/558134969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55957af75710 0x55957af7bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.851+0000 7f52e5ffb640 0 client.15783 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:27:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.851+0000 7f5306f69740 1 -- 172.21.15.23:0/558134969 >> 172.21.15.23:0/558134969 conn(0x55957af70030 msgr2=0x55957af71d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.851+0000 7f5306f69740 1 -- 172.21.15.23:0/558134969 shutdown_connections 2024-03-28T05:27:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.851+0000 7f5306f69740 1 -- 172.21.15.23:0/558134969 wait complete. 2024-03-28T05:27:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.851+0000 7f5306f69740 20 client.15783 trim_cache size 0 max 16384 2024-03-28T05:27:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.866+0000 7f5306f69740 1 Processor -- start 2024-03-28T05:27:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.866+0000 7f5306f69740 1 -- start start 2024-03-28T05:27:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.866+0000 7f5306f69740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55957af75710 0x55957af7e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.866+0000 7f5306f69740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af76030 0x55957af8a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.866+0000 7f5306f69740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55957af7ee00 0x55957af8a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.866+0000 7f5306f69740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc1ac9e0 con 0x55957af76030 2024-03-28T05:27:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.866+0000 7f5306f69740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc1aefa0 con 0x55957af75710 2024-03-28T05:27:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.866+0000 7f5306f69740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc1acfe0 con 0x55957af7ee00 2024-03-28T05:27:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.866+0000 7f5301919640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af76030 0x55957af8a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.866+0000 7f530291b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55957af7ee00 0x55957af8a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.866+0000 7f5301919640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af76030 0x55957af8a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51348/0 (socket says 172.21.15.23:51348) 2024-03-28T05:27:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.866+0000 7f5301919640 1 -- 172.21.15.23:0/3271705649 learned_addr learned my addr 172.21.15.23:0/3271705649 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:27:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.866+0000 7f530211a640 1 --2- 172.21.15.23:0/3271705649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55957af75710 0x55957af7e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.867+0000 7f530291b640 1 -- 172.21.15.23:0/3271705649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55957af75710 msgr2=0x55957af7e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.867+0000 7f530291b640 1 --2- 172.21.15.23:0/3271705649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55957af75710 0x55957af7e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.867+0000 7f530291b640 1 -- 172.21.15.23:0/3271705649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af76030 msgr2=0x55957af8a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.867+0000 7f530291b640 1 --2- 172.21.15.23:0/3271705649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af76030 0x55957af8a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.867+0000 7f530291b640 1 -- 172.21.15.23:0/3271705649 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55957af91990 con 0x55957af7ee00 2024-03-28T05:27:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.867+0000 7f5301919640 1 --2- 172.21.15.23:0/3271705649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af76030 0x55957af8a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:27:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.867+0000 7f530211a640 1 --2- 172.21.15.23:0/3271705649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55957af75710 0x55957af7e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:27:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.867+0000 7f530291b640 1 --2- 172.21.15.23:0/3271705649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55957af7ee00 0x55957af8a800 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f52f40348e0 tx=0x7f52f4005fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16fd110524dd1553 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.867+0000 7f52e7fff640 1 -- 172.21.15.23:0/3271705649 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f4063070 con 0x55957af7ee00 2024-03-28T05:27:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.867+0000 7f52e7fff640 1 -- 172.21.15.23:0/3271705649 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52f4052070 con 0x55957af7ee00 2024-03-28T05:27:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.867+0000 7f52e7fff640 1 -- 172.21.15.23:0/3271705649 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f4039070 con 0x55957af7ee00 2024-03-28T05:27:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.868+0000 7f5306f69740 1 -- 172.21.15.23:0/3271705649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55957af7ee00 msgr2=0x55957af8a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.868+0000 7f5306f69740 1 --2- 172.21.15.23:0/3271705649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55957af7ee00 0x55957af8a800 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f52f40348e0 tx=0x7f52f4005fb0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.868+0000 7f5306f69740 1 -- 172.21.15.23:0/3271705649 shutdown_connections 2024-03-28T05:27:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.868+0000 7f5306f69740 1 --2- 172.21.15.23:0/3271705649 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55957af7ee00 0x55957af8a800 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.868+0000 7f5306f69740 1 --2- 172.21.15.23:0/3271705649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af76030 0x55957af8a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.868+0000 7f5306f69740 1 --2- 172.21.15.23:0/3271705649 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55957af75710 0x55957af7e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.868+0000 7f5306f69740 1 -- 172.21.15.23:0/3271705649 >> 172.21.15.23:0/3271705649 conn(0x55957af70030 msgr2=0x55957af771c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.868+0000 7f5306f69740 1 -- 172.21.15.23:0/3271705649 shutdown_connections 2024-03-28T05:27:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.868+0000 7f5306f69740 1 -- 172.21.15.23:0/3271705649 wait complete. 2024-03-28T05:27:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.868+0000 7f5306f69740 1 -- start start 2024-03-28T05:27:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.868+0000 7f5306f69740 1 Processor -- start 2024-03-28T05:27:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f5306f69740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af75710 0x55957af78710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f5306f69740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55957af76030 0x55957af78c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f5306f69740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55957af7ee00 0x55957af79190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f5306f69740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc1ad620 con 0x55957af75710 2024-03-28T05:27:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f5306f69740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc1ae940 con 0x55957af76030 2024-03-28T05:27:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f5306f69740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc1adc80 con 0x55957af7ee00 2024-03-28T05:27:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f5306f69740 10 client.0 _ll_register_callbacks cb 0x7f52f4009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:27:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f530211a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af75710 0x55957af78710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f530211a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af75710 0x55957af78710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51354/0 (socket says 172.21.15.23:51354) 2024-03-28T05:27:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f530211a640 1 -- 172.21.15.23:0/3186655431 learned_addr learned my addr 172.21.15.23:0/3186655431 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:27:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f530291b640 1 --2- 172.21.15.23:0/3186655431 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55957af7ee00 0x55957af79190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f5301919640 1 --2- 172.21.15.23:0/3186655431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55957af76030 0x55957af78c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f530211a640 1 -- 172.21.15.23:0/3186655431 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55957af7ee00 msgr2=0x55957af79190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f530211a640 1 --2- 172.21.15.23:0/3186655431 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55957af7ee00 0x55957af79190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f530211a640 1 -- 172.21.15.23:0/3186655431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55957af76030 msgr2=0x55957af78c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f530211a640 1 --2- 172.21.15.23:0/3186655431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55957af76030 0x55957af78c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f530211a640 1 -- 172.21.15.23:0/3186655431 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55957afe9ba0 con 0x55957af75710 2024-03-28T05:27:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f530211a640 1 --2- 172.21.15.23:0/3186655431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af75710 0x55957af78710 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f52f809b5b0 tx=0x7f52f8009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7473f81d01ba10 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f52e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:27:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f52e5ffb640 1 -- 172.21.15.23:0/3186655431 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f80b8070 con 0x55957af75710 2024-03-28T05:27:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f52e5ffb640 1 -- 172.21.15.23:0/3186655431 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f52f80ccc20 con 0x55957af75710 2024-03-28T05:27:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f5306f69740 1 -- 172.21.15.23:0/3186655431 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f52f4052070 con 0x55957af75710 2024-03-28T05:27:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f52e5ffb640 1 -- 172.21.15.23:0/3186655431 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52f809b3f0 con 0x55957af75710 2024-03-28T05:27:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f5306f69740 10 client.15789 fetch_fsmap learned FSMap version 20 2024-03-28T05:27:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.869+0000 7f5306f69740 1 -- 172.21.15.23:0/3186655431 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55957afeaeb0 con 0x55957af75710 2024-03-28T05:27:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.870+0000 7f52e5ffb640 1 -- 172.21.15.23:0/3186655431 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52f4052070 con 0x55957af75710 2024-03-28T05:27:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.870+0000 7f52e5ffb640 1 -- 172.21.15.23:0/3186655431 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f52f80a6030 con 0x55957af75710 2024-03-28T05:27:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.870+0000 7f5306f69740 10 client.15789 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:27:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.870+0000 7f5306f69740 10 client.15789 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:27:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.870+0000 7f5306f69740 20 client.15789 populate_metadata read hostname 'smithi023' 2024-03-28T05:27:23.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.870+0000 7f5306f69740 1 --2- 172.21.15.23:0/3186655431 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55957afeb1a0 0x55957afed590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:23.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.870+0000 7f5306f69740 4 client.15789 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:27:23.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.870+0000 7f5306f69740 1 -- 172.21.15.23:0/3186655431 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55957aee13a0 con 0x55957afeb1a0 2024-03-28T05:27:23.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.870+0000 7f5301919640 1 --2- 172.21.15.23:0/3186655431 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55957afeb1a0 0x55957afed590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:23.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.870+0000 7f5301919640 1 --2- 172.21.15.23:0/3186655431 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55957afeb1a0 0x55957afed590 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a422c0173a5d985 server_cookie=d6bb9e5a144740c4 in_seq=0 out_seq=0 2024-03-28T05:27:23.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:23.870+0000 7f52e5ffb640 10 client.15789 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:27:24.323 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.322+0000 7f52e5ffb640 1 -- 172.21.15.23:0/3186655431 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55957aee13a0 con 0x55957afeb1a0 2024-03-28T05:27:24.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.322+0000 7f52e5ffb640 10 client.15789 handle_command_reply: tid=0 2024-03-28T05:27:24.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.322+0000 7f5306f69740 1 client.15789 shutdown 2024-03-28T05:27:24.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.322+0000 7f5306f69740 1 -- 172.21.15.23:0/3186655431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af75710 msgr2=0x55957af78710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:24.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.322+0000 7f5306f69740 1 --2- 172.21.15.23:0/3186655431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af75710 0x55957af78710 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f52f809b5b0 tx=0x7f52f8009790 comp rx=0 tx=0).stop 2024-03-28T05:27:24.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.322+0000 7f5306f69740 1 -- 172.21.15.23:0/3186655431 shutdown_connections 2024-03-28T05:27:24.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.323+0000 7f5306f69740 1 --2- 172.21.15.23:0/3186655431 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55957af7ee00 0x55957af79190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:24.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.323+0000 7f5306f69740 1 --2- 172.21.15.23:0/3186655431 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55957afeb1a0 0x55957afed590 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:24.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.323+0000 7f5306f69740 1 --2- 172.21.15.23:0/3186655431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55957af76030 0x55957af78c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:24.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.323+0000 7f5306f69740 1 --2- 172.21.15.23:0/3186655431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55957af75710 0x55957af78710 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:24.324 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.323+0000 7f52e5ffb640 0 client.15789 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:27:24.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.323+0000 7f5306f69740 1 -- 172.21.15.23:0/3186655431 >> 172.21.15.23:0/3186655431 conn(0x55957af70030 msgr2=0x55957af6e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:24.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.323+0000 7f5306f69740 1 -- 172.21.15.23:0/3186655431 shutdown_connections 2024-03-28T05:27:24.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.323+0000 7f5306f69740 1 -- 172.21.15.23:0/3186655431 wait complete. 2024-03-28T05:27:24.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.323+0000 7f5306f69740 20 client.15789 trim_cache size 0 max 16384 2024-03-28T05:27:24.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:27:24.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:27:24.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:27:24.325 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:27:24.326 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.323+0000 7f530437e640 1 -- 172.21.15.23:0/349669228 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f52e80a2a10 msgr2=0x7f52e80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:24.326 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.323+0000 7f530437e640 1 --2- 172.21.15.23:0/349669228 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f52e80a2a10 0x7f52e80a4ed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f52f4004af0 tx=0x7f52f40323c0 comp rx=0 tx=0).stop 2024-03-28T05:27:24.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.323+0000 7f530437e640 1 -- 172.21.15.23:0/349669228 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52fc1007f0 msgr2=0x7f52fc197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:24.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.323+0000 7f530437e640 1 --2- 172.21.15.23:0/349669228 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52fc1007f0 0x7f52fc197ef0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f52f800e9f0 tx=0x7f52f800edf0 comp rx=0 tx=0).stop 2024-03-28T05:27:24.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.323+0000 7f530437e640 1 -- 172.21.15.23:0/349669228 shutdown_connections 2024-03-28T05:27:24.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.323+0000 7f530437e640 1 --2- 172.21.15.23:0/349669228 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f52e80a2a10 0x7f52e80a4ed0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:24.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.323+0000 7f530437e640 1 --2- 172.21.15.23:0/349669228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f52fc105540 0x7f52fc19c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:24.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.323+0000 7f530437e640 1 --2- 172.21.15.23:0/349669228 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f52fc1011b0 0x7f52fc198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:24.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.323+0000 7f530437e640 1 --2- 172.21.15.23:0/349669228 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52fc1007f0 0x7f52fc197ef0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:24.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.323+0000 7f530437e640 1 -- 172.21.15.23:0/349669228 >> 172.21.15.23:0/349669228 conn(0x7f52fc0773f0 msgr2=0x7f52fc068f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:24.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.323+0000 7f530437e640 1 -- 172.21.15.23:0/349669228 shutdown_connections 2024-03-28T05:27:24.327 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:24.323+0000 7f530437e640 1 -- 172.21.15.23:0/349669228 wait complete. 2024-03-28T05:27:25.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:25 smithi098 ceph-mon[36087]: pgmap v118: 129 pgs: 129 active+clean; 108 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 463 KiB/s wr, 198 op/s 2024-03-28T05:27:25.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:25 smithi113 ceph-mon[37164]: pgmap v118: 129 pgs: 129 active+clean; 108 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 463 KiB/s wr, 198 op/s 2024-03-28T05:27:25.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:25 smithi023 ceph-mon[31954]: pgmap v118: 129 pgs: 129 active+clean; 108 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 463 KiB/s wr, 198 op/s 2024-03-28T05:27:26.399 INFO:tasks.workunit.client.0.smithi023.stdout: 36 225382 97 151917 179 93737 357 2024-03-28T05:27:27.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:27 smithi098 ceph-mon[36087]: pgmap v119: 129 pgs: 129 active+clean; 104 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 811 KiB/s wr, 466 op/s 2024-03-28T05:27:27.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:27 smithi113 ceph-mon[37164]: pgmap v119: 129 pgs: 129 active+clean; 104 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 811 KiB/s wr, 466 op/s 2024-03-28T05:27:27.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:27 smithi023 ceph-mon[31954]: pgmap v119: 129 pgs: 129 active+clean; 104 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 811 KiB/s wr, 466 op/s 2024-03-28T05:27:29.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:29 smithi098 ceph-mon[36087]: pgmap v120: 129 pgs: 129 active+clean; 104 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 611 KiB/s wr, 292 op/s 2024-03-28T05:27:29.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:29 smithi113 ceph-mon[37164]: pgmap v120: 129 pgs: 129 active+clean; 104 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 611 KiB/s wr, 292 op/s 2024-03-28T05:27:29.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:29 smithi023 ceph-mon[31954]: pgmap v120: 129 pgs: 129 active+clean; 104 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 611 KiB/s wr, 292 op/s 2024-03-28T05:27:31.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:31 smithi098 ceph-mon[36087]: pgmap v121: 129 pgs: 129 active+clean; 107 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 963 KiB/s wr, 321 op/s 2024-03-28T05:27:31.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:31 smithi113 ceph-mon[37164]: pgmap v121: 129 pgs: 129 active+clean; 107 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 963 KiB/s wr, 321 op/s 2024-03-28T05:27:31.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:31 smithi023 ceph-mon[31954]: pgmap v121: 129 pgs: 129 active+clean; 107 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 963 KiB/s wr, 321 op/s 2024-03-28T05:27:33.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:33 smithi098 ceph-mon[36087]: pgmap v122: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 2.2 MiB/s wr, 415 op/s 2024-03-28T05:27:33.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:33 smithi113 ceph-mon[37164]: pgmap v122: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 2.2 MiB/s wr, 415 op/s 2024-03-28T05:27:33.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:33 smithi023 ceph-mon[31954]: pgmap v122: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 2.2 MiB/s wr, 415 op/s 2024-03-28T05:27:35.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:35 smithi098 ceph-mon[36087]: pgmap v123: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 2.2 MiB/s wr, 408 op/s 2024-03-28T05:27:35.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:35 smithi113 ceph-mon[37164]: pgmap v123: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 2.2 MiB/s wr, 408 op/s 2024-03-28T05:27:35.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:35 smithi023 ceph-mon[31954]: pgmap v123: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 2.2 MiB/s wr, 408 op/s 2024-03-28T05:27:36.399 INFO:tasks.workunit.client.0.smithi023.stdout: 37 258395 84 178242 73 104198 207 2024-03-28T05:27:37.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:37 smithi098 ceph-mon[36087]: pgmap v124: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 2.3 MiB/s wr, 431 op/s 2024-03-28T05:27:37.429 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.429+0000 7fbf0ccf8640 1 -- 172.21.15.23:0/3720187805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf08107770 msgr2=0x7fbf08107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:37.429 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.429+0000 7fbf0ccf8640 1 --2- 172.21.15.23:0/3720187805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf08107770 0x7fbf08107b50 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fbef8009930 tx=0x7fbef802bc30 comp rx=0 tx=0).stop 2024-03-28T05:27:37.429 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.429+0000 7fbf0ccf8640 1 -- 172.21.15.23:0/3720187805 shutdown_connections 2024-03-28T05:27:37.429 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.429+0000 7fbf0ccf8640 1 --2- 172.21.15.23:0/3720187805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf080696c0 0x7fbf0810e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.430 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.429+0000 7fbf0ccf8640 1 --2- 172.21.15.23:0/3720187805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf08068d20 0x7fbf08069180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.430 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.429+0000 7fbf0ccf8640 1 --2- 172.21.15.23:0/3720187805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf08107770 0x7fbf08107b50 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.430 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.429+0000 7fbf0ccf8640 1 -- 172.21.15.23:0/3720187805 >> 172.21.15.23:0/3720187805 conn(0x7fbf080773f0 msgr2=0x7fbf08077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:37.430 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.429+0000 7fbf0ccf8640 1 -- 172.21.15.23:0/3720187805 shutdown_connections 2024-03-28T05:27:37.430 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.429+0000 7fbf0ccf8640 1 -- 172.21.15.23:0/3720187805 wait complete. 2024-03-28T05:27:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.430+0000 7fbf0ccf8640 1 Processor -- start 2024-03-28T05:27:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.430+0000 7fbf0ccf8640 1 -- start start 2024-03-28T05:27:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.430+0000 7fbf0ccf8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf08068d20 0x7fbf081008c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.430+0000 7fbf0ccf8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf080696c0 0x7fbf08100e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.430+0000 7fbf0ccf8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf08107770 0x7fbf08101340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.430+0000 7fbf0ccf8640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf08111a60 con 0x7fbf08068d20 2024-03-28T05:27:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.430+0000 7fbf0ccf8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf081118e0 con 0x7fbf080696c0 2024-03-28T05:27:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.430+0000 7fbf0ccf8640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf08111be0 con 0x7fbf08107770 2024-03-28T05:27:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.430+0000 7fbf0659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf08068d20 0x7fbf081008c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.430+0000 7fbf06d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf08107770 0x7fbf08101340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.430+0000 7fbf0659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf08068d20 0x7fbf081008c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50642/0 (socket says 172.21.15.23:50642) 2024-03-28T05:27:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.430+0000 7fbf0659c640 1 -- 172.21.15.23:0/3854563573 learned_addr learned my addr 172.21.15.23:0/3854563573 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:27:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.430+0000 7fbf05d9b640 1 --2- 172.21.15.23:0/3854563573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf080696c0 0x7fbf08100e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.430+0000 7fbf06d9d640 1 -- 172.21.15.23:0/3854563573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf080696c0 msgr2=0x7fbf08100e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.430+0000 7fbf06d9d640 1 --2- 172.21.15.23:0/3854563573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf080696c0 0x7fbf08100e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.430+0000 7fbf06d9d640 1 -- 172.21.15.23:0/3854563573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf08068d20 msgr2=0x7fbf081008c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.430+0000 7fbf06d9d640 1 --2- 172.21.15.23:0/3854563573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf08068d20 0x7fbf081008c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.430+0000 7fbf06d9d640 1 -- 172.21.15.23:0/3854563573 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf081a63c0 con 0x7fbf08107770 2024-03-28T05:27:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.430+0000 7fbf0659c640 1 --2- 172.21.15.23:0/3854563573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf08068d20 0x7fbf081008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:27:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.430+0000 7fbf06d9d640 1 --2- 172.21.15.23:0/3854563573 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf08107770 0x7fbf08101340 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbf00005fd0 tx=0x7fbf0000ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.431+0000 7fbef77fe640 1 -- 172.21.15.23:0/3854563573 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0000cd80 con 0x7fbf08107770 2024-03-28T05:27:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.431+0000 7fbf0ccf8640 1 -- 172.21.15.23:0/3854563573 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf081a66b0 con 0x7fbf08107770 2024-03-28T05:27:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.431+0000 7fbef77fe640 1 -- 172.21.15.23:0/3854563573 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf00005020 con 0x7fbf08107770 2024-03-28T05:27:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.431+0000 7fbef77fe640 1 -- 172.21.15.23:0/3854563573 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf00010720 con 0x7fbf08107770 2024-03-28T05:27:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.431+0000 7fbf0ccf8640 1 -- 172.21.15.23:0/3854563573 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf081a6c50 con 0x7fbf08107770 2024-03-28T05:27:37.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.433+0000 7fbef77fe640 1 -- 172.21.15.23:0/3854563573 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fbf000051d0 con 0x7fbf08107770 2024-03-28T05:27:37.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.433+0000 7fbef77fe640 1 --2- 172.21.15.23:0/3854563573 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fbef00a2a10 0x7fbef00a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:37.439 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.433+0000 7fbf0f9c3740 1 Processor -- start 2024-03-28T05:27:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.434+0000 7fbf0659c640 1 --2- 172.21.15.23:0/3854563573 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fbef00a2a10 0x7fbef00a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.434+0000 7fbef77fe640 1 -- 172.21.15.23:0/3854563573 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fbf000cf6f0 con 0x7fbf08107770 2024-03-28T05:27:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.434+0000 7fbf0f9c3740 1 -- start start 2024-03-28T05:27:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.434+0000 7fbf0f9c3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a477d710 0x5627a477daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.434+0000 7fbf0f9c3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627a477e030 0x5627a47868c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.434+0000 7fbf0f9c3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627a4786e00 0x5627a47891f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.434+0000 7fbf0f9c3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627a46e1b10 con 0x5627a477d710 2024-03-28T05:27:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.434+0000 7fbf0f9c3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627a468a900 con 0x5627a4786e00 2024-03-28T05:27:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.434+0000 7fbf0f9c3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627a46ddbb0 con 0x5627a477e030 2024-03-28T05:27:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.434+0000 7fbf0659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a477d710 0x5627a477daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.434+0000 7fbf05d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627a477e030 0x5627a47868c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.434+0000 7fbf0659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a477d710 0x5627a477daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50650/0 (socket says 172.21.15.23:50650) 2024-03-28T05:27:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.434+0000 7fbf0659c640 1 -- 172.21.15.23:0/3419333739 learned_addr learned my addr 172.21.15.23:0/3419333739 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:27:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.434+0000 7fbf0659c640 1 --2- 172.21.15.23:0/3854563573 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fbef00a2a10 0x7fbef00a4ed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbef8000c00 tx=0x7fbef8009640 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.434+0000 7fbf06d9d640 1 --2- 172.21.15.23:0/3419333739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627a4786e00 0x5627a47891f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.434+0000 7fbf0659c640 1 -- 172.21.15.23:0/3419333739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627a477e030 msgr2=0x5627a47868c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.434+0000 7fbf0659c640 1 --2- 172.21.15.23:0/3419333739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627a477e030 0x5627a47868c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.434+0000 7fbf0659c640 1 -- 172.21.15.23:0/3419333739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627a4786e00 msgr2=0x5627a47891f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.434+0000 7fbf0659c640 1 --2- 172.21.15.23:0/3419333739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627a4786e00 0x5627a47891f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.434+0000 7fbf0659c640 1 -- 172.21.15.23:0/3419333739 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627a43f6210 con 0x5627a477d710 2024-03-28T05:27:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.435+0000 7fbf0659c640 1 --2- 172.21.15.23:0/3419333739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a477d710 0x5627a477daf0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fbef8036040 tx=0x7fbef803f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d7f780fe17eabe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.435+0000 7fbef4ff9640 1 -- 172.21.15.23:0/3419333739 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef804e070 con 0x5627a477d710 2024-03-28T05:27:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.435+0000 7fbef4ff9640 1 -- 172.21.15.23:0/3419333739 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbef8004f60 con 0x5627a477d710 2024-03-28T05:27:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.435+0000 7fbef4ff9640 1 -- 172.21.15.23:0/3419333739 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef80038e0 con 0x5627a477d710 2024-03-28T05:27:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.435+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/3419333739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a477d710 msgr2=0x5627a477daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.435+0000 7fbf0f9c3740 1 --2- 172.21.15.23:0/3419333739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a477d710 0x5627a477daf0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fbef8036040 tx=0x7fbef803f000 comp rx=0 tx=0).stop 2024-03-28T05:27:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.435+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/3419333739 shutdown_connections 2024-03-28T05:27:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.435+0000 7fbf0f9c3740 1 --2- 172.21.15.23:0/3419333739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627a4786e00 0x5627a47891f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.435+0000 7fbf0f9c3740 1 --2- 172.21.15.23:0/3419333739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627a477e030 0x5627a47868c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.435+0000 7fbf0f9c3740 1 --2- 172.21.15.23:0/3419333739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a477d710 0x5627a477daf0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.435+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/3419333739 >> 172.21.15.23:0/3419333739 conn(0x5627a4778030 msgr2=0x5627a4778410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.436+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/3419333739 shutdown_connections 2024-03-28T05:27:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.436+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/3419333739 wait complete. 2024-03-28T05:27:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.436+0000 7fbf0f9c3740 1 -- start start 2024-03-28T05:27:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.436+0000 7fbf0f9c3740 1 Processor -- start 2024-03-28T05:27:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.436+0000 7fbf0f9c3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627a477d710 0x5627a4783e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.436+0000 7fbf0f9c3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627a477e030 0x5627a4784340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.436+0000 7fbf0f9c3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a4786e00 0x5627a4784880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.436+0000 7fbf0f9c3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627a47899d0 con 0x5627a4786e00 2024-03-28T05:27:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.436+0000 7fbf0f9c3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627a469dd50 con 0x5627a477d710 2024-03-28T05:27:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.436+0000 7fbf0f9c3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627a4789b50 con 0x5627a477e030 2024-03-28T05:27:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.436+0000 7fbf0f9c3740 10 client.0 _ll_register_callbacks cb 0x5627a469d720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:27:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.437+0000 7fbf06d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a4786e00 0x5627a4784880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.437+0000 7fbf06d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a4786e00 0x5627a4784880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50666/0 (socket says 172.21.15.23:50666) 2024-03-28T05:27:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.437+0000 7fbf06d9d640 1 -- 172.21.15.23:0/2706270493 learned_addr learned my addr 172.21.15.23:0/2706270493 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:27:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.437+0000 7fbf05d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627a477e030 0x5627a4784340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.437+0000 7fbf0659c640 1 --2- 172.21.15.23:0/2706270493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627a477d710 0x5627a4783e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.437+0000 7fbf06d9d640 1 -- 172.21.15.23:0/2706270493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627a477e030 msgr2=0x5627a4784340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.437+0000 7fbf06d9d640 1 --2- 172.21.15.23:0/2706270493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627a477e030 0x5627a4784340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.437+0000 7fbf06d9d640 1 -- 172.21.15.23:0/2706270493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627a477d710 msgr2=0x5627a4783e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.437+0000 7fbf06d9d640 1 --2- 172.21.15.23:0/2706270493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627a477d710 0x5627a4783e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.437+0000 7fbf06d9d640 1 -- 172.21.15.23:0/2706270493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5627a44c2620 con 0x5627a4786e00 2024-03-28T05:27:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.437+0000 7fbf06d9d640 1 --2- 172.21.15.23:0/2706270493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a4786e00 0x5627a4784880 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fbf0009b950 tx=0x7fbf0009c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4950c3ee859d003e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.437+0000 7fbeedffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:27:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.437+0000 7fbeedffb640 1 -- 172.21.15.23:0/2706270493 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf000b3020 con 0x5627a4786e00 2024-03-28T05:27:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.437+0000 7fbeedffb640 1 -- 172.21.15.23:0/2706270493 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fbf000cf310 con 0x5627a4786e00 2024-03-28T05:27:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.437+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/2706270493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbef8004f60 con 0x5627a4786e00 2024-03-28T05:27:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.437+0000 7fbeedffb640 1 -- 172.21.15.23:0/2706270493 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf000cf510 con 0x5627a4786e00 2024-03-28T05:27:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.437+0000 7fbf0f9c3740 10 client.15801 fetch_fsmap learned FSMap version 20 2024-03-28T05:27:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.437+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/2706270493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5627a4795c50 con 0x5627a4786e00 2024-03-28T05:27:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.438+0000 7fbeedffb640 1 -- 172.21.15.23:0/2706270493 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbef8004f60 con 0x5627a4786e00 2024-03-28T05:27:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.438+0000 7fbeedffb640 1 -- 172.21.15.23:0/2706270493 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fbf000860b0 con 0x5627a4786e00 2024-03-28T05:27:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.438+0000 7fbf0f9c3740 10 client.15801 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:27:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.438+0000 7fbf0f9c3740 10 client.15801 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:27:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.438+0000 7fbf0f9c3740 20 client.15801 populate_metadata read hostname 'smithi023' 2024-03-28T05:27:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.438+0000 7fbf0f9c3740 1 --2- 172.21.15.23:0/2706270493 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5627a47970c0 0x5627a47994b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.438+0000 7fbf0f9c3740 4 client.15801 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:27:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.438+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/2706270493 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5627a46ef050 con 0x5627a47970c0 2024-03-28T05:27:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.438+0000 7fbf0659c640 1 --2- 172.21.15.23:0/2706270493 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5627a47970c0 0x5627a47994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.438+0000 7fbf0659c640 1 --2- 172.21.15.23:0/2706270493 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5627a47970c0 0x5627a47994b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57d63527011fb2f8 server_cookie=4b9aeb81a14cac15 in_seq=0 out_seq=0 2024-03-28T05:27:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.438+0000 7fbeedffb640 10 client.15801 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:27:37.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.443+0000 7fbeedffb640 1 -- 172.21.15.23:0/2706270493 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5627a46ef050 con 0x5627a47970c0 2024-03-28T05:27:37.445 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.443+0000 7fbeedffb640 10 client.15801 handle_command_reply: tid=0 2024-03-28T05:27:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.444+0000 7fbf0f9c3740 1 client.15801 shutdown 2024-03-28T05:27:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.444+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/2706270493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a4786e00 msgr2=0x5627a4784880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.444+0000 7fbf0f9c3740 1 --2- 172.21.15.23:0/2706270493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a4786e00 0x5627a4784880 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fbf0009b950 tx=0x7fbf0009c180 comp rx=0 tx=0).stop 2024-03-28T05:27:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.444+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/2706270493 shutdown_connections 2024-03-28T05:27:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.444+0000 7fbf0f9c3740 1 --2- 172.21.15.23:0/2706270493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a4786e00 0x5627a4784880 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.444+0000 7fbf0f9c3740 1 --2- 172.21.15.23:0/2706270493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627a477e030 0x5627a4784340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.444+0000 7fbf0f9c3740 1 --2- 172.21.15.23:0/2706270493 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5627a47970c0 0x5627a47994b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.444+0000 7fbf0f9c3740 1 --2- 172.21.15.23:0/2706270493 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627a477d710 0x5627a4783e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.444+0000 7fbeedffb640 0 client.15801 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:27:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.444+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/2706270493 >> 172.21.15.23:0/2706270493 conn(0x5627a4778030 msgr2=0x5627a4779d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.444+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/2706270493 shutdown_connections 2024-03-28T05:27:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.444+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/2706270493 wait complete. 2024-03-28T05:27:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.444+0000 7fbf0f9c3740 20 client.15801 trim_cache size 0 max 16384 2024-03-28T05:27:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.458+0000 7fbf0f9c3740 1 Processor -- start 2024-03-28T05:27:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.458+0000 7fbf0f9c3740 1 -- start start 2024-03-28T05:27:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.459+0000 7fbf0f9c3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627a477d710 0x5627a4786530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.459+0000 7fbf0f9c3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627a477e030 0x5627a4779b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.459+0000 7fbf0f9c3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a4786e00 0x5627a4792260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.459+0000 7fbf0f9c3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf081aec00 con 0x5627a4786e00 2024-03-28T05:27:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.459+0000 7fbf0f9c3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf081b11c0 con 0x5627a477d710 2024-03-28T05:27:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.459+0000 7fbf0f9c3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf081af200 con 0x5627a477e030 2024-03-28T05:27:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.459+0000 7fbf06d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a4786e00 0x5627a4792260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.459+0000 7fbf06d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a4786e00 0x5627a4792260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50670/0 (socket says 172.21.15.23:50670) 2024-03-28T05:27:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.459+0000 7fbf06d9d640 1 -- 172.21.15.23:0/3742876202 learned_addr learned my addr 172.21.15.23:0/3742876202 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:27:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.459+0000 7fbf05d9b640 1 --2- 172.21.15.23:0/3742876202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627a477e030 0x5627a4779b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.459+0000 7fbf0659c640 1 --2- 172.21.15.23:0/3742876202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627a477d710 0x5627a4786530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.459+0000 7fbf06d9d640 1 -- 172.21.15.23:0/3742876202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627a477e030 msgr2=0x5627a4779b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.459+0000 7fbf06d9d640 1 --2- 172.21.15.23:0/3742876202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627a477e030 0x5627a4779b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.459+0000 7fbf06d9d640 1 -- 172.21.15.23:0/3742876202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627a477d710 msgr2=0x5627a4786530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.459+0000 7fbf06d9d640 1 --2- 172.21.15.23:0/3742876202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627a477d710 0x5627a4786530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.459+0000 7fbf06d9d640 1 -- 172.21.15.23:0/3742876202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627a4777850 con 0x5627a4786e00 2024-03-28T05:27:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.459+0000 7fbf06d9d640 1 --2- 172.21.15.23:0/3742876202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a4786e00 0x5627a4792260 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fbf00097e20 tx=0x7fbf0009fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a478d9b76280c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.459+0000 7fbeeffff640 1 -- 172.21.15.23:0/3742876202 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf000b3050 con 0x5627a4786e00 2024-03-28T05:27:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.459+0000 7fbeeffff640 1 -- 172.21.15.23:0/3742876202 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf000ba170 con 0x5627a4786e00 2024-03-28T05:27:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.459+0000 7fbeeffff640 1 -- 172.21.15.23:0/3742876202 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf000a3f30 con 0x5627a4786e00 2024-03-28T05:27:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.460+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/3742876202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a4786e00 msgr2=0x5627a4792260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.460+0000 7fbf0f9c3740 1 --2- 172.21.15.23:0/3742876202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a4786e00 0x5627a4792260 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fbf00097e20 tx=0x7fbf0009fc90 comp rx=0 tx=0).stop 2024-03-28T05:27:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.460+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/3742876202 shutdown_connections 2024-03-28T05:27:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.460+0000 7fbf0f9c3740 1 --2- 172.21.15.23:0/3742876202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a4786e00 0x5627a4792260 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.460+0000 7fbf0f9c3740 1 --2- 172.21.15.23:0/3742876202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627a477e030 0x5627a4779b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.460+0000 7fbf0f9c3740 1 --2- 172.21.15.23:0/3742876202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627a477d710 0x5627a4786530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.460+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/3742876202 >> 172.21.15.23:0/3742876202 conn(0x5627a4778030 msgr2=0x5627a4779410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.460+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/3742876202 shutdown_connections 2024-03-28T05:27:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.460+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/3742876202 wait complete. 2024-03-28T05:27:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.460+0000 7fbf0f9c3740 1 -- start start 2024-03-28T05:27:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.460+0000 7fbf0f9c3740 1 Processor -- start 2024-03-28T05:27:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbf0f9c3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627a477d710 0x5627a478a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbf0f9c3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a477e030 0x5627a478ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbf0f9c3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627a4786e00 0x5627a47f1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbf0f9c3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf081af840 con 0x5627a477e030 2024-03-28T05:27:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbf0f9c3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf081b0b60 con 0x5627a477d710 2024-03-28T05:27:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbf0f9c3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf081afea0 con 0x5627a4786e00 2024-03-28T05:27:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbf0f9c3740 10 client.0 _ll_register_callbacks cb 0x7fbf0009b320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:27:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbf05d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a477e030 0x5627a478ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbf05d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a477e030 0x5627a478ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50672/0 (socket says 172.21.15.23:50672) 2024-03-28T05:27:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbf05d9b640 1 -- 172.21.15.23:0/1832134999 learned_addr learned my addr 172.21.15.23:0/1832134999 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:27:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbf06d9d640 1 --2- 172.21.15.23:0/1832134999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627a4786e00 0x5627a47f1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:37.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbf05d9b640 1 -- 172.21.15.23:0/1832134999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627a4786e00 msgr2=0x5627a47f1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbf05d9b640 1 --2- 172.21.15.23:0/1832134999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627a4786e00 0x5627a47f1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbf05d9b640 1 -- 172.21.15.23:0/1832134999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627a477d710 msgr2=0x5627a478a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbf05d9b640 1 --2- 172.21.15.23:0/1832134999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627a477d710 0x5627a478a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbf05d9b640 1 -- 172.21.15.23:0/1832134999 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5627a47f1ec0 con 0x5627a477e030 2024-03-28T05:27:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbf05d9b640 1 --2- 172.21.15.23:0/1832134999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a477e030 0x5627a478ad70 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x5627a477d1c0 tx=0x7fbefc00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c9491de6998bd70 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbeedffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:27:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbeedffb640 1 -- 172.21.15.23:0/1832134999 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc021020 con 0x5627a477e030 2024-03-28T05:27:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/1832134999 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbf000ba170 con 0x5627a477e030 2024-03-28T05:27:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbeedffb640 1 -- 172.21.15.23:0/1832134999 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fbefc005220 con 0x5627a477e030 2024-03-28T05:27:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbeedffb640 1 -- 172.21.15.23:0/1832134999 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbefc029050 con 0x5627a477e030 2024-03-28T05:27:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbf0f9c3740 10 client.15807 fetch_fsmap learned FSMap version 20 2024-03-28T05:27:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.461+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/1832134999 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5627a47f31b0 con 0x5627a477e030 2024-03-28T05:27:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.462+0000 7fbeedffb640 1 -- 172.21.15.23:0/1832134999 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf000ba170 con 0x5627a477e030 2024-03-28T05:27:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.462+0000 7fbeedffb640 1 -- 172.21.15.23:0/1832134999 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fbefc01d4d0 con 0x5627a477e030 2024-03-28T05:27:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.462+0000 7fbf0f9c3740 10 client.15807 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:27:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.462+0000 7fbf0f9c3740 10 client.15807 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:27:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.462+0000 7fbf0f9c3740 20 client.15807 populate_metadata read hostname 'smithi023' 2024-03-28T05:27:37.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.462+0000 7fbf0f9c3740 1 --2- 172.21.15.23:0/1832134999 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5627a47f3910 0x5627a47f5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.462+0000 7fbf0f9c3740 4 client.15807 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:27:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.462+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/1832134999 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5627a46e93a0 con 0x5627a47f3910 2024-03-28T05:27:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.462+0000 7fbf0659c640 1 --2- 172.21.15.23:0/1832134999 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5627a47f3910 0x5627a47f5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.462+0000 7fbeedffb640 10 client.15807 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:27:37.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.462+0000 7fbf0659c640 1 --2- 172.21.15.23:0/1832134999 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5627a47f3910 0x5627a47f5d30 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f84e7cebf0aa2f server_cookie=d0af2aa8f57a3666 in_seq=0 out_seq=0 2024-03-28T05:27:37.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:37 smithi113 ceph-mon[37164]: pgmap v124: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 2.3 MiB/s wr, 431 op/s 2024-03-28T05:27:37.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:37 smithi023 ceph-mon[31954]: pgmap v124: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 2.3 MiB/s wr, 431 op/s 2024-03-28T05:27:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.898+0000 7fbeedffb640 1 -- 172.21.15.23:0/1832134999 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5627a46e93a0 con 0x5627a47f3910 2024-03-28T05:27:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.898+0000 7fbeedffb640 10 client.15807 handle_command_reply: tid=0 2024-03-28T05:27:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.898+0000 7fbf0f9c3740 1 client.15807 shutdown 2024-03-28T05:27:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.898+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/1832134999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a477e030 msgr2=0x5627a478ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.898+0000 7fbf0f9c3740 1 --2- 172.21.15.23:0/1832134999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a477e030 0x5627a478ad70 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x5627a477d1c0 tx=0x7fbefc00bf10 comp rx=0 tx=0).stop 2024-03-28T05:27:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.898+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/1832134999 shutdown_connections 2024-03-28T05:27:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.898+0000 7fbf0f9c3740 1 --2- 172.21.15.23:0/1832134999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627a4786e00 0x5627a47f1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.898+0000 7fbf0f9c3740 1 --2- 172.21.15.23:0/1832134999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627a477e030 0x5627a478ad70 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.898+0000 7fbf0f9c3740 1 --2- 172.21.15.23:0/1832134999 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5627a47f3910 0x5627a47f5d30 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.898+0000 7fbf0f9c3740 1 --2- 172.21.15.23:0/1832134999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627a477d710 0x5627a478a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.898+0000 7fbeedffb640 0 client.15807 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:27:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.898+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/1832134999 >> 172.21.15.23:0/1832134999 conn(0x5627a4778030 msgr2=0x5627a4777020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.898+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/1832134999 shutdown_connections 2024-03-28T05:27:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.898+0000 7fbf0f9c3740 1 -- 172.21.15.23:0/1832134999 wait complete. 2024-03-28T05:27:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.898+0000 7fbf0f9c3740 20 client.15807 trim_cache size 0 max 16384 2024-03-28T05:27:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:27:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:27:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:27:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:27:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.899+0000 7fbf0ccf8640 1 -- 172.21.15.23:0/3854563573 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fbef00a2a10 msgr2=0x7fbef00a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.899+0000 7fbf0ccf8640 1 --2- 172.21.15.23:0/3854563573 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fbef00a2a10 0x7fbef00a4ed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbef8000c00 tx=0x7fbef8009640 comp rx=0 tx=0).stop 2024-03-28T05:27:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.899+0000 7fbf0ccf8640 1 -- 172.21.15.23:0/3854563573 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf08107770 msgr2=0x7fbf08101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.899+0000 7fbf0ccf8640 1 --2- 172.21.15.23:0/3854563573 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf08107770 0x7fbf08101340 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbf00005fd0 tx=0x7fbf0000ee60 comp rx=0 tx=0).stop 2024-03-28T05:27:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.899+0000 7fbf0ccf8640 1 -- 172.21.15.23:0/3854563573 shutdown_connections 2024-03-28T05:27:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.899+0000 7fbf0ccf8640 1 --2- 172.21.15.23:0/3854563573 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf08107770 0x7fbf08101340 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.899+0000 7fbf0ccf8640 1 --2- 172.21.15.23:0/3854563573 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fbef00a2a10 0x7fbef00a4ed0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.899+0000 7fbf0ccf8640 1 --2- 172.21.15.23:0/3854563573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf080696c0 0x7fbf08100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.902 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.899+0000 7fbf0ccf8640 1 --2- 172.21.15.23:0/3854563573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf08068d20 0x7fbf081008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:37.902 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.899+0000 7fbf0ccf8640 1 -- 172.21.15.23:0/3854563573 >> 172.21.15.23:0/3854563573 conn(0x7fbf080773f0 msgr2=0x7fbf080fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:37.902 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.899+0000 7fbf0ccf8640 1 -- 172.21.15.23:0/3854563573 shutdown_connections 2024-03-28T05:27:37.902 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:37.899+0000 7fbf0ccf8640 1 -- 172.21.15.23:0/3854563573 wait complete. 2024-03-28T05:27:39.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:39 smithi098 ceph-mon[36087]: pgmap v125: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 2.0 MiB/s wr, 163 op/s 2024-03-28T05:27:39.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:39 smithi113 ceph-mon[37164]: pgmap v125: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 2.0 MiB/s wr, 163 op/s 2024-03-28T05:27:39.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:39 smithi023 ceph-mon[31954]: pgmap v125: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 2.0 MiB/s wr, 163 op/s 2024-03-28T05:27:41.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:41 smithi098 ceph-mon[36087]: pgmap v126: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 2.2 MiB/s wr, 224 op/s 2024-03-28T05:27:41.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:41 smithi113 ceph-mon[37164]: pgmap v126: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 2.2 MiB/s wr, 224 op/s 2024-03-28T05:27:41.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:41 smithi023 ceph-mon[31954]: pgmap v126: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 2.2 MiB/s wr, 224 op/s 2024-03-28T05:27:43.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:43 smithi098 ceph-mon[36087]: pgmap v127: 129 pgs: 129 active+clean; 120 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 1.9 MiB/s wr, 341 op/s 2024-03-28T05:27:43.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:43 smithi113 ceph-mon[37164]: pgmap v127: 129 pgs: 129 active+clean; 120 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 1.9 MiB/s wr, 341 op/s 2024-03-28T05:27:43.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:43 smithi023 ceph-mon[31954]: pgmap v127: 129 pgs: 129 active+clean; 120 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 1.9 MiB/s wr, 341 op/s 2024-03-28T05:27:44.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:44 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:27:44.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:44 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:27:44.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:44 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:27:44.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:44 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:27:44.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:44 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:27:44.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:44 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:27:44.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:44 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:27:44.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:44 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:27:44.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:44 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:27:44.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:44 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:27:44.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:44 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:27:44.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:44 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:27:46.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:45 smithi113 ceph-mon[37164]: pgmap v128: 129 pgs: 129 active+clean; 120 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 401 KiB/s wr, 230 op/s 2024-03-28T05:27:46.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:45 smithi023 ceph-mon[31954]: pgmap v128: 129 pgs: 129 active+clean; 120 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 401 KiB/s wr, 230 op/s 2024-03-28T05:27:46.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:45 smithi098 ceph-mon[36087]: pgmap v128: 129 pgs: 129 active+clean; 120 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 401 KiB/s wr, 230 op/s 2024-03-28T05:27:46.399 INFO:tasks.workunit.client.0.smithi023.stdout: 41 439223 168 302274 125 195309 356 2024-03-28T05:27:48.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:47 smithi113 ceph-mon[37164]: pgmap v129: 129 pgs: 129 active+clean; 123 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 722 KiB/s wr, 277 op/s 2024-03-28T05:27:48.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:47 smithi023 ceph-mon[31954]: pgmap v129: 129 pgs: 129 active+clean; 123 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 722 KiB/s wr, 277 op/s 2024-03-28T05:27:48.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:47 smithi098 ceph-mon[36087]: pgmap v129: 129 pgs: 129 active+clean; 123 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 722 KiB/s wr, 277 op/s 2024-03-28T05:27:50.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:49 smithi113 ceph-mon[37164]: pgmap v130: 129 pgs: 129 active+clean; 123 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 571 KiB/s wr, 254 op/s 2024-03-28T05:27:50.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:49 smithi023 ceph-mon[31954]: pgmap v130: 129 pgs: 129 active+clean; 123 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 571 KiB/s wr, 254 op/s 2024-03-28T05:27:50.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:49 smithi098 ceph-mon[36087]: pgmap v130: 129 pgs: 129 active+clean; 123 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 571 KiB/s wr, 254 op/s 2024-03-28T05:27:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.991+0000 7f5809ff1640 1 -- 172.21.15.23:0/59131542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58041011b0 msgr2=0x7f580410d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.991+0000 7f5809ff1640 1 --2- 172.21.15.23:0/59131542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58041011b0 0x7f580410d3d0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f57fc009ca0 tx=0x7f57fc02c0f0 comp rx=0 tx=0).stop 2024-03-28T05:27:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.991+0000 7f5809ff1640 1 -- 172.21.15.23:0/59131542 shutdown_connections 2024-03-28T05:27:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.991+0000 7f5809ff1640 1 --2- 172.21.15.23:0/59131542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58041011b0 0x7f580410d3d0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.991+0000 7f5809ff1640 1 --2- 172.21.15.23:0/59131542 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58041007f0 0x7f5804100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:50.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.991+0000 7f5809ff1640 1 --2- 172.21.15.23:0/59131542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5804105540 0x7f5804105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.991+0000 7f5809ff1640 1 -- 172.21.15.23:0/59131542 >> 172.21.15.23:0/59131542 conn(0x7f58040773f0 msgr2=0x7f5804077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.991+0000 7f5809ff1640 1 -- 172.21.15.23:0/59131542 shutdown_connections 2024-03-28T05:27:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.991+0000 7f5809ff1640 1 -- 172.21.15.23:0/59131542 wait complete. 2024-03-28T05:27:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.992+0000 7f5809ff1640 1 Processor -- start 2024-03-28T05:27:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.992+0000 7f5809ff1640 1 -- start start 2024-03-28T05:27:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.992+0000 7f5809ff1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58041007f0 0x7f5804196dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.992+0000 7f5809ff1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58041011b0 0x7f5804197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.992+0000 7f5809ff1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5804105540 0x7f580419b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.992+0000 7f5809ff1640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5804110a20 con 0x7f5804105540 2024-03-28T05:27:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.992+0000 7f5809ff1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58041108a0 con 0x7f58041007f0 2024-03-28T05:27:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.992+0000 7f5809ff1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5804110ba0 con 0x7f58041011b0 2024-03-28T05:27:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.993+0000 7f5803fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5804105540 0x7f580419b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.993+0000 7f5803fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5804105540 0x7f580419b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59322/0 (socket says 172.21.15.23:59322) 2024-03-28T05:27:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.993+0000 7f5803fff640 1 -- 172.21.15.23:0/2290876575 learned_addr learned my addr 172.21.15.23:0/2290876575 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:27:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.993+0000 7f5802ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58041011b0 0x7f5804197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:50.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.993+0000 7f58037fe640 1 --2- 172.21.15.23:0/2290876575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58041007f0 0x7f5804196dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.993+0000 7f5803fff640 1 -- 172.21.15.23:0/2290876575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58041011b0 msgr2=0x7f5804197300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.993+0000 7f5803fff640 1 --2- 172.21.15.23:0/2290876575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58041011b0 0x7f5804197300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.993+0000 7f5803fff640 1 -- 172.21.15.23:0/2290876575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58041007f0 msgr2=0x7f5804196dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.993+0000 7f5803fff640 1 --2- 172.21.15.23:0/2290876575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58041007f0 0x7f5804196dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.993+0000 7f5803fff640 1 -- 172.21.15.23:0/2290876575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580419bd60 con 0x7f5804105540 2024-03-28T05:27:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.993+0000 7f58037fe640 1 --2- 172.21.15.23:0/2290876575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58041007f0 0x7f5804196dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:27:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.993+0000 7f5802ffd640 1 --2- 172.21.15.23:0/2290876575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58041011b0 0x7f5804197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:27:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.993+0000 7f5803fff640 1 --2- 172.21.15.23:0/2290876575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5804105540 0x7f580419b630 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f57fc02c700 tx=0x7f57fc032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.993+0000 7f5800ff9640 1 -- 172.21.15.23:0/2290876575 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57fc042020 con 0x7f5804105540 2024-03-28T05:27:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.993+0000 7f5800ff9640 1 -- 172.21.15.23:0/2290876575 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57fc034640 con 0x7f5804105540 2024-03-28T05:27:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.993+0000 7f5809ff1640 1 -- 172.21.15.23:0/2290876575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f580419bff0 con 0x7f5804105540 2024-03-28T05:27:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.993+0000 7f5800ff9640 1 -- 172.21.15.23:0/2290876575 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57fc039040 con 0x7f5804105540 2024-03-28T05:27:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.993+0000 7f5809ff1640 1 -- 172.21.15.23:0/2290876575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58041a38e0 con 0x7f5804105540 2024-03-28T05:27:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.995+0000 7f5800ff9640 1 -- 172.21.15.23:0/2290876575 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f57fc045020 con 0x7f5804105540 2024-03-28T05:27:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.996+0000 7f5800ff9640 1 --2- 172.21.15.23:0/2290876575 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f57ec0a2a10 0x7f57ec0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.996+0000 7f580cbd8740 1 Processor -- start 2024-03-28T05:27:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.996+0000 7f58037fe640 1 --2- 172.21.15.23:0/2290876575 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f57ec0a2a10 0x7f57ec0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.996+0000 7f5800ff9640 1 -- 172.21.15.23:0/2290876575 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f57fc0f0d50 con 0x7f5804105540 2024-03-28T05:27:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.996+0000 7f580cbd8740 1 -- start start 2024-03-28T05:27:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.996+0000 7f580cbd8740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609961b0 0x562760996590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.996+0000 7f580cbd8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562760996ad0 0x5627609a1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.996+0000 7f580cbd8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627609a2010 0x5627609a4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:51.003 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.996+0000 7f580cbd8740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627608fab10 con 0x5627609961b0 2024-03-28T05:27:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.996+0000 7f580cbd8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627608a3900 con 0x5627609a2010 2024-03-28T05:27:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.996+0000 7f580cbd8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627608f6bb0 con 0x562760996ad0 2024-03-28T05:27:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.996+0000 7f58037fe640 1 --2- 172.21.15.23:0/2290876575 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f57ec0a2a10 0x7f57ec0a4ed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f57f4009b40 tx=0x7f57f4005e50 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.997+0000 7f58037fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609961b0 0x562760996590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.997+0000 7f58037fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609961b0 0x562760996590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59334/0 (socket says 172.21.15.23:59334) 2024-03-28T05:27:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.997+0000 7f58037fe640 1 -- 172.21.15.23:0/589809087 learned_addr learned my addr 172.21.15.23:0/589809087 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:27:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.997+0000 7f5802ffd640 1 --2- 172.21.15.23:0/589809087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562760996ad0 0x5627609a1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.997+0000 7f5803fff640 1 --2- 172.21.15.23:0/589809087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627609a2010 0x5627609a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.997+0000 7f58037fe640 1 -- 172.21.15.23:0/589809087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562760996ad0 msgr2=0x5627609a1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.997+0000 7f58037fe640 1 --2- 172.21.15.23:0/589809087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562760996ad0 0x5627609a1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.997+0000 7f58037fe640 1 -- 172.21.15.23:0/589809087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627609a2010 msgr2=0x5627609a4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.997+0000 7f58037fe640 1 --2- 172.21.15.23:0/589809087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627609a2010 0x5627609a4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.997+0000 7f58037fe640 1 -- 172.21.15.23:0/589809087 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56276060f210 con 0x5627609961b0 2024-03-28T05:27:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.997+0000 7f5803fff640 1 --2- 172.21.15.23:0/589809087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627609a2010 0x5627609a4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:27:51.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.997+0000 7f5802ffd640 1 --2- 172.21.15.23:0/589809087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562760996ad0 0x5627609a1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:27:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.997+0000 7f58037fe640 1 --2- 172.21.15.23:0/589809087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609961b0 0x562760996590 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f57f400efb0 tx=0x7f57f4011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4be2ecac0eb5c228 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.997+0000 7f57e9ffb640 1 -- 172.21.15.23:0/589809087 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f4004020 con 0x5627609961b0 2024-03-28T05:27:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.997+0000 7f57e9ffb640 1 -- 172.21.15.23:0/589809087 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57f4017030 con 0x5627609961b0 2024-03-28T05:27:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.997+0000 7f57e9ffb640 1 -- 172.21.15.23:0/589809087 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f401e640 con 0x5627609961b0 2024-03-28T05:27:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.998+0000 7f580cbd8740 1 -- 172.21.15.23:0/589809087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609961b0 msgr2=0x562760996590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.998+0000 7f580cbd8740 1 --2- 172.21.15.23:0/589809087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609961b0 0x562760996590 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f57f400efb0 tx=0x7f57f4011000 comp rx=0 tx=0).stop 2024-03-28T05:27:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.998+0000 7f580cbd8740 1 -- 172.21.15.23:0/589809087 shutdown_connections 2024-03-28T05:27:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.998+0000 7f580cbd8740 1 --2- 172.21.15.23:0/589809087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627609a2010 0x5627609a4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.998+0000 7f580cbd8740 1 --2- 172.21.15.23:0/589809087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562760996ad0 0x5627609a1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.998+0000 7f580cbd8740 1 --2- 172.21.15.23:0/589809087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609961b0 0x562760996590 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.998+0000 7f580cbd8740 1 -- 172.21.15.23:0/589809087 >> 172.21.15.23:0/589809087 conn(0x562760991030 msgr2=0x562760991410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.998+0000 7f580cbd8740 1 -- 172.21.15.23:0/589809087 shutdown_connections 2024-03-28T05:27:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.998+0000 7f580cbd8740 1 -- 172.21.15.23:0/589809087 wait complete. 2024-03-28T05:27:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.998+0000 7f580cbd8740 1 -- start start 2024-03-28T05:27:51.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f580cbd8740 1 Processor -- start 2024-03-28T05:27:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f580cbd8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627609961b0 0x56276099efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f580cbd8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562760996ad0 0x56276099f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f580cbd8740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609a2010 0x56276099fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f580cbd8740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627609a4be0 con 0x5627609a2010 2024-03-28T05:27:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f580cbd8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627608b6d50 con 0x562760996ad0 2024-03-28T05:27:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f580cbd8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627609a4d60 con 0x5627609961b0 2024-03-28T05:27:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f580cbd8740 10 client.0 _ll_register_callbacks cb 0x5627608b6720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:27:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f5803fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609a2010 0x56276099fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f58037fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627609961b0 0x56276099efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f5803fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609a2010 0x56276099fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59350/0 (socket says 172.21.15.23:59350) 2024-03-28T05:27:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f5803fff640 1 -- 172.21.15.23:0/1417640204 learned_addr learned my addr 172.21.15.23:0/1417640204 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:27:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f58037fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627609961b0 0x56276099efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:50266/0 (socket says 172.21.15.23:50266) 2024-03-28T05:27:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f5802ffd640 1 --2- 172.21.15.23:0/1417640204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562760996ad0 0x56276099f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:51.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f5803fff640 1 -- 172.21.15.23:0/1417640204 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627609961b0 msgr2=0x56276099efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f5803fff640 1 --2- 172.21.15.23:0/1417640204 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627609961b0 0x56276099efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f5803fff640 1 -- 172.21.15.23:0/1417640204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562760996ad0 msgr2=0x56276099f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f5803fff640 1 --2- 172.21.15.23:0/1417640204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562760996ad0 0x56276099f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f5803fff640 1 -- 172.21.15.23:0/1417640204 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5627606db620 con 0x5627609a2010 2024-03-28T05:27:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f58037fe640 1 --2- 172.21.15.23:0/1417640204 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627609961b0 0x56276099efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:27:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f5803fff640 1 --2- 172.21.15.23:0/1417640204 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609a2010 0x56276099fa50 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f57fc0b9b00 tx=0x7f57fc0bca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc78c7e4693717c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f57e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:27:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f57e2ffd640 1 -- 172.21.15.23:0/1417640204 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57fc0bd130 con 0x5627609a2010 2024-03-28T05:27:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:50.999+0000 7f57e2ffd640 1 -- 172.21.15.23:0/1417640204 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f57fc003970 con 0x5627609a2010 2024-03-28T05:27:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.000+0000 7f580cbd8740 1 -- 172.21.15.23:0/1417640204 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f57f4017030 con 0x5627609a2010 2024-03-28T05:27:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.000+0000 7f57e2ffd640 1 -- 172.21.15.23:0/1417640204 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57fc0c1280 con 0x5627609a2010 2024-03-28T05:27:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.000+0000 7f580cbd8740 10 client.15819 fetch_fsmap learned FSMap version 20 2024-03-28T05:27:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.000+0000 7f580cbd8740 1 -- 172.21.15.23:0/1417640204 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5627609b0dd0 con 0x5627609a2010 2024-03-28T05:27:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.000+0000 7f57e2ffd640 1 -- 172.21.15.23:0/1417640204 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57f4017030 con 0x5627609a2010 2024-03-28T05:27:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.000+0000 7f57e2ffd640 1 -- 172.21.15.23:0/1417640204 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f57fc0c5440 con 0x5627609a2010 2024-03-28T05:27:51.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.000+0000 7f580cbd8740 10 client.15819 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:27:51.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.000+0000 7f580cbd8740 10 client.15819 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:27:51.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.000+0000 7f580cbd8740 20 client.15819 populate_metadata read hostname 'smithi023' 2024-03-28T05:27:51.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.000+0000 7f580cbd8740 1 --2- 172.21.15.23:0/1417640204 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5627609b2230 0x5627609b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:51.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.000+0000 7f580cbd8740 4 client.15819 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:27:51.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.000+0000 7f580cbd8740 1 -- 172.21.15.23:0/1417640204 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562760908050 con 0x5627609b2230 2024-03-28T05:27:51.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.000+0000 7f58037fe640 1 --2- 172.21.15.23:0/1417640204 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5627609b2230 0x5627609b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:51.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.001+0000 7f58037fe640 1 --2- 172.21.15.23:0/1417640204 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5627609b2230 0x5627609b4650 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb86c8070d552ac0 server_cookie=7538e9ba820fbfe5 in_seq=0 out_seq=0 2024-03-28T05:27:51.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.001+0000 7f57e2ffd640 10 client.15819 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:27:51.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.002+0000 7f57e2ffd640 1 -- 172.21.15.23:0/1417640204 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562760908050 con 0x5627609b2230 2024-03-28T05:27:51.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.002+0000 7f57e2ffd640 10 client.15819 handle_command_reply: tid=0 2024-03-28T05:27:51.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.002+0000 7f580cbd8740 1 client.15819 shutdown 2024-03-28T05:27:51.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.002+0000 7f580cbd8740 1 -- 172.21.15.23:0/1417640204 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609a2010 msgr2=0x56276099fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:51.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.002+0000 7f580cbd8740 1 --2- 172.21.15.23:0/1417640204 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609a2010 0x56276099fa50 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f57fc0b9b00 tx=0x7f57fc0bca90 comp rx=0 tx=0).stop 2024-03-28T05:27:51.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.003+0000 7f580cbd8740 1 -- 172.21.15.23:0/1417640204 shutdown_connections 2024-03-28T05:27:51.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.003+0000 7f580cbd8740 1 --2- 172.21.15.23:0/1417640204 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609a2010 0x56276099fa50 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.009 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.003+0000 7f580cbd8740 1 --2- 172.21.15.23:0/1417640204 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5627609b2230 0x5627609b4650 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.009 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.003+0000 7f580cbd8740 1 --2- 172.21.15.23:0/1417640204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562760996ad0 0x56276099f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.009 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.003+0000 7f580cbd8740 1 --2- 172.21.15.23:0/1417640204 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627609961b0 0x56276099efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.009 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.003+0000 7f57e2ffd640 0 client.15819 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:27:51.009 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.003+0000 7f580cbd8740 1 -- 172.21.15.23:0/1417640204 >> 172.21.15.23:0/1417640204 conn(0x562760991030 msgr2=0x562760992d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:51.009 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.003+0000 7f580cbd8740 1 -- 172.21.15.23:0/1417640204 shutdown_connections 2024-03-28T05:27:51.009 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.003+0000 7f580cbd8740 1 -- 172.21.15.23:0/1417640204 wait complete. 2024-03-28T05:27:51.009 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.003+0000 7f580cbd8740 20 client.15819 trim_cache size 0 max 16384 2024-03-28T05:27:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.017+0000 7f580cbd8740 1 Processor -- start 2024-03-28T05:27:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.017+0000 7f580cbd8740 1 -- start start 2024-03-28T05:27:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.017+0000 7f580cbd8740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609961b0 0x5627609a1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.017+0000 7f580cbd8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562760996ad0 0x5627609ad4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.017+0000 7f580cbd8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627609a2010 0x5627609ad9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.017+0000 7f580cbd8740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58041aa6d0 con 0x5627609961b0 2024-03-28T05:27:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.017+0000 7f580cbd8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58041acc90 con 0x562760996ad0 2024-03-28T05:27:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.017+0000 7f580cbd8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58041aacd0 con 0x5627609a2010 2024-03-28T05:27:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.017+0000 7f58037fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609961b0 0x5627609a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.017+0000 7f58037fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609961b0 0x5627609a1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59366/0 (socket says 172.21.15.23:59366) 2024-03-28T05:27:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.017+0000 7f58037fe640 1 -- 172.21.15.23:0/1264528554 learned_addr learned my addr 172.21.15.23:0/1264528554 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:27:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.017+0000 7f5803fff640 1 --2- 172.21.15.23:0/1264528554 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627609a2010 0x5627609ad9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:51.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.017+0000 7f5802ffd640 1 --2- 172.21.15.23:0/1264528554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562760996ad0 0x5627609ad4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.017+0000 7f58037fe640 1 -- 172.21.15.23:0/1264528554 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627609a2010 msgr2=0x5627609ad9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.017+0000 7f58037fe640 1 --2- 172.21.15.23:0/1264528554 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627609a2010 0x5627609ad9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.017+0000 7f58037fe640 1 -- 172.21.15.23:0/1264528554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562760996ad0 msgr2=0x5627609ad4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.017+0000 7f58037fe640 1 --2- 172.21.15.23:0/1264528554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562760996ad0 0x5627609ad4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.017+0000 7f58037fe640 1 -- 172.21.15.23:0/1264528554 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562760990e60 con 0x5627609961b0 2024-03-28T05:27:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.017+0000 7f5803fff640 1 --2- 172.21.15.23:0/1264528554 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627609a2010 0x5627609ad9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:27:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.018+0000 7f58037fe640 1 --2- 172.21.15.23:0/1264528554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609961b0 0x5627609a1700 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f57f4003df0 tx=0x7f57f4003e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49840767cff56c6c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.018+0000 7f57e97fa640 1 -- 172.21.15.23:0/1264528554 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f40312c0 con 0x5627609961b0 2024-03-28T05:27:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.018+0000 7f57e97fa640 1 -- 172.21.15.23:0/1264528554 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57f4017070 con 0x5627609961b0 2024-03-28T05:27:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.018+0000 7f57e97fa640 1 -- 172.21.15.23:0/1264528554 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f4005580 con 0x5627609961b0 2024-03-28T05:27:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.018+0000 7f580cbd8740 1 -- 172.21.15.23:0/1264528554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609961b0 msgr2=0x5627609a1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.018+0000 7f580cbd8740 1 --2- 172.21.15.23:0/1264528554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609961b0 0x5627609a1700 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f57f4003df0 tx=0x7f57f4003e20 comp rx=0 tx=0).stop 2024-03-28T05:27:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.019+0000 7f580cbd8740 1 -- 172.21.15.23:0/1264528554 shutdown_connections 2024-03-28T05:27:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.019+0000 7f580cbd8740 1 --2- 172.21.15.23:0/1264528554 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5627609a2010 0x5627609ad9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.019+0000 7f580cbd8740 1 --2- 172.21.15.23:0/1264528554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562760996ad0 0x5627609ad4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.019+0000 7f580cbd8740 1 --2- 172.21.15.23:0/1264528554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609961b0 0x5627609a1700 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.019+0000 7f580cbd8740 1 -- 172.21.15.23:0/1264528554 >> 172.21.15.23:0/1264528554 conn(0x562760991030 msgr2=0x562760991410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.019+0000 7f580cbd8740 1 -- 172.21.15.23:0/1264528554 shutdown_connections 2024-03-28T05:27:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.019+0000 7f580cbd8740 1 -- 172.21.15.23:0/1264528554 wait complete. 2024-03-28T05:27:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.019+0000 7f580cbd8740 1 -- start start 2024-03-28T05:27:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.019+0000 7f580cbd8740 1 Processor -- start 2024-03-28T05:27:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.019+0000 7f580cbd8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627609961b0 0x5627609a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.019+0000 7f580cbd8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562760996ad0 0x5627609a5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.019+0000 7f580cbd8740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609a2010 0x5627609a6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.019+0000 7f580cbd8740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58041ab310 con 0x5627609a2010 2024-03-28T05:27:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.019+0000 7f580cbd8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58041ac630 con 0x5627609961b0 2024-03-28T05:27:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.019+0000 7f580cbd8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58041ab970 con 0x562760996ad0 2024-03-28T05:27:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.019+0000 7f580cbd8740 10 client.0 _ll_register_callbacks cb 0x7f57fc0bcf00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:27:51.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.019+0000 7f5803fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609a2010 0x5627609a6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.020+0000 7f5802ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562760996ad0 0x5627609a5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.020+0000 7f5803fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609a2010 0x5627609a6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59368/0 (socket says 172.21.15.23:59368) 2024-03-28T05:27:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.020+0000 7f5803fff640 1 -- 172.21.15.23:0/291863735 learned_addr learned my addr 172.21.15.23:0/291863735 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:27:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.020+0000 7f58037fe640 1 --2- 172.21.15.23:0/291863735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627609961b0 0x5627609a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.020+0000 7f5803fff640 1 -- 172.21.15.23:0/291863735 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562760996ad0 msgr2=0x5627609a5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.020+0000 7f5803fff640 1 --2- 172.21.15.23:0/291863735 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562760996ad0 0x5627609a5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.020+0000 7f5803fff640 1 -- 172.21.15.23:0/291863735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627609961b0 msgr2=0x5627609a55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:27:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.020+0000 7f5803fff640 1 --2- 172.21.15.23:0/291863735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627609961b0 0x5627609a55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.020+0000 7f5803fff640 1 -- 172.21.15.23:0/291863735 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5627609a4400 con 0x5627609a2010 2024-03-28T05:27:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.020+0000 7f5802ffd640 1 --2- 172.21.15.23:0/291863735 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562760996ad0 0x5627609a5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:27:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.020+0000 7f5803fff640 1 --2- 172.21.15.23:0/291863735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609a2010 0x5627609a6030 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f57fc009dc0 tx=0x7f57fc0bcbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bed764486296e767 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:27:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.020+0000 7f57e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:27:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.020+0000 7f57e2ffd640 1 -- 172.21.15.23:0/291863735 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57fc0b93d0 con 0x5627609a2010 2024-03-28T05:27:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.020+0000 7f57e2ffd640 1 -- 172.21.15.23:0/291863735 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f57fc003970 con 0x5627609a2010 2024-03-28T05:27:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.020+0000 7f580cbd8740 1 -- 172.21.15.23:0/291863735 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f57f4017070 con 0x5627609a2010 2024-03-28T05:27:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.020+0000 7f57e2ffd640 1 -- 172.21.15.23:0/291863735 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57fc0bd130 con 0x5627609a2010 2024-03-28T05:27:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.020+0000 7f580cbd8740 10 client.15825 fetch_fsmap learned FSMap version 20 2024-03-28T05:27:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.020+0000 7f580cbd8740 1 -- 172.21.15.23:0/291863735 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x562760a0dcb0 con 0x5627609a2010 2024-03-28T05:27:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.020+0000 7f57e2ffd640 1 -- 172.21.15.23:0/291863735 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57f4017070 con 0x5627609a2010 2024-03-28T05:27:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.021+0000 7f57e2ffd640 1 -- 172.21.15.23:0/291863735 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f57fc0a60d0 con 0x5627609a2010 2024-03-28T05:27:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.021+0000 7f580cbd8740 10 client.15825 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:27:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.021+0000 7f580cbd8740 10 client.15825 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:27:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.021+0000 7f580cbd8740 20 client.15825 populate_metadata read hostname 'smithi023' 2024-03-28T05:27:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.021+0000 7f580cbd8740 1 --2- 172.21.15.23:0/291863735 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x562760a0e210 0x562760a10630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:27:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.021+0000 7f580cbd8740 4 client.15825 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:27:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.021+0000 7f580cbd8740 1 -- 172.21.15.23:0/291863735 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5627609023a0 con 0x562760a0e210 2024-03-28T05:27:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.021+0000 7f58037fe640 1 --2- 172.21.15.23:0/291863735 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x562760a0e210 0x562760a10630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:27:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.021+0000 7f57e2ffd640 10 client.15825 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:27:51.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.021+0000 7f58037fe640 1 --2- 172.21.15.23:0/291863735 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x562760a0e210 0x562760a10630 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=640772e039f18399 server_cookie=28f7022bbd839064 in_seq=0 out_seq=0 2024-03-28T05:27:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.411+0000 7f57e2ffd640 1 -- 172.21.15.23:0/291863735 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5627609023a0 con 0x562760a0e210 2024-03-28T05:27:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.411+0000 7f57e2ffd640 10 client.15825 handle_command_reply: tid=0 2024-03-28T05:27:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.411+0000 7f580cbd8740 1 client.15825 shutdown 2024-03-28T05:27:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.411+0000 7f580cbd8740 1 -- 172.21.15.23:0/291863735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609a2010 msgr2=0x5627609a6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.411+0000 7f580cbd8740 1 --2- 172.21.15.23:0/291863735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609a2010 0x5627609a6030 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f57fc009dc0 tx=0x7f57fc0bcbc0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.411+0000 7f580cbd8740 1 -- 172.21.15.23:0/291863735 shutdown_connections 2024-03-28T05:27:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.411+0000 7f580cbd8740 1 --2- 172.21.15.23:0/291863735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5627609a2010 0x5627609a6030 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.412 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.411+0000 7f580cbd8740 1 --2- 172.21.15.23:0/291863735 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562760996ad0 0x5627609a5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.413 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.411+0000 7f580cbd8740 1 --2- 172.21.15.23:0/291863735 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x562760a0e210 0x562760a10630 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.413 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.411+0000 7f580cbd8740 1 --2- 172.21.15.23:0/291863735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5627609961b0 0x5627609a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.413 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.411+0000 7f57e2ffd640 0 client.15825 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:27:51.413 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.411+0000 7f580cbd8740 1 -- 172.21.15.23:0/291863735 >> 172.21.15.23:0/291863735 conn(0x562760991030 msgr2=0x56276099ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:51.413 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.411+0000 7f580cbd8740 1 -- 172.21.15.23:0/291863735 shutdown_connections 2024-03-28T05:27:51.413 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.412+0000 7f580cbd8740 1 -- 172.21.15.23:0/291863735 wait complete. 2024-03-28T05:27:51.413 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.412+0000 7f580cbd8740 20 client.15825 trim_cache size 0 max 16384 2024-03-28T05:27:51.413 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:27:51.413 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:27:51.413 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:27:51.413 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:27:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.412+0000 7f5809ff1640 1 -- 172.21.15.23:0/2290876575 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f57ec0a2a10 msgr2=0x7f57ec0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.412+0000 7f5809ff1640 1 --2- 172.21.15.23:0/2290876575 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f57ec0a2a10 0x7f57ec0a4ed0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f57f4009b40 tx=0x7f57f4005e50 comp rx=0 tx=0).stop 2024-03-28T05:27:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.412+0000 7f5809ff1640 1 -- 172.21.15.23:0/2290876575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5804105540 msgr2=0x7f580419b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:27:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.412+0000 7f5809ff1640 1 --2- 172.21.15.23:0/2290876575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5804105540 0x7f580419b630 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f57fc02c700 tx=0x7f57fc032f20 comp rx=0 tx=0).stop 2024-03-28T05:27:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.412+0000 7f5809ff1640 1 -- 172.21.15.23:0/2290876575 shutdown_connections 2024-03-28T05:27:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.412+0000 7f5809ff1640 1 --2- 172.21.15.23:0/2290876575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5804105540 0x7f580419b630 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.412+0000 7f5809ff1640 1 --2- 172.21.15.23:0/2290876575 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f58041011b0 0x7f5804197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.412+0000 7f5809ff1640 1 --2- 172.21.15.23:0/2290876575 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f57ec0a2a10 0x7f57ec0a4ed0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.412+0000 7f5809ff1640 1 --2- 172.21.15.23:0/2290876575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58041007f0 0x7f5804196dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:27:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.412+0000 7f5809ff1640 1 -- 172.21.15.23:0/2290876575 >> 172.21.15.23:0/2290876575 conn(0x7f58040773f0 msgr2=0x7f58040fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:27:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.413+0000 7f5809ff1640 1 -- 172.21.15.23:0/2290876575 shutdown_connections 2024-03-28T05:27:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:27:51.413+0000 7f5809ff1640 1 -- 172.21.15.23:0/2290876575 wait complete. 2024-03-28T05:27:52.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:51 smithi113 ceph-mon[37164]: pgmap v131: 129 pgs: 129 active+clean; 129 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 1.1 MiB/s wr, 280 op/s 2024-03-28T05:27:52.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:51 smithi023 ceph-mon[31954]: pgmap v131: 129 pgs: 129 active+clean; 129 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 1.1 MiB/s wr, 280 op/s 2024-03-28T05:27:52.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:51 smithi098 ceph-mon[36087]: pgmap v131: 129 pgs: 129 active+clean; 129 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 1.1 MiB/s wr, 280 op/s 2024-03-28T05:27:54.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:53 smithi113 ceph-mon[37164]: pgmap v132: 129 pgs: 129 active+clean; 141 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.9 MiB/s wr, 309 op/s 2024-03-28T05:27:54.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:53 smithi023 ceph-mon[31954]: pgmap v132: 129 pgs: 129 active+clean; 141 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.9 MiB/s wr, 309 op/s 2024-03-28T05:27:54.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:53 smithi098 ceph-mon[36087]: pgmap v132: 129 pgs: 129 active+clean; 141 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.9 MiB/s wr, 309 op/s 2024-03-28T05:27:56.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:55 smithi113 ceph-mon[37164]: pgmap v133: 129 pgs: 129 active+clean; 141 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 1.9 MiB/s wr, 162 op/s 2024-03-28T05:27:56.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:55 smithi023 ceph-mon[31954]: pgmap v133: 129 pgs: 129 active+clean; 141 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 1.9 MiB/s wr, 162 op/s 2024-03-28T05:27:56.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:55 smithi098 ceph-mon[36087]: pgmap v133: 129 pgs: 129 active+clean; 141 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 1.9 MiB/s wr, 162 op/s 2024-03-28T05:27:56.400 INFO:tasks.workunit.client.0.smithi023.stdout: 42 405194 166 277745 105 193574 382 2024-03-28T05:27:58.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:57 smithi113 ceph-mon[37164]: pgmap v134: 129 pgs: 129 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 2.3 MiB/s wr, 434 op/s 2024-03-28T05:27:58.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:57 smithi023 ceph-mon[31954]: pgmap v134: 129 pgs: 129 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 2.3 MiB/s wr, 434 op/s 2024-03-28T05:27:58.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:57 smithi098 ceph-mon[36087]: pgmap v134: 129 pgs: 129 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 2.3 MiB/s wr, 434 op/s 2024-03-28T05:28:00.245 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:27:59 smithi113 ceph-mon[37164]: pgmap v135: 129 pgs: 129 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 2.0 MiB/s wr, 388 op/s 2024-03-28T05:28:00.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:27:59 smithi023 ceph-mon[31954]: pgmap v135: 129 pgs: 129 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 2.0 MiB/s wr, 388 op/s 2024-03-28T05:28:00.368 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:27:59 smithi098 ceph-mon[36087]: pgmap v135: 129 pgs: 129 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 2.0 MiB/s wr, 388 op/s 2024-03-28T05:28:02.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:01 smithi113 ceph-mon[37164]: pgmap v136: 129 pgs: 129 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 2.0 MiB/s wr, 403 op/s 2024-03-28T05:28:02.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:01 smithi023 ceph-mon[31954]: pgmap v136: 129 pgs: 129 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 2.0 MiB/s wr, 403 op/s 2024-03-28T05:28:02.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:01 smithi098 ceph-mon[36087]: pgmap v136: 129 pgs: 129 active+clean; 136 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 2.0 MiB/s wr, 403 op/s 2024-03-28T05:28:04.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:03 smithi113 ceph-mon[37164]: pgmap v137: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 1.8 MiB/s wr, 441 op/s 2024-03-28T05:28:04.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:03 smithi023 ceph-mon[31954]: pgmap v137: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 1.8 MiB/s wr, 441 op/s 2024-03-28T05:28:04.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:03 smithi098 ceph-mon[36087]: pgmap v137: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 1.8 MiB/s wr, 441 op/s 2024-03-28T05:28:04.515 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.513+0000 7f15c56e8640 1 -- 172.21.15.23:0/3924346642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15c0101650 msgr2=0x7f15c0101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:04.515 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.513+0000 7f15c56e8640 1 --2- 172.21.15.23:0/3924346642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15c0101650 0x7f15c0101ad0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009930 tx=0x7f15b402bc30 comp rx=0 tx=0).stop 2024-03-28T05:28:04.515 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.513+0000 7f15c56e8640 1 -- 172.21.15.23:0/3924346642 shutdown_connections 2024-03-28T05:28:04.515 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.513+0000 7f15c56e8640 1 --2- 172.21.15.23:0/3924346642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15c0102010 0x7f15c010e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.515 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.513+0000 7f15c56e8640 1 --2- 172.21.15.23:0/3924346642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15c0101650 0x7f15c0101ad0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.515 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.513+0000 7f15c56e8640 1 --2- 172.21.15.23:0/3924346642 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15c0107660 0x7f15c0107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.515 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.513+0000 7f15c56e8640 1 -- 172.21.15.23:0/3924346642 >> 172.21.15.23:0/3924346642 conn(0x7f15c00fd160 msgr2=0x7f15c00ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:04.515 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.513+0000 7f15c56e8640 1 -- 172.21.15.23:0/3924346642 shutdown_connections 2024-03-28T05:28:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.513+0000 7f15c56e8640 1 -- 172.21.15.23:0/3924346642 wait complete. 2024-03-28T05:28:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.514+0000 7f15c56e8640 1 Processor -- start 2024-03-28T05:28:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.514+0000 7f15c56e8640 1 -- start start 2024-03-28T05:28:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.514+0000 7f15c56e8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15c0101650 0x7f15c0197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.514+0000 7f15c56e8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15c0102010 0x7f15c01983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.514+0000 7f15c56e8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15c0107660 0x7f15c019c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.514+0000 7f15c56e8640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c0111a30 con 0x7f15c0102010 2024-03-28T05:28:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.514+0000 7f15c56e8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c01118b0 con 0x7f15c0101650 2024-03-28T05:28:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.514+0000 7f15c56e8640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c0111bb0 con 0x7f15c0107660 2024-03-28T05:28:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.514+0000 7f15be7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15c0102010 0x7f15c01983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.514+0000 7f15bf7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15c0107660 0x7f15c019c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.515+0000 7f15be7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15c0102010 0x7f15c01983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48764/0 (socket says 172.21.15.23:48764) 2024-03-28T05:28:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.515+0000 7f15bf7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15c0107660 0x7f15c019c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:36614/0 (socket says 172.21.15.23:36614) 2024-03-28T05:28:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.515+0000 7f15be7fc640 1 -- 172.21.15.23:0/3889061590 learned_addr learned my addr 172.21.15.23:0/3889061590 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.515+0000 7f15beffd640 1 --2- 172.21.15.23:0/3889061590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15c0101650 0x7f15c0197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:04.517 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.515+0000 7f15be7fc640 1 -- 172.21.15.23:0/3889061590 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15c0107660 msgr2=0x7f15c019c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:04.517 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.515+0000 7f15be7fc640 1 --2- 172.21.15.23:0/3889061590 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15c0107660 0x7f15c019c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.517 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.515+0000 7f15be7fc640 1 -- 172.21.15.23:0/3889061590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15c0101650 msgr2=0x7f15c0197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:04.517 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.515+0000 7f15be7fc640 1 --2- 172.21.15.23:0/3889061590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15c0101650 0x7f15c0197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.517 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.515+0000 7f15be7fc640 1 -- 172.21.15.23:0/3889061590 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c019ce80 con 0x7f15c0102010 2024-03-28T05:28:04.517 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.515+0000 7f15bf7fe640 1 --2- 172.21.15.23:0/3889061590 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15c0107660 0x7f15c019c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:28:04.517 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.515+0000 7f15beffd640 1 --2- 172.21.15.23:0/3889061590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15c0101650 0x7f15c0197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:28:04.517 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.515+0000 7f15be7fc640 1 --2- 172.21.15.23:0/3889061590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15c0102010 0x7f15c01983d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009230 tx=0x7f15b402cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:04.517 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.515+0000 7f15a7fff640 1 -- 172.21.15.23:0/3889061590 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b4041070 con 0x7f15c0102010 2024-03-28T05:28:04.517 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.515+0000 7f15a7fff640 1 -- 172.21.15.23:0/3889061590 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15b4004450 con 0x7f15c0102010 2024-03-28T05:28:04.517 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.515+0000 7f15c56e8640 1 -- 172.21.15.23:0/3889061590 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c019d110 con 0x7f15c0102010 2024-03-28T05:28:04.517 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.515+0000 7f15a7fff640 1 -- 172.21.15.23:0/3889061590 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b4003e50 con 0x7f15c0102010 2024-03-28T05:28:04.517 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.515+0000 7f15c56e8640 1 -- 172.21.15.23:0/3889061590 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c01a4a00 con 0x7f15c0102010 2024-03-28T05:28:04.520 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.518+0000 7f15a7fff640 1 -- 172.21.15.23:0/3889061590 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f15b4044070 con 0x7f15c0102010 2024-03-28T05:28:04.520 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.518+0000 7f15c83aa740 1 Processor -- start 2024-03-28T05:28:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.518+0000 7f15c83aa740 1 -- start start 2024-03-28T05:28:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.519+0000 7f15c83aa740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56cc710 0x5598d56ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.519+0000 7f15c83aa740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56cd030 0x5598d56d58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.519+0000 7f15c83aa740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56d5e00 0x5598d56d81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.519+0000 7f15c83aa740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598d5630b10 con 0x5598d56cd030 2024-03-28T05:28:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.519+0000 7f15c83aa740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598d55d9900 con 0x5598d56cc710 2024-03-28T05:28:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.519+0000 7f15c83aa740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598d562cbb0 con 0x5598d56d5e00 2024-03-28T05:28:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.519+0000 7f15be7fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56d5e00 0x5598d56d81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.519+0000 7f15beffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56cd030 0x5598d56d58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.519+0000 7f15beffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56cd030 0x5598d56d58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48768/0 (socket says 172.21.15.23:48768) 2024-03-28T05:28:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.519+0000 7f15be7fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56d5e00 0x5598d56d81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:36616/0 (socket says 172.21.15.23:36616) 2024-03-28T05:28:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.519+0000 7f15beffd640 1 -- 172.21.15.23:0/280799881 learned_addr learned my addr 172.21.15.23:0/280799881 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.519+0000 7f15a7fff640 1 --2- 172.21.15.23:0/3889061590 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f15a80a2bb0 0x7f15a80a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.519+0000 7f15bf7fe640 1 --2- 172.21.15.23:0/280799881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56cc710 0x5598d56ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.519+0000 7f15be7fc640 1 -- 172.21.15.23:0/280799881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56cc710 msgr2=0x5598d56ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.519+0000 7f15be7fc640 1 --2- 172.21.15.23:0/280799881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56cc710 0x5598d56ccaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.519+0000 7f15be7fc640 1 -- 172.21.15.23:0/280799881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56cd030 msgr2=0x5598d56d58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.519+0000 7f15be7fc640 1 --2- 172.21.15.23:0/280799881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56cd030 0x5598d56d58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.519+0000 7f15be7fc640 1 -- 172.21.15.23:0/280799881 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5598d5345210 con 0x5598d56d5e00 2024-03-28T05:28:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.519+0000 7f15bf7fe640 1 --2- 172.21.15.23:0/280799881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56cc710 0x5598d56ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:28:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.519+0000 7f15bf7fe640 1 --2- 172.21.15.23:0/3889061590 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f15a80a2bb0 0x7f15a80a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.520+0000 7f15a7fff640 1 -- 172.21.15.23:0/3889061590 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f15b40f0ac0 con 0x7f15c0102010 2024-03-28T05:28:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.520+0000 7f15be7fc640 1 --2- 172.21.15.23:0/280799881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56d5e00 0x5598d56d81f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f15b40edf90 tx=0x7f15b404b770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bab15d02b9e6e37 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.520+0000 7f15a57fa640 1 -- 172.21.15.23:0/280799881 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b40bc9e0 con 0x5598d56d5e00 2024-03-28T05:28:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.520+0000 7f15a57fa640 1 -- 172.21.15.23:0/280799881 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15b402cd20 con 0x5598d56d5e00 2024-03-28T05:28:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.520+0000 7f15a57fa640 1 -- 172.21.15.23:0/280799881 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b40bc800 con 0x5598d56d5e00 2024-03-28T05:28:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.521+0000 7f15c83aa740 1 -- 172.21.15.23:0/280799881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56d5e00 msgr2=0x5598d56d81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.521+0000 7f15bf7fe640 1 --2- 172.21.15.23:0/3889061590 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f15a80a2bb0 0x7f15a80a5070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f15b80099e0 tx=0x7f15b8004a40 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.521+0000 7f15c83aa740 1 --2- 172.21.15.23:0/280799881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56d5e00 0x5598d56d81f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f15b40edf90 tx=0x7f15b404b770 comp rx=0 tx=0).stop 2024-03-28T05:28:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.521+0000 7f15c83aa740 1 -- 172.21.15.23:0/280799881 shutdown_connections 2024-03-28T05:28:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.521+0000 7f15c83aa740 1 --2- 172.21.15.23:0/280799881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56d5e00 0x5598d56d81f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.521+0000 7f15c83aa740 1 --2- 172.21.15.23:0/280799881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56cd030 0x5598d56d58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.521+0000 7f15c83aa740 1 --2- 172.21.15.23:0/280799881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56cc710 0x5598d56ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.521+0000 7f15c83aa740 1 -- 172.21.15.23:0/280799881 >> 172.21.15.23:0/280799881 conn(0x5598d56c7030 msgr2=0x5598d56c7410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.521+0000 7f15c83aa740 1 -- 172.21.15.23:0/280799881 shutdown_connections 2024-03-28T05:28:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.521+0000 7f15c83aa740 1 -- 172.21.15.23:0/280799881 wait complete. 2024-03-28T05:28:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.521+0000 7f15c83aa740 1 -- start start 2024-03-28T05:28:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.521+0000 7f15c83aa740 1 Processor -- start 2024-03-28T05:28:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.522+0000 7f15c83aa740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56cc710 0x5598d56d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.522+0000 7f15c83aa740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56cd030 0x5598d56d3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.522+0000 7f15c83aa740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56d5e00 0x5598d56d3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.522+0000 7f15c83aa740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598d562cbb0 con 0x5598d56cc710 2024-03-28T05:28:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.522+0000 7f15c83aa740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598d55ecd50 con 0x5598d56d5e00 2024-03-28T05:28:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.522+0000 7f15c83aa740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598d56d87f0 con 0x5598d56cd030 2024-03-28T05:28:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.522+0000 7f15c83aa740 10 client.0 _ll_register_callbacks cb 0x5598d55ec720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:28:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.522+0000 7f15beffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56cc710 0x5598d56d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.522+0000 7f15beffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56cc710 0x5598d56d2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48782/0 (socket says 172.21.15.23:48782) 2024-03-28T05:28:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.522+0000 7f15be7fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56cd030 0x5598d56d3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.522+0000 7f15beffd640 1 -- 172.21.15.23:0/3303610452 learned_addr learned my addr 172.21.15.23:0/3303610452 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.522+0000 7f15bf7fe640 1 --2- 172.21.15.23:0/3303610452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56d5e00 0x5598d56d3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.522+0000 7f15be7fc640 1 -- 172.21.15.23:0/3303610452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56d5e00 msgr2=0x5598d56d3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.522+0000 7f15be7fc640 1 --2- 172.21.15.23:0/3303610452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56d5e00 0x5598d56d3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.522+0000 7f15be7fc640 1 -- 172.21.15.23:0/3303610452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56cc710 msgr2=0x5598d56d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.522+0000 7f15be7fc640 1 --2- 172.21.15.23:0/3303610452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56cc710 0x5598d56d2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.522+0000 7f15be7fc640 1 -- 172.21.15.23:0/3303610452 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5598d5411620 con 0x5598d56cd030 2024-03-28T05:28:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.523+0000 7f15be7fc640 1 --2- 172.21.15.23:0/3303610452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56cd030 0x5598d56d3350 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x5598d56c81c0 tx=0x7f15b4009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36b2c8b4cc635eba server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.523+0000 7f15a2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-28T05:28:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.523+0000 7f15a2ff5640 1 -- 172.21.15.23:0/3303610452 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b40bc800 con 0x5598d56cd030 2024-03-28T05:28:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.523+0000 7f15a2ff5640 1 -- 172.21.15.23:0/3303610452 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f15b40b8d30 con 0x5598d56cd030 2024-03-28T05:28:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.523+0000 7f15c83aa740 1 -- 172.21.15.23:0/3303610452 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f15b402cd20 con 0x5598d56cd030 2024-03-28T05:28:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.523+0000 7f15a2ff5640 1 -- 172.21.15.23:0/3303610452 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15b40bc9e0 con 0x5598d56cd030 2024-03-28T05:28:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.523+0000 7f15c83aa740 10 client.25669 fetch_fsmap learned FSMap version 20 2024-03-28T05:28:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.523+0000 7f15c83aa740 1 -- 172.21.15.23:0/3303610452 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5598d56e4820 con 0x5598d56cd030 2024-03-28T05:28:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.523+0000 7f15a2ff5640 1 -- 172.21.15.23:0/3303610452 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15b402cd20 con 0x5598d56cd030 2024-03-28T05:28:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.523+0000 7f15a2ff5640 1 -- 172.21.15.23:0/3303610452 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f15b40b9510 con 0x5598d56cd030 2024-03-28T05:28:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.523+0000 7f15c83aa740 10 client.25669 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:28:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.523+0000 7f15c83aa740 10 client.25669 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:28:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.523+0000 7f15c83aa740 20 client.25669 populate_metadata read hostname 'smithi023' 2024-03-28T05:28:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.523+0000 7f15c83aa740 1 --2- 172.21.15.23:0/3303610452 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5598d56e5b70 0x5598d56e7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.523+0000 7f15c83aa740 4 client.25669 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:28:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.523+0000 7f15c83aa740 1 -- 172.21.15.23:0/3303610452 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5598d563e050 con 0x5598d56e5b70 2024-03-28T05:28:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.524+0000 7f15beffd640 1 --2- 172.21.15.23:0/3303610452 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5598d56e5b70 0x5598d56e7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.524+0000 7f15a2ff5640 10 client.25669 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:28:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.524+0000 7f15beffd640 1 --2- 172.21.15.23:0/3303610452 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5598d56e5b70 0x5598d56e7f60 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46bc3a27c42c5512 server_cookie=8b9a47cc4d587fb5 in_seq=0 out_seq=0 2024-03-28T05:28:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.526+0000 7f15a2ff5640 1 -- 172.21.15.23:0/3303610452 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5598d563e050 con 0x5598d56e5b70 2024-03-28T05:28:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.526+0000 7f15a2ff5640 10 client.25669 handle_command_reply: tid=0 2024-03-28T05:28:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.526+0000 7f15c83aa740 1 client.25669 shutdown 2024-03-28T05:28:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.526+0000 7f15c83aa740 1 -- 172.21.15.23:0/3303610452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56cd030 msgr2=0x5598d56d3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.526+0000 7f15c83aa740 1 --2- 172.21.15.23:0/3303610452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56cd030 0x5598d56d3350 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x5598d56c81c0 tx=0x7f15b4009260 comp rx=0 tx=0).stop 2024-03-28T05:28:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.526+0000 7f15c83aa740 1 -- 172.21.15.23:0/3303610452 shutdown_connections 2024-03-28T05:28:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.526+0000 7f15c83aa740 1 --2- 172.21.15.23:0/3303610452 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5598d56e5b70 0x5598d56e7f60 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.526+0000 7f15c83aa740 1 --2- 172.21.15.23:0/3303610452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56d5e00 0x5598d56d3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.526+0000 7f15c83aa740 1 --2- 172.21.15.23:0/3303610452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56cd030 0x5598d56d3350 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.526+0000 7f15a2ff5640 0 client.25669 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:28:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.526+0000 7f15c83aa740 1 --2- 172.21.15.23:0/3303610452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56cc710 0x5598d56d2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.526+0000 7f15c83aa740 1 -- 172.21.15.23:0/3303610452 >> 172.21.15.23:0/3303610452 conn(0x5598d56c7030 msgr2=0x5598d56c8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.526+0000 7f15c83aa740 1 -- 172.21.15.23:0/3303610452 shutdown_connections 2024-03-28T05:28:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.526+0000 7f15c83aa740 1 -- 172.21.15.23:0/3303610452 wait complete. 2024-03-28T05:28:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.527+0000 7f15c83aa740 20 client.25669 trim_cache size 0 max 16384 2024-03-28T05:28:04.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.541+0000 7f15c83aa740 1 Processor -- start 2024-03-28T05:28:04.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.541+0000 7f15c83aa740 1 -- start start 2024-03-28T05:28:04.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.542+0000 7f15c83aa740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56cc710 0x5598d56d5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:04.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.542+0000 7f15c83aa740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56cd030 0x5598d56cd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:04.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.542+0000 7f15c83aa740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56d5e00 0x5598d56e0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:04.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.542+0000 7f15c83aa740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c01aca20 con 0x5598d56d5e00 2024-03-28T05:28:04.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.542+0000 7f15c83aa740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c01aefe0 con 0x5598d56cc710 2024-03-28T05:28:04.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.542+0000 7f15c83aa740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c01ad020 con 0x5598d56cd030 2024-03-28T05:28:04.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.542+0000 7f15beffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56cc710 0x5598d56d5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:04.546 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.542+0000 7f15bf7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56d5e00 0x5598d56e0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.542+0000 7f15be7fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56cd030 0x5598d56cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.542+0000 7f15bf7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56d5e00 0x5598d56e0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48784/0 (socket says 172.21.15.23:48784) 2024-03-28T05:28:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.542+0000 7f15be7fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56cd030 0x5598d56cd590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:36634/0 (socket says 172.21.15.23:36634) 2024-03-28T05:28:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.542+0000 7f15beffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56cc710 0x5598d56d5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.23:38036/0 (socket says 172.21.15.23:38036) 2024-03-28T05:28:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.542+0000 7f15bf7fe640 1 -- 172.21.15.23:0/1982289512 learned_addr learned my addr 172.21.15.23:0/1982289512 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.542+0000 7f15bf7fe640 1 -- 172.21.15.23:0/1982289512 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56cd030 msgr2=0x5598d56cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.542+0000 7f15bf7fe640 1 --2- 172.21.15.23:0/1982289512 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56cd030 0x5598d56cd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.542+0000 7f15bf7fe640 1 -- 172.21.15.23:0/1982289512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56cc710 msgr2=0x5598d56d5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.542+0000 7f15bf7fe640 1 --2- 172.21.15.23:0/1982289512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56cc710 0x5598d56d5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.542+0000 7f15bf7fe640 1 -- 172.21.15.23:0/1982289512 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5598d56d81f0 con 0x5598d56d5e00 2024-03-28T05:28:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.542+0000 7f15beffd640 1 --2- 172.21.15.23:0/1982289512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56cc710 0x5598d56d5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:28:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.542+0000 7f15bf7fe640 1 --2- 172.21.15.23:0/1982289512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56d5e00 0x5598d56e0f00 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f15b8012920 tx=0x7f15b8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ae1ff0cead9cb2c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.543+0000 7f15a4ff9640 1 -- 172.21.15.23:0/1982289512 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b801f8b0 con 0x5598d56d5e00 2024-03-28T05:28:04.547 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.543+0000 7f15a4ff9640 1 -- 172.21.15.23:0/1982289512 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15b8010460 con 0x5598d56d5e00 2024-03-28T05:28:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.543+0000 7f15a4ff9640 1 -- 172.21.15.23:0/1982289512 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b801e670 con 0x5598d56d5e00 2024-03-28T05:28:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.543+0000 7f15c83aa740 1 -- 172.21.15.23:0/1982289512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56d5e00 msgr2=0x5598d56e0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.543+0000 7f15c83aa740 1 --2- 172.21.15.23:0/1982289512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56d5e00 0x5598d56e0f00 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f15b8012920 tx=0x7f15b8015000 comp rx=0 tx=0).stop 2024-03-28T05:28:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.543+0000 7f15c83aa740 1 -- 172.21.15.23:0/1982289512 shutdown_connections 2024-03-28T05:28:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.543+0000 7f15c83aa740 1 --2- 172.21.15.23:0/1982289512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56d5e00 0x5598d56e0f00 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.543+0000 7f15c83aa740 1 --2- 172.21.15.23:0/1982289512 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56cd030 0x5598d56cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.543+0000 7f15c83aa740 1 --2- 172.21.15.23:0/1982289512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56cc710 0x5598d56d5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.543+0000 7f15c83aa740 1 -- 172.21.15.23:0/1982289512 >> 172.21.15.23:0/1982289512 conn(0x5598d56c7030 msgr2=0x5598d56c7410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.543+0000 7f15c83aa740 1 -- 172.21.15.23:0/1982289512 shutdown_connections 2024-03-28T05:28:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.543+0000 7f15c83aa740 1 -- 172.21.15.23:0/1982289512 wait complete. 2024-03-28T05:28:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.544+0000 7f15c83aa740 1 -- start start 2024-03-28T05:28:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.544+0000 7f15c83aa740 1 Processor -- start 2024-03-28T05:28:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.544+0000 7f15c83aa740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56cc710 0x5598d56cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.544+0000 7f15c83aa740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56cd030 0x5598d56cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:04.548 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.544+0000 7f15c83aa740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56d5e00 0x5598d5740350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:04.549 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.544+0000 7f15c83aa740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c01ad660 con 0x5598d56cc710 2024-03-28T05:28:04.549 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.544+0000 7f15c83aa740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c01ae980 con 0x5598d56d5e00 2024-03-28T05:28:04.549 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.544+0000 7f15c83aa740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c01adcc0 con 0x5598d56cd030 2024-03-28T05:28:04.549 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.544+0000 7f15c83aa740 10 client.0 _ll_register_callbacks cb 0x7f15b40b91d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:28:04.549 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.544+0000 7f15beffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56cc710 0x5598d56cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:04.549 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.544+0000 7f15beffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56cc710 0x5598d56cf780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48794/0 (socket says 172.21.15.23:48794) 2024-03-28T05:28:04.549 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.544+0000 7f15beffd640 1 -- 172.21.15.23:0/1905643795 learned_addr learned my addr 172.21.15.23:0/1905643795 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:04.549 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.544+0000 7f15be7fc640 1 --2- 172.21.15.23:0/1905643795 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56cd030 0x5598d56cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:04.549 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.544+0000 7f15bf7fe640 1 --2- 172.21.15.23:0/1905643795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56d5e00 0x5598d5740350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:04.549 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.544+0000 7f15beffd640 1 -- 172.21.15.23:0/1905643795 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56cd030 msgr2=0x5598d56cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:04.549 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.544+0000 7f15beffd640 1 --2- 172.21.15.23:0/1905643795 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56cd030 0x5598d56cfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.549 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.544+0000 7f15beffd640 1 -- 172.21.15.23:0/1905643795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56d5e00 msgr2=0x5598d5740350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:04.549 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.544+0000 7f15beffd640 1 --2- 172.21.15.23:0/1905643795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56d5e00 0x5598d5740350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:04.549 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.544+0000 7f15beffd640 1 -- 172.21.15.23:0/1905643795 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5598d56d0550 con 0x5598d56cc710 2024-03-28T05:28:04.549 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.545+0000 7f15beffd640 1 --2- 172.21.15.23:0/1905643795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56cc710 0x5598d56cf780 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f15b0000c00 tx=0x7f15b0004b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e188d97c3ba320 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.545+0000 7f15a2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:28:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.545+0000 7f15a2ff5640 1 -- 172.21.15.23:0/1905643795 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b001edb0 con 0x5598d56cc710 2024-03-28T05:28:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.545+0000 7f15c83aa740 1 -- 172.21.15.23:0/1905643795 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f15b8010460 con 0x5598d56cc710 2024-03-28T05:28:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.545+0000 7f15a2ff5640 1 -- 172.21.15.23:0/1905643795 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f15b0012450 con 0x5598d56cc710 2024-03-28T05:28:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.545+0000 7f15a2ff5640 1 -- 172.21.15.23:0/1905643795 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15b002f050 con 0x5598d56cc710 2024-03-28T05:28:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.545+0000 7f15c83aa740 10 client.15843 fetch_fsmap learned FSMap version 20 2024-03-28T05:28:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.545+0000 7f15c83aa740 1 -- 172.21.15.23:0/1905643795 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5598d57419d0 con 0x5598d56cc710 2024-03-28T05:28:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.545+0000 7f15a2ff5640 1 -- 172.21.15.23:0/1905643795 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15b8010460 con 0x5598d56cc710 2024-03-28T05:28:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.545+0000 7f15a2ff5640 1 -- 172.21.15.23:0/1905643795 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f15b00264f0 con 0x5598d56cc710 2024-03-28T05:28:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.545+0000 7f15c83aa740 10 client.15843 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:28:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.545+0000 7f15c83aa740 10 client.15843 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:28:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.545+0000 7f15c83aa740 20 client.15843 populate_metadata read hostname 'smithi023' 2024-03-28T05:28:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.545+0000 7f15c83aa740 1 --2- 172.21.15.23:0/1905643795 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5598d5742050 0x5598d5744470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.546+0000 7f15c83aa740 4 client.15843 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:28:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.546+0000 7f15c83aa740 1 -- 172.21.15.23:0/1905643795 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5598d56383a0 con 0x5598d5742050 2024-03-28T05:28:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.546+0000 7f15be7fc640 1 --2- 172.21.15.23:0/1905643795 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5598d5742050 0x5598d5744470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.546+0000 7f15a2ff5640 10 client.15843 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:28:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:04.546+0000 7f15be7fc640 1 --2- 172.21.15.23:0/1905643795 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5598d5742050 0x5598d5744470 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bf154d0f22bd281 server_cookie=ccd6d1d0237978ba in_seq=0 out_seq=0 2024-03-28T05:28:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.083+0000 7f15a2ff5640 1 -- 172.21.15.23:0/1905643795 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5598d56383a0 con 0x5598d5742050 2024-03-28T05:28:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.083+0000 7f15a2ff5640 10 client.15843 handle_command_reply: tid=0 2024-03-28T05:28:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.083+0000 7f15c83aa740 1 client.15843 shutdown 2024-03-28T05:28:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.083+0000 7f15c83aa740 1 -- 172.21.15.23:0/1905643795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56cc710 msgr2=0x5598d56cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.083+0000 7f15c83aa740 1 --2- 172.21.15.23:0/1905643795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56cc710 0x5598d56cf780 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f15b0000c00 tx=0x7f15b0004b70 comp rx=0 tx=0).stop 2024-03-28T05:28:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.083+0000 7f15c83aa740 1 -- 172.21.15.23:0/1905643795 shutdown_connections 2024-03-28T05:28:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.083+0000 7f15c83aa740 1 --2- 172.21.15.23:0/1905643795 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5598d5742050 0x5598d5744470 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.083+0000 7f15c83aa740 1 --2- 172.21.15.23:0/1905643795 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598d56d5e00 0x5598d5740350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.083+0000 7f15c83aa740 1 --2- 172.21.15.23:0/1905643795 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5598d56cd030 0x5598d56cfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.083+0000 7f15a2ff5640 0 client.15843 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:28:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.083+0000 7f15c83aa740 1 --2- 172.21.15.23:0/1905643795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5598d56cc710 0x5598d56cf780 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.083+0000 7f15c83aa740 1 -- 172.21.15.23:0/1905643795 >> 172.21.15.23:0/1905643795 conn(0x5598d56c7030 msgr2=0x5598d56cd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.083+0000 7f15c83aa740 1 -- 172.21.15.23:0/1905643795 shutdown_connections 2024-03-28T05:28:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.083+0000 7f15c83aa740 1 -- 172.21.15.23:0/1905643795 wait complete. 2024-03-28T05:28:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.084+0000 7f15c83aa740 20 client.15843 trim_cache size 0 max 16384 2024-03-28T05:28:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:28:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:28:05.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:28:05.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:28:05.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.084+0000 7f15c56e8640 1 -- 172.21.15.23:0/3889061590 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f15a80a2bb0 msgr2=0x7f15a80a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.084+0000 7f15c56e8640 1 --2- 172.21.15.23:0/3889061590 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f15a80a2bb0 0x7f15a80a5070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f15b80099e0 tx=0x7f15b8004a40 comp rx=0 tx=0).stop 2024-03-28T05:28:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.084+0000 7f15c56e8640 1 -- 172.21.15.23:0/3889061590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15c0102010 msgr2=0x7f15c01983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.084+0000 7f15c56e8640 1 --2- 172.21.15.23:0/3889061590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15c0102010 0x7f15c01983d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009230 tx=0x7f15b402cb30 comp rx=0 tx=0).stop 2024-03-28T05:28:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.084+0000 7f15c56e8640 1 -- 172.21.15.23:0/3889061590 shutdown_connections 2024-03-28T05:28:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.084+0000 7f15c56e8640 1 --2- 172.21.15.23:0/3889061590 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15c0107660 0x7f15c019c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.084+0000 7f15c56e8640 1 --2- 172.21.15.23:0/3889061590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15c0102010 0x7f15c01983d0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.084+0000 7f15c56e8640 1 --2- 172.21.15.23:0/3889061590 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f15a80a2bb0 0x7f15a80a5070 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.084+0000 7f15c56e8640 1 --2- 172.21.15.23:0/3889061590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15c0101650 0x7f15c0197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.084+0000 7f15c56e8640 1 -- 172.21.15.23:0/3889061590 >> 172.21.15.23:0/3889061590 conn(0x7f15c00fd160 msgr2=0x7f15c00fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.084+0000 7f15c56e8640 1 -- 172.21.15.23:0/3889061590 shutdown_connections 2024-03-28T05:28:05.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:05.084+0000 7f15c56e8640 1 -- 172.21.15.23:0/3889061590 wait complete. 2024-03-28T05:28:06.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:05 smithi113 ceph-mon[37164]: pgmap v138: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 717 KiB/s wr, 351 op/s 2024-03-28T05:28:06.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:05 smithi023 ceph-mon[31954]: pgmap v138: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 717 KiB/s wr, 351 op/s 2024-03-28T05:28:06.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:05 smithi098 ceph-mon[36087]: pgmap v138: 129 pgs: 129 active+clean; 139 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 717 KiB/s wr, 351 op/s 2024-03-28T05:28:06.400 INFO:tasks.workunit.client.0.smithi023.stdout: 43 142041 124 97373 124 64272 317 2024-03-28T05:28:07.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:07 smithi023 ceph-mon[31954]: pgmap v139: 129 pgs: 129 active+clean; 142 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 997 KiB/s rd, 1.1 MiB/s wr, 686 op/s 2024-03-28T05:28:07.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:07 smithi098 ceph-mon[36087]: pgmap v139: 129 pgs: 129 active+clean; 142 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 997 KiB/s rd, 1.1 MiB/s wr, 686 op/s 2024-03-28T05:28:07.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:07 smithi113 ceph-mon[37164]: pgmap v139: 129 pgs: 129 active+clean; 142 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 997 KiB/s rd, 1.1 MiB/s wr, 686 op/s 2024-03-28T05:28:09.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:09 smithi098 ceph-mon[36087]: pgmap v140: 129 pgs: 129 active+clean; 142 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 680 KiB/s wr, 413 op/s 2024-03-28T05:28:09.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:09 smithi113 ceph-mon[37164]: pgmap v140: 129 pgs: 129 active+clean; 142 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 680 KiB/s wr, 413 op/s 2024-03-28T05:28:09.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:09 smithi023 ceph-mon[31954]: pgmap v140: 129 pgs: 129 active+clean; 142 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 680 KiB/s wr, 413 op/s 2024-03-28T05:28:11.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:11 smithi098 ceph-mon[36087]: pgmap v141: 129 pgs: 129 active+clean; 149 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 455 op/s 2024-03-28T05:28:11.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:11 smithi113 ceph-mon[37164]: pgmap v141: 129 pgs: 129 active+clean; 149 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 455 op/s 2024-03-28T05:28:11.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:11 smithi023 ceph-mon[31954]: pgmap v141: 129 pgs: 129 active+clean; 149 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 455 op/s 2024-03-28T05:28:13.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:13 smithi098 ceph-mon[36087]: pgmap v142: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 565 op/s 2024-03-28T05:28:13.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:13 smithi113 ceph-mon[37164]: pgmap v142: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 565 op/s 2024-03-28T05:28:13.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:13 smithi023 ceph-mon[31954]: pgmap v142: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 565 op/s 2024-03-28T05:28:15.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:15 smithi098 ceph-mon[36087]: pgmap v143: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 2.6 MiB/s wr, 501 op/s 2024-03-28T05:28:15.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:15 smithi113 ceph-mon[37164]: pgmap v143: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 2.6 MiB/s wr, 501 op/s 2024-03-28T05:28:15.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:15 smithi023 ceph-mon[31954]: pgmap v143: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 2.6 MiB/s wr, 501 op/s 2024-03-28T05:28:16.400 INFO:tasks.workunit.client.0.smithi023.stdout: 45 200279 129 132792 117 95840 391 2024-03-28T05:28:17.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:17 smithi098 ceph-mon[36087]: pgmap v144: 129 pgs: 129 active+clean; 165 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 573 op/s 2024-03-28T05:28:17.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:17 smithi113 ceph-mon[37164]: pgmap v144: 129 pgs: 129 active+clean; 165 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 573 op/s 2024-03-28T05:28:17.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:17 smithi023 ceph-mon[31954]: pgmap v144: 129 pgs: 129 active+clean; 165 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 573 op/s 2024-03-28T05:28:18.179 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.177+0000 7f1392fab640 1 -- 172.21.15.23:0/1678238742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f138c102910 msgr2=0x7f138c102cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:18.179 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.177+0000 7f1392fab640 1 --2- 172.21.15.23:0/1678238742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f138c102910 0x7f138c102cf0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f1388009900 tx=0x7f138802bcd0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.179 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.177+0000 7f1392fab640 1 -- 172.21.15.23:0/1678238742 shutdown_connections 2024-03-28T05:28:18.179 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.177+0000 7f1392fab640 1 --2- 172.21.15.23:0/1678238742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f138c10bce0 0x7f138c10e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.179 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.177+0000 7f1392fab640 1 --2- 172.21.15.23:0/1678238742 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f138c103230 0x7f138c10b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.179 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.177+0000 7f1392fab640 1 --2- 172.21.15.23:0/1678238742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f138c102910 0x7f138c102cf0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.179 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.177+0000 7f1392fab640 1 -- 172.21.15.23:0/1678238742 >> 172.21.15.23:0/1678238742 conn(0x7f138c077700 msgr2=0x7f138c077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:18.179 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.178+0000 7f1392fab640 1 -- 172.21.15.23:0/1678238742 shutdown_connections 2024-03-28T05:28:18.179 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.178+0000 7f1392fab640 1 -- 172.21.15.23:0/1678238742 wait complete. 2024-03-28T05:28:18.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.178+0000 7f1392fab640 1 Processor -- start 2024-03-28T05:28:18.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.178+0000 7f1392fab640 1 -- start start 2024-03-28T05:28:18.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.178+0000 7f1392fab640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f138c102910 0x7f138c197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:18.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.178+0000 7f1392fab640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f138c103230 0x7f138c198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:18.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.178+0000 7f1392fab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f138c10bce0 0x7f138c19c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:18.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.178+0000 7f1392fab640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c111a70 con 0x7f138c102910 2024-03-28T05:28:18.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.178+0000 7f1392fab640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c1118f0 con 0x7f138c10bce0 2024-03-28T05:28:18.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.178+0000 7f1392fab640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c111bf0 con 0x7f138c103230 2024-03-28T05:28:18.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.178+0000 7f1390d47640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f138c102910 0x7f138c197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:18.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.178+0000 7f1390d47640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f138c102910 0x7f138c197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43160/0 (socket says 172.21.15.23:43160) 2024-03-28T05:28:18.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.178+0000 7f1390d47640 1 -- 172.21.15.23:0/1794148095 learned_addr learned my addr 172.21.15.23:0/1794148095 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:18.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.178+0000 7f1383fff640 1 --2- 172.21.15.23:0/1794148095 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f138c103230 0x7f138c198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:18.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.179+0000 7f1390d47640 1 -- 172.21.15.23:0/1794148095 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f138c103230 msgr2=0x7f138c198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:18.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.179+0000 7f1390d47640 1 --2- 172.21.15.23:0/1794148095 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f138c103230 0x7f138c198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.180 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.179+0000 7f1390d47640 1 -- 172.21.15.23:0/1794148095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f138c10bce0 msgr2=0x7f138c19c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:18.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.179+0000 7f1390d47640 1 --2- 172.21.15.23:0/1794148095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f138c10bce0 0x7f138c19c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.179+0000 7f1390d47640 1 -- 172.21.15.23:0/1794148095 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138c19cec0 con 0x7f138c102910 2024-03-28T05:28:18.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.179+0000 7f1390d47640 1 --2- 172.21.15.23:0/1794148095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f138c102910 0x7f138c197ed0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f1388005940 tx=0x7f138802cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:18.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.179+0000 7f1381ffb640 1 -- 172.21.15.23:0/1794148095 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1388041070 con 0x7f138c102910 2024-03-28T05:28:18.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.179+0000 7f1381ffb640 1 -- 172.21.15.23:0/1794148095 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1388004450 con 0x7f138c102910 2024-03-28T05:28:18.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.179+0000 7f1392fab640 1 -- 172.21.15.23:0/1794148095 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f138c19d150 con 0x7f138c102910 2024-03-28T05:28:18.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.179+0000 7f1381ffb640 1 -- 172.21.15.23:0/1794148095 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1388003df0 con 0x7f138c102910 2024-03-28T05:28:18.181 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.179+0000 7f1392fab640 1 -- 172.21.15.23:0/1794148095 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f138c1a4a40 con 0x7f138c102910 2024-03-28T05:28:18.186 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.182+0000 7f1395b8a740 1 Processor -- start 2024-03-28T05:28:18.186 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.182+0000 7f1395b8a740 1 -- start start 2024-03-28T05:28:18.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.182+0000 7f1381ffb640 1 -- 172.21.15.23:0/1794148095 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f1388044070 con 0x7f138c102910 2024-03-28T05:28:18.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.182+0000 7f1395b8a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe0710 0x55d008fe0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:18.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.182+0000 7f1395b8a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d008fe1030 0x55d008fe98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:18.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.182+0000 7f1395b8a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d008fe9e00 0x55d008fec1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:18.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.182+0000 7f1395b8a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d008f44b10 con 0x55d008fe0710 2024-03-28T05:28:18.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.182+0000 7f1395b8a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d008eed900 con 0x55d008fe1030 2024-03-28T05:28:18.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.182+0000 7f1395b8a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d008f40bb0 con 0x55d008fe9e00 2024-03-28T05:28:18.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.183+0000 7f1391548640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe0710 0x55d008fe0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:18.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.183+0000 7f1383fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d008fe9e00 0x55d008fec1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:18.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.183+0000 7f1391548640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe0710 0x55d008fe0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43166/0 (socket says 172.21.15.23:43166) 2024-03-28T05:28:18.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.183+0000 7f1391548640 1 -- 172.21.15.23:0/2595203490 learned_addr learned my addr 172.21.15.23:0/2595203490 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:18.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.183+0000 7f1390d47640 1 --2- 172.21.15.23:0/2595203490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d008fe1030 0x55d008fe98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:18.187 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.183+0000 7f1391548640 1 -- 172.21.15.23:0/2595203490 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d008fe9e00 msgr2=0x55d008fec1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.183+0000 7f1391548640 1 --2- 172.21.15.23:0/2595203490 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d008fe9e00 0x55d008fec1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.183+0000 7f1391548640 1 -- 172.21.15.23:0/2595203490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d008fe1030 msgr2=0x55d008fe98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.183+0000 7f1391548640 1 --2- 172.21.15.23:0/2595203490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d008fe1030 0x55d008fe98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.183+0000 7f1391548640 1 -- 172.21.15.23:0/2595203490 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d008c59210 con 0x55d008fe0710 2024-03-28T05:28:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.183+0000 7f1381ffb640 1 --2- 172.21.15.23:0/1794148095 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f13740a2a10 0x7f13740a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.183+0000 7f1390d47640 1 --2- 172.21.15.23:0/1794148095 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f13740a2a10 0x7f13740a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.183+0000 7f1391548640 1 --2- 172.21.15.23:0/2595203490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe0710 0x55d008fe0af0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1384009cd0 tx=0x7f138400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cb56e4003defa60 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.183+0000 7f1381ffb640 1 -- 172.21.15.23:0/1794148095 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f13880f0cc0 con 0x7f138c102910 2024-03-28T05:28:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.183+0000 7f1372ffd640 1 -- 172.21.15.23:0/2595203490 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1384016070 con 0x55d008fe0710 2024-03-28T05:28:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.183+0000 7f1372ffd640 1 -- 172.21.15.23:0/2595203490 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1384005020 con 0x55d008fe0710 2024-03-28T05:28:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.183+0000 7f1372ffd640 1 -- 172.21.15.23:0/2595203490 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13840116d0 con 0x55d008fe0710 2024-03-28T05:28:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.184+0000 7f1395b8a740 1 -- 172.21.15.23:0/2595203490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe0710 msgr2=0x55d008fe0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.184+0000 7f1390d47640 1 --2- 172.21.15.23:0/1794148095 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f13740a2a10 0x7f13740a4ed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f13880ee240 tx=0x7f1388003f90 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.184+0000 7f1395b8a740 1 --2- 172.21.15.23:0/2595203490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe0710 0x55d008fe0af0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1384009cd0 tx=0x7f138400e6c0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.184+0000 7f1395b8a740 1 -- 172.21.15.23:0/2595203490 shutdown_connections 2024-03-28T05:28:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.184+0000 7f1395b8a740 1 --2- 172.21.15.23:0/2595203490 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d008fe9e00 0x55d008fec1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.184+0000 7f1395b8a740 1 --2- 172.21.15.23:0/2595203490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d008fe1030 0x55d008fe98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.184+0000 7f1395b8a740 1 --2- 172.21.15.23:0/2595203490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe0710 0x55d008fe0af0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.184+0000 7f1395b8a740 1 -- 172.21.15.23:0/2595203490 >> 172.21.15.23:0/2595203490 conn(0x55d008fdb030 msgr2=0x55d008fdb410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.184+0000 7f1395b8a740 1 -- 172.21.15.23:0/2595203490 shutdown_connections 2024-03-28T05:28:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.184+0000 7f1395b8a740 1 -- 172.21.15.23:0/2595203490 wait complete. 2024-03-28T05:28:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.185+0000 7f1395b8a740 1 -- start start 2024-03-28T05:28:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.185+0000 7f1395b8a740 1 Processor -- start 2024-03-28T05:28:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.185+0000 7f1395b8a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe0710 0x55d008fe6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.185+0000 7f1395b8a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d008fe1030 0x55d008fe72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.185+0000 7f1395b8a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d008fe9e00 0x55d008fe77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.185+0000 7f1395b8a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d008feca30 con 0x55d008fe0710 2024-03-28T05:28:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.185+0000 7f1395b8a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d008f00d50 con 0x55d008fe1030 2024-03-28T05:28:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.185+0000 7f1395b8a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d008fecbb0 con 0x55d008fe9e00 2024-03-28T05:28:18.189 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.185+0000 7f1395b8a740 10 client.0 _ll_register_callbacks cb 0x55d008f00720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:28:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.185+0000 7f1391548640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe0710 0x55d008fe6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.185+0000 7f1391548640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe0710 0x55d008fe6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43174/0 (socket says 172.21.15.23:43174) 2024-03-28T05:28:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.185+0000 7f1391548640 1 -- 172.21.15.23:0/2005508012 learned_addr learned my addr 172.21.15.23:0/2005508012 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.185+0000 7f1383fff640 1 --2- 172.21.15.23:0/2005508012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d008fe1030 0x55d008fe72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.185+0000 7f1391548640 1 --2- 172.21.15.23:0/2005508012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d008fe9e00 0x55d008fe77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.185+0000 7f1391548640 1 -- 172.21.15.23:0/2005508012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d008fe9e00 msgr2=0x55d008fe77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.185+0000 7f1391548640 1 --2- 172.21.15.23:0/2005508012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d008fe9e00 0x55d008fe77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.185+0000 7f1391548640 1 -- 172.21.15.23:0/2005508012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d008fe1030 msgr2=0x55d008fe72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.185+0000 7f1391548640 1 --2- 172.21.15.23:0/2005508012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d008fe1030 0x55d008fe72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.185+0000 7f1391548640 1 -- 172.21.15.23:0/2005508012 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d008d25620 con 0x55d008fe0710 2024-03-28T05:28:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.186+0000 7f1391548640 1 --2- 172.21.15.23:0/2005508012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe0710 0x55d008fe6d60 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f1384006010 tx=0x7f138400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abcaf41d039824ca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.186+0000 7f136bfff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:28:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.186+0000 7f136bfff640 1 -- 172.21.15.23:0/2005508012 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1384016040 con 0x55d008fe0710 2024-03-28T05:28:18.190 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.186+0000 7f136bfff640 1 -- 172.21.15.23:0/2005508012 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f13840054d0 con 0x55d008fe0710 2024-03-28T05:28:18.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.186+0000 7f1395b8a740 1 -- 172.21.15.23:0/2005508012 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1384005020 con 0x55d008fe0710 2024-03-28T05:28:18.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.186+0000 7f136bfff640 1 -- 172.21.15.23:0/2005508012 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1384031050 con 0x55d008fe0710 2024-03-28T05:28:18.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.186+0000 7f1395b8a740 10 client.15855 fetch_fsmap learned FSMap version 20 2024-03-28T05:28:18.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.186+0000 7f1395b8a740 1 -- 172.21.15.23:0/2005508012 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55d008ff8c00 con 0x55d008fe0710 2024-03-28T05:28:18.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.186+0000 7f136bfff640 1 -- 172.21.15.23:0/2005508012 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1384005020 con 0x55d008fe0710 2024-03-28T05:28:18.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.186+0000 7f136bfff640 1 -- 172.21.15.23:0/2005508012 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f138400f040 con 0x55d008fe0710 2024-03-28T05:28:18.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.186+0000 7f1395b8a740 10 client.15855 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:28:18.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.186+0000 7f1395b8a740 10 client.15855 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:28:18.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.186+0000 7f1395b8a740 20 client.15855 populate_metadata read hostname 'smithi023' 2024-03-28T05:28:18.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.186+0000 7f1395b8a740 1 --2- 172.21.15.23:0/2005508012 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d008ff9fb0 0x55d008ffc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:18.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.186+0000 7f1395b8a740 4 client.15855 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:28:18.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.186+0000 7f1395b8a740 1 -- 172.21.15.23:0/2005508012 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d008f52050 con 0x55d008ff9fb0 2024-03-28T05:28:18.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.187+0000 7f1383fff640 1 --2- 172.21.15.23:0/2005508012 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d008ff9fb0 0x55d008ffc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:18.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.187+0000 7f1383fff640 1 --2- 172.21.15.23:0/2005508012 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d008ff9fb0 0x55d008ffc3a0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe4e69e7a2e02c0e server_cookie=711d06a68cccdbe6 in_seq=0 out_seq=0 2024-03-28T05:28:18.191 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.187+0000 7f136bfff640 10 client.15855 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:28:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.188+0000 7f136bfff640 1 -- 172.21.15.23:0/2005508012 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d008f52050 con 0x55d008ff9fb0 2024-03-28T05:28:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.188+0000 7f136bfff640 10 client.15855 handle_command_reply: tid=0 2024-03-28T05:28:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.188+0000 7f1395b8a740 1 client.15855 shutdown 2024-03-28T05:28:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.189+0000 7f1395b8a740 1 -- 172.21.15.23:0/2005508012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe0710 msgr2=0x55d008fe6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.189+0000 7f1395b8a740 1 --2- 172.21.15.23:0/2005508012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe0710 0x55d008fe6d60 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f1384006010 tx=0x7f138400eef0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.189+0000 7f1395b8a740 1 -- 172.21.15.23:0/2005508012 shutdown_connections 2024-03-28T05:28:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.189+0000 7f1395b8a740 1 --2- 172.21.15.23:0/2005508012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d008fe9e00 0x55d008fe77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.189+0000 7f1395b8a740 1 --2- 172.21.15.23:0/2005508012 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d008ff9fb0 0x55d008ffc3a0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.189+0000 7f1395b8a740 1 --2- 172.21.15.23:0/2005508012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d008fe1030 0x55d008fe72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.189+0000 7f1395b8a740 1 --2- 172.21.15.23:0/2005508012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe0710 0x55d008fe6d60 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.189+0000 7f1395b8a740 1 -- 172.21.15.23:0/2005508012 >> 172.21.15.23:0/2005508012 conn(0x55d008fdb030 msgr2=0x55d008fdcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.189+0000 7f136bfff640 0 client.15855 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:28:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.189+0000 7f1395b8a740 1 -- 172.21.15.23:0/2005508012 shutdown_connections 2024-03-28T05:28:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.189+0000 7f1395b8a740 1 -- 172.21.15.23:0/2005508012 wait complete. 2024-03-28T05:28:18.193 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.189+0000 7f1395b8a740 20 client.15855 trim_cache size 0 max 16384 2024-03-28T05:28:18.205 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.204+0000 7f1395b8a740 1 Processor -- start 2024-03-28T05:28:18.205 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.204+0000 7f1395b8a740 1 -- start start 2024-03-28T05:28:18.205 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.204+0000 7f1395b8a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe0710 0x55d008fe9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:18.205 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.204+0000 7f1395b8a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d008fe1030 0x55d008fd9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:18.206 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.204+0000 7f1395b8a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d008fe9e00 0x55d008ff52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:18.206 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.204+0000 7f1395b8a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c1acab0 con 0x55d008fe0710 2024-03-28T05:28:18.206 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.204+0000 7f1395b8a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c1af070 con 0x55d008fe9e00 2024-03-28T05:28:18.206 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.204+0000 7f1395b8a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c1ad0b0 con 0x55d008fe1030 2024-03-28T05:28:18.206 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.204+0000 7f1391548640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe0710 0x55d008fe9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:18.206 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.204+0000 7f1391548640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe0710 0x55d008fe9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43176/0 (socket says 172.21.15.23:43176) 2024-03-28T05:28:18.206 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.205+0000 7f1391548640 1 -- 172.21.15.23:0/4281021660 learned_addr learned my addr 172.21.15.23:0/4281021660 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:18.206 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.205+0000 7f1383fff640 1 --2- 172.21.15.23:0/4281021660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d008fe1030 0x55d008fd9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:18.206 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.205+0000 7f1391548640 1 -- 172.21.15.23:0/4281021660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d008fe1030 msgr2=0x55d008fd9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:18.206 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.205+0000 7f1391548640 1 --2- 172.21.15.23:0/4281021660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d008fe1030 0x55d008fd9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.206 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.205+0000 7f1391548640 1 -- 172.21.15.23:0/4281021660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d008fe9e00 msgr2=0x55d008ff52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:18.206 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.205+0000 7f1391548640 1 --2- 172.21.15.23:0/4281021660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d008fe9e00 0x55d008ff52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.206 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.205+0000 7f1391548640 1 -- 172.21.15.23:0/4281021660 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d008fe52b0 con 0x55d008fe0710 2024-03-28T05:28:18.206 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.205+0000 7f1391548640 1 --2- 172.21.15.23:0/4281021660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe0710 0x55d008fe9490 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f1384005800 tx=0x7f1384004890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4d66c4e9f6ad496 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:18.207 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.205+0000 7f13727fc640 1 -- 172.21.15.23:0/4281021660 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138403a070 con 0x55d008fe0710 2024-03-28T05:28:18.207 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.205+0000 7f13727fc640 1 -- 172.21.15.23:0/4281021660 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f138402d070 con 0x55d008fe0710 2024-03-28T05:28:18.207 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.205+0000 7f13727fc640 1 -- 172.21.15.23:0/4281021660 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13840051d0 con 0x55d008fe0710 2024-03-28T05:28:18.207 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.206+0000 7f1395b8a740 1 -- 172.21.15.23:0/4281021660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe0710 msgr2=0x55d008fe9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:18.207 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.206+0000 7f1395b8a740 1 --2- 172.21.15.23:0/4281021660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe0710 0x55d008fe9490 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f1384005800 tx=0x7f1384004890 comp rx=0 tx=0).stop 2024-03-28T05:28:18.207 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.206+0000 7f1395b8a740 1 -- 172.21.15.23:0/4281021660 shutdown_connections 2024-03-28T05:28:18.207 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.206+0000 7f1395b8a740 1 --2- 172.21.15.23:0/4281021660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d008fe9e00 0x55d008ff52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.207 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.206+0000 7f1395b8a740 1 --2- 172.21.15.23:0/4281021660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d008fe1030 0x55d008fd9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.207 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.206+0000 7f1395b8a740 1 --2- 172.21.15.23:0/4281021660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe0710 0x55d008fe9490 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.207 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.206+0000 7f1395b8a740 1 -- 172.21.15.23:0/4281021660 >> 172.21.15.23:0/4281021660 conn(0x55d008fdb030 msgr2=0x55d008fd9790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:18.207 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.206+0000 7f1395b8a740 1 -- 172.21.15.23:0/4281021660 shutdown_connections 2024-03-28T05:28:18.207 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.206+0000 7f1395b8a740 1 -- 172.21.15.23:0/4281021660 wait complete. 2024-03-28T05:28:18.207 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.206+0000 7f1395b8a740 1 -- start start 2024-03-28T05:28:18.207 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.206+0000 7f1395b8a740 1 Processor -- start 2024-03-28T05:28:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f1395b8a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d008fe0710 0x55d008ff4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f1395b8a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe1030 0x55d008ff54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f1395b8a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d008fe9e00 0x55d009050be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f1395b8a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c1ad6f0 con 0x55d008fe1030 2024-03-28T05:28:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f1395b8a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c1aea10 con 0x55d008fe9e00 2024-03-28T05:28:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f1395b8a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c1add50 con 0x55d008fe0710 2024-03-28T05:28:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f1395b8a740 10 client.0 _ll_register_callbacks cb 0x7f1384009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:28:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f1383fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe1030 0x55d008ff54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:18.210 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f1383fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe1030 0x55d008ff54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43192/0 (socket says 172.21.15.23:43192) 2024-03-28T05:28:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f1383fff640 1 -- 172.21.15.23:0/894550316 learned_addr learned my addr 172.21.15.23:0/894550316 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f1391548640 1 --2- 172.21.15.23:0/894550316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d008fe0710 0x55d008ff4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f1391548640 1 --2- 172.21.15.23:0/894550316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d008fe9e00 0x55d009050be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f1383fff640 1 -- 172.21.15.23:0/894550316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d008fe0710 msgr2=0x55d008ff4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f1383fff640 1 --2- 172.21.15.23:0/894550316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d008fe0710 0x55d008ff4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f1383fff640 1 -- 172.21.15.23:0/894550316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d008fe9e00 msgr2=0x55d009050be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f1383fff640 1 --2- 172.21.15.23:0/894550316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d008fe9e00 0x55d009050be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f1383fff640 1 -- 172.21.15.23:0/894550316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d009051250 con 0x55d008fe1030 2024-03-28T05:28:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f1391548640 1 --2- 172.21.15.23:0/894550316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d008fe9e00 0x55d009050be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:28:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f1383fff640 1 --2- 172.21.15.23:0/894550316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe1030 0x55d008ff54e0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f137c0098e0 tx=0x7f137c004100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3628a996f68a0580 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f136bfff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:28:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f136bfff640 1 -- 172.21.15.23:0/894550316 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137c024030 con 0x55d008fe1030 2024-03-28T05:28:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f136bfff640 1 -- 172.21.15.23:0/894550316 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f137c020b00 con 0x55d008fe1030 2024-03-28T05:28:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.207+0000 7f1395b8a740 1 -- 172.21.15.23:0/894550316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f138402d070 con 0x55d008fe1030 2024-03-28T05:28:18.211 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.208+0000 7f136bfff640 1 -- 172.21.15.23:0/894550316 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f137c02e050 con 0x55d008fe1030 2024-03-28T05:28:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.208+0000 7f1395b8a740 10 client.15861 fetch_fsmap learned FSMap version 20 2024-03-28T05:28:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.208+0000 7f1395b8a740 1 -- 172.21.15.23:0/894550316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55d009059860 con 0x55d008fe1030 2024-03-28T05:28:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.208+0000 7f136bfff640 1 -- 172.21.15.23:0/894550316 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f138402d070 con 0x55d008fe1030 2024-03-28T05:28:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.208+0000 7f136bfff640 1 -- 172.21.15.23:0/894550316 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f137c010660 con 0x55d008fe1030 2024-03-28T05:28:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.208+0000 7f1395b8a740 10 client.15861 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:28:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.208+0000 7f1395b8a740 10 client.15861 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:28:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.208+0000 7f1395b8a740 20 client.15861 populate_metadata read hostname 'smithi023' 2024-03-28T05:28:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.208+0000 7f1395b8a740 1 --2- 172.21.15.23:0/894550316 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d009059fc0 0x55d00905c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.208+0000 7f1395b8a740 4 client.15861 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:28:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.208+0000 7f1395b8a740 1 -- 172.21.15.23:0/894550316 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d008f4c3a0 con 0x55d009059fc0 2024-03-28T05:28:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.208+0000 7f1391548640 1 --2- 172.21.15.23:0/894550316 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d009059fc0 0x55d00905c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.209+0000 7f136bfff640 10 client.15861 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:28:18.212 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.209+0000 7f1391548640 1 --2- 172.21.15.23:0/894550316 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d009059fc0 0x55d00905c3e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76d22e68e682e77a server_cookie=971c8b9e49b4dddd in_seq=0 out_seq=0 2024-03-28T05:28:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.728+0000 7f136bfff640 1 -- 172.21.15.23:0/894550316 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d008f4c3a0 con 0x55d009059fc0 2024-03-28T05:28:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.728+0000 7f136bfff640 10 client.15861 handle_command_reply: tid=0 2024-03-28T05:28:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.728+0000 7f1395b8a740 1 client.15861 shutdown 2024-03-28T05:28:18.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.728+0000 7f1395b8a740 1 -- 172.21.15.23:0/894550316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe1030 msgr2=0x55d008ff54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:18.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.728+0000 7f1395b8a740 1 --2- 172.21.15.23:0/894550316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe1030 0x55d008ff54e0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f137c0098e0 tx=0x7f137c004100 comp rx=0 tx=0).stop 2024-03-28T05:28:18.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.729+0000 7f1395b8a740 1 -- 172.21.15.23:0/894550316 shutdown_connections 2024-03-28T05:28:18.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.729+0000 7f1395b8a740 1 --2- 172.21.15.23:0/894550316 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d009059fc0 0x55d00905c3e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.729+0000 7f1395b8a740 1 --2- 172.21.15.23:0/894550316 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d008fe9e00 0x55d009050be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.729+0000 7f1395b8a740 1 --2- 172.21.15.23:0/894550316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d008fe1030 0x55d008ff54e0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.729+0000 7f1395b8a740 1 --2- 172.21.15.23:0/894550316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d008fe0710 0x55d008ff4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.729+0000 7f1395b8a740 1 -- 172.21.15.23:0/894550316 >> 172.21.15.23:0/894550316 conn(0x55d008fdb030 msgr2=0x55d008fe75b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:18.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.729+0000 7f136bfff640 0 client.15861 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:28:18.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.729+0000 7f1395b8a740 1 -- 172.21.15.23:0/894550316 shutdown_connections 2024-03-28T05:28:18.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.729+0000 7f1395b8a740 1 -- 172.21.15.23:0/894550316 wait complete. 2024-03-28T05:28:18.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.729+0000 7f1395b8a740 20 client.15861 trim_cache size 0 max 16384 2024-03-28T05:28:18.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:28:18.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:28:18.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:28:18.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:28:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.730+0000 7f1392fab640 1 -- 172.21.15.23:0/1794148095 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f13740a2a10 msgr2=0x7f13740a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.730+0000 7f1392fab640 1 --2- 172.21.15.23:0/1794148095 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f13740a2a10 0x7f13740a4ed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f13880ee240 tx=0x7f1388003f90 comp rx=0 tx=0).stop 2024-03-28T05:28:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.730+0000 7f1392fab640 1 -- 172.21.15.23:0/1794148095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f138c102910 msgr2=0x7f138c197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.730+0000 7f1392fab640 1 --2- 172.21.15.23:0/1794148095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f138c102910 0x7f138c197ed0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f1388005940 tx=0x7f138802cc40 comp rx=0 tx=0).stop 2024-03-28T05:28:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.730+0000 7f1392fab640 1 -- 172.21.15.23:0/1794148095 shutdown_connections 2024-03-28T05:28:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.730+0000 7f1392fab640 1 --2- 172.21.15.23:0/1794148095 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f13740a2a10 0x7f13740a4ed0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.730+0000 7f1392fab640 1 --2- 172.21.15.23:0/1794148095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f138c10bce0 0x7f138c19c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.730+0000 7f1392fab640 1 --2- 172.21.15.23:0/1794148095 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f138c103230 0x7f138c198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.730+0000 7f1392fab640 1 --2- 172.21.15.23:0/1794148095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f138c102910 0x7f138c197ed0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.730+0000 7f1392fab640 1 -- 172.21.15.23:0/1794148095 >> 172.21.15.23:0/1794148095 conn(0x7f138c077700 msgr2=0x7f138c0fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.730+0000 7f1392fab640 1 -- 172.21.15.23:0/1794148095 shutdown_connections 2024-03-28T05:28:18.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:18.730+0000 7f1392fab640 1 -- 172.21.15.23:0/1794148095 wait complete. 2024-03-28T05:28:19.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:19 smithi098 ceph-mon[36087]: pgmap v145: 129 pgs: 129 active+clean; 165 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 239 op/s 2024-03-28T05:28:19.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:19 smithi113 ceph-mon[37164]: pgmap v145: 129 pgs: 129 active+clean; 165 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 239 op/s 2024-03-28T05:28:19.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:19 smithi023 ceph-mon[31954]: pgmap v145: 129 pgs: 129 active+clean; 165 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 239 op/s 2024-03-28T05:28:21.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:21 smithi098 ceph-mon[36087]: pgmap v146: 129 pgs: 129 active+clean; 155 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 331 op/s 2024-03-28T05:28:21.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:21 smithi113 ceph-mon[37164]: pgmap v146: 129 pgs: 129 active+clean; 155 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 331 op/s 2024-03-28T05:28:21.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:21 smithi023 ceph-mon[31954]: pgmap v146: 129 pgs: 129 active+clean; 155 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 331 op/s 2024-03-28T05:28:23.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:23 smithi098 ceph-mon[36087]: pgmap v147: 129 pgs: 129 active+clean; 151 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 524 op/s 2024-03-28T05:28:23.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:23 smithi113 ceph-mon[37164]: pgmap v147: 129 pgs: 129 active+clean; 151 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 524 op/s 2024-03-28T05:28:23.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:23 smithi023 ceph-mon[31954]: pgmap v147: 129 pgs: 129 active+clean; 151 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 524 op/s 2024-03-28T05:28:25.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:25 smithi098 ceph-mon[36087]: pgmap v148: 129 pgs: 129 active+clean; 151 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 738 KiB/s wr, 399 op/s 2024-03-28T05:28:25.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:25 smithi113 ceph-mon[37164]: pgmap v148: 129 pgs: 129 active+clean; 151 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 738 KiB/s wr, 399 op/s 2024-03-28T05:28:25.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:25 smithi023 ceph-mon[31954]: pgmap v148: 129 pgs: 129 active+clean; 151 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 738 KiB/s wr, 399 op/s 2024-03-28T05:28:26.400 INFO:tasks.workunit.client.0.smithi023.stdout: 47 206969 64 136552 176 98886 333 2024-03-28T05:28:27.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:27 smithi113 ceph-mon[37164]: pgmap v149: 129 pgs: 129 active+clean; 155 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 440 op/s 2024-03-28T05:28:27.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:27 smithi023 ceph-mon[31954]: pgmap v149: 129 pgs: 129 active+clean; 155 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 440 op/s 2024-03-28T05:28:27.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:27 smithi098 ceph-mon[36087]: pgmap v149: 129 pgs: 129 active+clean; 155 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 440 op/s 2024-03-28T05:28:29.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:29 smithi113 ceph-mon[37164]: pgmap v150: 129 pgs: 129 active+clean; 155 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 682 KiB/s wr, 367 op/s 2024-03-28T05:28:29.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:29 smithi023 ceph-mon[31954]: pgmap v150: 129 pgs: 129 active+clean; 155 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 682 KiB/s wr, 367 op/s 2024-03-28T05:28:29.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:29 smithi098 ceph-mon[36087]: pgmap v150: 129 pgs: 129 active+clean; 155 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 682 KiB/s wr, 367 op/s 2024-03-28T05:28:31.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:31 smithi113 ceph-mon[37164]: pgmap v151: 129 pgs: 129 active+clean; 158 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 1017 KiB/s wr, 392 op/s 2024-03-28T05:28:31.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:31 smithi023 ceph-mon[31954]: pgmap v151: 129 pgs: 129 active+clean; 158 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 1017 KiB/s wr, 392 op/s 2024-03-28T05:28:31.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:31 smithi098 ceph-mon[36087]: pgmap v151: 129 pgs: 129 active+clean; 158 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 1017 KiB/s wr, 392 op/s 2024-03-28T05:28:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.820+0000 7fcf8b59e640 1 -- 172.21.15.23:0/575558258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf8c1014a0 msgr2=0x7fcf8c101920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.821+0000 7fcf8b59e640 1 --2- 172.21.15.23:0/575558258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf8c1014a0 0x7fcf8c101920 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fcf80009900 tx=0x7fcf8002bc30 comp rx=0 tx=0).stop 2024-03-28T05:28:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.821+0000 7fcf8b59e640 1 -- 172.21.15.23:0/575558258 shutdown_connections 2024-03-28T05:28:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.821+0000 7fcf8b59e640 1 --2- 172.21.15.23:0/575558258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf8c101e60 0x7fcf8c10e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.821+0000 7fcf8b59e640 1 --2- 172.21.15.23:0/575558258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf8c1014a0 0x7fcf8c101920 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.821+0000 7fcf8b59e640 1 --2- 172.21.15.23:0/575558258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf8c107560 0x7fcf8c107940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.821+0000 7fcf8b59e640 1 -- 172.21.15.23:0/575558258 >> 172.21.15.23:0/575558258 conn(0x7fcf8c0fcff0 msgr2=0x7fcf8c0ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:31.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.821+0000 7fcf8b59e640 1 -- 172.21.15.23:0/575558258 shutdown_connections 2024-03-28T05:28:31.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.821+0000 7fcf8b59e640 1 -- 172.21.15.23:0/575558258 wait complete. 2024-03-28T05:28:31.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.821+0000 7fcf8b59e640 1 Processor -- start 2024-03-28T05:28:31.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.821+0000 7fcf8b59e640 1 -- start start 2024-03-28T05:28:31.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.822+0000 7fcf8b59e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf8c1014a0 0x7fcf8c0739a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:31.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.822+0000 7fcf8b59e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf8c101e60 0x7fcf8c073ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:31.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.822+0000 7fcf8b59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf8c107560 0x7fcf8c074420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:31.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.822+0000 7fcf8b59e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c111890 con 0x7fcf8c101e60 2024-03-28T05:28:31.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.822+0000 7fcf8b59e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c111710 con 0x7fcf8c107560 2024-03-28T05:28:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.822+0000 7fcf8b59e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c111a10 con 0x7fcf8c1014a0 2024-03-28T05:28:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.822+0000 7fcf89d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf8c101e60 0x7fcf8c073ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.822+0000 7fcf8a59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf8c1014a0 0x7fcf8c0739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.822+0000 7fcf89d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf8c101e60 0x7fcf8c073ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45850/0 (socket says 172.21.15.23:45850) 2024-03-28T05:28:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.822+0000 7fcf89d9b640 1 -- 172.21.15.23:0/1002641691 learned_addr learned my addr 172.21.15.23:0/1002641691 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.822+0000 7fcf8ad9d640 1 --2- 172.21.15.23:0/1002641691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf8c107560 0x7fcf8c074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.822+0000 7fcf8a59c640 1 -- 172.21.15.23:0/1002641691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf8c107560 msgr2=0x7fcf8c074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.822+0000 7fcf8a59c640 1 --2- 172.21.15.23:0/1002641691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf8c107560 0x7fcf8c074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.822+0000 7fcf8a59c640 1 -- 172.21.15.23:0/1002641691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf8c101e60 msgr2=0x7fcf8c073ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.822+0000 7fcf8a59c640 1 --2- 172.21.15.23:0/1002641691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf8c101e60 0x7fcf8c073ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.822+0000 7fcf8a59c640 1 -- 172.21.15.23:0/1002641691 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8c1a6190 con 0x7fcf8c1014a0 2024-03-28T05:28:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.822+0000 7fcf89d9b640 1 --2- 172.21.15.23:0/1002641691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf8c101e60 0x7fcf8c073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:28:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.822+0000 7fcf8a59c640 1 --2- 172.21.15.23:0/1002641691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf8c1014a0 0x7fcf8c0739a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c005fd0 tx=0x7fcf7c00dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.822+0000 7fcf7b7fe640 1 -- 172.21.15.23:0/1002641691 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c009c30 con 0x7fcf8c1014a0 2024-03-28T05:28:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.823+0000 7fcf8b59e640 1 -- 172.21.15.23:0/1002641691 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf8c1a6480 con 0x7fcf8c1014a0 2024-03-28T05:28:31.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.823+0000 7fcf7b7fe640 1 -- 172.21.15.23:0/1002641691 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf7c009dd0 con 0x7fcf8c1014a0 2024-03-28T05:28:31.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.823+0000 7fcf7b7fe640 1 -- 172.21.15.23:0/1002641691 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c010460 con 0x7fcf8c1014a0 2024-03-28T05:28:31.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.823+0000 7fcf8b59e640 1 -- 172.21.15.23:0/1002641691 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf8c1a6a20 con 0x7fcf8c1014a0 2024-03-28T05:28:31.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.825+0000 7fcf937f7740 1 Processor -- start 2024-03-28T05:28:31.830 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.825+0000 7fcf7b7fe640 1 -- 172.21.15.23:0/1002641691 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fcf7c003f50 con 0x7fcf8c1014a0 2024-03-28T05:28:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.825+0000 7fcf937f7740 1 -- start start 2024-03-28T05:28:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.825+0000 7fcf937f7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f47910 0x5624d9f47cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.825+0000 7fcf937f7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624d9f48230 0x5624d9f50ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.825+0000 7fcf937f7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f51000 0x5624d9f533f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.825+0000 7fcf937f7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624d9eabb10 con 0x5624d9f51000 2024-03-28T05:28:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.825+0000 7fcf937f7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624d9e54900 con 0x5624d9f47910 2024-03-28T05:28:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.825+0000 7fcf937f7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624d9ea7bb0 con 0x5624d9f48230 2024-03-28T05:28:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.826+0000 7fcf89d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f51000 0x5624d9f533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.826+0000 7fcf89d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f51000 0x5624d9f533f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45856/0 (socket says 172.21.15.23:45856) 2024-03-28T05:28:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.826+0000 7fcf8a59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624d9f48230 0x5624d9f50ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.826+0000 7fcf89d9b640 1 -- 172.21.15.23:0/793272383 learned_addr learned my addr 172.21.15.23:0/793272383 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.826+0000 7fcf8ad9d640 1 --2- 172.21.15.23:0/793272383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f47910 0x5624d9f47cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.826+0000 7fcf89d9b640 1 -- 172.21.15.23:0/793272383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624d9f48230 msgr2=0x5624d9f50ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:31.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.826+0000 7fcf89d9b640 1 --2- 172.21.15.23:0/793272383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624d9f48230 0x5624d9f50ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.826+0000 7fcf89d9b640 1 -- 172.21.15.23:0/793272383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f47910 msgr2=0x5624d9f47cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:31.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.826+0000 7fcf89d9b640 1 --2- 172.21.15.23:0/793272383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f47910 0x5624d9f47cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.826+0000 7fcf7b7fe640 1 --2- 172.21.15.23:0/1002641691 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fcf740a2960 0x7fcf740a4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:31.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.826+0000 7fcf89d9b640 1 -- 172.21.15.23:0/793272383 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624d9bc0210 con 0x5624d9f51000 2024-03-28T05:28:31.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.826+0000 7fcf8ad9d640 1 --2- 172.21.15.23:0/793272383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f47910 0x5624d9f47cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:28:31.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.826+0000 7fcf8ad9d640 1 --2- 172.21.15.23:0/1002641691 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fcf740a2960 0x7fcf740a4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:31.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.826+0000 7fcf89d9b640 1 --2- 172.21.15.23:0/793272383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f51000 0x5624d9f533f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fcf8002c240 tx=0x7fcf80005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6371cdcdfb73d93 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:31.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.826+0000 7fcf7b7fe640 1 -- 172.21.15.23:0/1002641691 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fcf7c0cf100 con 0x7fcf8c1014a0 2024-03-28T05:28:31.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.826+0000 7fcf78ff9640 1 -- 172.21.15.23:0/793272383 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf80042070 con 0x5624d9f51000 2024-03-28T05:28:31.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.826+0000 7fcf78ff9640 1 -- 172.21.15.23:0/793272383 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf80004c70 con 0x5624d9f51000 2024-03-28T05:28:31.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.826+0000 7fcf78ff9640 1 -- 172.21.15.23:0/793272383 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf80005340 con 0x5624d9f51000 2024-03-28T05:28:31.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.827+0000 7fcf937f7740 1 -- 172.21.15.23:0/793272383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f51000 msgr2=0x5624d9f533f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:31.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.827+0000 7fcf937f7740 1 --2- 172.21.15.23:0/793272383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f51000 0x5624d9f533f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fcf8002c240 tx=0x7fcf80005e60 comp rx=0 tx=0).stop 2024-03-28T05:28:31.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.827+0000 7fcf8ad9d640 1 --2- 172.21.15.23:0/1002641691 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fcf740a2960 0x7fcf740a4e20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcf84005fd0 tx=0x7fcf84009500 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:31.832 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.827+0000 7fcf937f7740 1 -- 172.21.15.23:0/793272383 shutdown_connections 2024-03-28T05:28:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.827+0000 7fcf937f7740 1 --2- 172.21.15.23:0/793272383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f51000 0x5624d9f533f0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.827+0000 7fcf937f7740 1 --2- 172.21.15.23:0/793272383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624d9f48230 0x5624d9f50ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.827+0000 7fcf937f7740 1 --2- 172.21.15.23:0/793272383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f47910 0x5624d9f47cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.827+0000 7fcf937f7740 1 -- 172.21.15.23:0/793272383 >> 172.21.15.23:0/793272383 conn(0x5624d9f42230 msgr2=0x5624d9f42610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.827+0000 7fcf937f7740 1 -- 172.21.15.23:0/793272383 shutdown_connections 2024-03-28T05:28:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.827+0000 7fcf937f7740 1 -- 172.21.15.23:0/793272383 wait complete. 2024-03-28T05:28:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.828+0000 7fcf937f7740 1 -- start start 2024-03-28T05:28:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.828+0000 7fcf937f7740 1 Processor -- start 2024-03-28T05:28:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.828+0000 7fcf937f7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f47910 0x5624d9f4e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.828+0000 7fcf937f7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f48230 0x5624d9f4e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.828+0000 7fcf937f7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624d9f51000 0x5624d9f4eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.828+0000 7fcf937f7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624d9f53bd0 con 0x5624d9f48230 2024-03-28T05:28:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.828+0000 7fcf937f7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624d9e67d50 con 0x5624d9f47910 2024-03-28T05:28:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.828+0000 7fcf937f7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624d9f53d50 con 0x5624d9f51000 2024-03-28T05:28:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.828+0000 7fcf937f7740 10 client.0 _ll_register_callbacks cb 0x5624d9e67720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:28:31.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.828+0000 7fcf89d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f48230 0x5624d9f4e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:31.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.828+0000 7fcf89d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f48230 0x5624d9f4e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45858/0 (socket says 172.21.15.23:45858) 2024-03-28T05:28:31.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.828+0000 7fcf89d9b640 1 -- 172.21.15.23:0/1599207243 learned_addr learned my addr 172.21.15.23:0/1599207243 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:31.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.828+0000 7fcf8ad9d640 1 --2- 172.21.15.23:0/1599207243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624d9f51000 0x5624d9f4eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:31.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.828+0000 7fcf8a59c640 1 --2- 172.21.15.23:0/1599207243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f47910 0x5624d9f4e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:31.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.828+0000 7fcf89d9b640 1 -- 172.21.15.23:0/1599207243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624d9f51000 msgr2=0x5624d9f4eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:31.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.828+0000 7fcf89d9b640 1 --2- 172.21.15.23:0/1599207243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624d9f51000 0x5624d9f4eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.828+0000 7fcf89d9b640 1 -- 172.21.15.23:0/1599207243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f47910 msgr2=0x5624d9f4e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:31.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.828+0000 7fcf89d9b640 1 --2- 172.21.15.23:0/1599207243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f47910 0x5624d9f4e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.828+0000 7fcf89d9b640 1 -- 172.21.15.23:0/1599207243 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5624d9c8c620 con 0x5624d9f48230 2024-03-28T05:28:31.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.828+0000 7fcf89d9b640 1 --2- 172.21.15.23:0/1599207243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f48230 0x5624d9f4e5a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fcf80000c00 tx=0x7fcf80009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dab1186fd6d82e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:31.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.828+0000 7fcf71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:28:31.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.829+0000 7fcf71ffb640 1 -- 172.21.15.23:0/1599207243 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf80042040 con 0x5624d9f48230 2024-03-28T05:28:31.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.829+0000 7fcf71ffb640 1 -- 172.21.15.23:0/1599207243 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fcf80047460 con 0x5624d9f48230 2024-03-28T05:28:31.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.829+0000 7fcf937f7740 1 -- 172.21.15.23:0/1599207243 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5624d9eb9050 con 0x5624d9f48230 2024-03-28T05:28:31.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.829+0000 7fcf71ffb640 1 -- 172.21.15.23:0/1599207243 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf8005a050 con 0x5624d9f48230 2024-03-28T05:28:31.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.829+0000 7fcf937f7740 10 client.15873 fetch_fsmap learned FSMap version 20 2024-03-28T05:28:31.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.829+0000 7fcf937f7740 1 -- 172.21.15.23:0/1599207243 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5624d9f5fe60 con 0x5624d9f48230 2024-03-28T05:28:31.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.829+0000 7fcf71ffb640 1 -- 172.21.15.23:0/1599207243 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5624d9eb9050 con 0x5624d9f48230 2024-03-28T05:28:31.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.829+0000 7fcf71ffb640 1 -- 172.21.15.23:0/1599207243 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fcf80034560 con 0x5624d9f48230 2024-03-28T05:28:31.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.829+0000 7fcf937f7740 10 client.15873 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:28:31.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.829+0000 7fcf937f7740 10 client.15873 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:28:31.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.829+0000 7fcf937f7740 20 client.15873 populate_metadata read hostname 'smithi023' 2024-03-28T05:28:31.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.829+0000 7fcf937f7740 1 --2- 172.21.15.23:0/1599207243 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5624d9f611b0 0x5624d9f635a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:31.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.829+0000 7fcf937f7740 4 client.15873 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:28:31.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.829+0000 7fcf937f7740 1 -- 172.21.15.23:0/1599207243 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5624d9eb33a0 con 0x5624d9f611b0 2024-03-28T05:28:31.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.829+0000 7fcf8a59c640 1 --2- 172.21.15.23:0/1599207243 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5624d9f611b0 0x5624d9f635a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:31.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.830+0000 7fcf71ffb640 10 client.15873 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:28:31.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.830+0000 7fcf8a59c640 1 --2- 172.21.15.23:0/1599207243 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5624d9f611b0 0x5624d9f635a0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27f04021fc449c18 server_cookie=c7ff73e40828898b in_seq=0 out_seq=0 2024-03-28T05:28:31.836 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.832+0000 7fcf71ffb640 1 -- 172.21.15.23:0/1599207243 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5624d9eb33a0 con 0x5624d9f611b0 2024-03-28T05:28:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.832+0000 7fcf71ffb640 10 client.15873 handle_command_reply: tid=0 2024-03-28T05:28:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.832+0000 7fcf937f7740 1 client.15873 shutdown 2024-03-28T05:28:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.832+0000 7fcf937f7740 1 -- 172.21.15.23:0/1599207243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f48230 msgr2=0x5624d9f4e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.832+0000 7fcf937f7740 1 --2- 172.21.15.23:0/1599207243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f48230 0x5624d9f4e5a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fcf80000c00 tx=0x7fcf80009260 comp rx=0 tx=0).stop 2024-03-28T05:28:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.832+0000 7fcf937f7740 1 -- 172.21.15.23:0/1599207243 shutdown_connections 2024-03-28T05:28:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.832+0000 7fcf937f7740 1 --2- 172.21.15.23:0/1599207243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624d9f51000 0x5624d9f4eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.832+0000 7fcf937f7740 1 --2- 172.21.15.23:0/1599207243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f48230 0x5624d9f4e5a0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.832+0000 7fcf937f7740 1 --2- 172.21.15.23:0/1599207243 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5624d9f611b0 0x5624d9f635a0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.832+0000 7fcf937f7740 1 --2- 172.21.15.23:0/1599207243 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f47910 0x5624d9f4e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.832+0000 7fcf71ffb640 0 client.15873 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:28:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.832+0000 7fcf937f7740 1 -- 172.21.15.23:0/1599207243 >> 172.21.15.23:0/1599207243 conn(0x5624d9f42230 msgr2=0x5624d9f43f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.832+0000 7fcf937f7740 1 -- 172.21.15.23:0/1599207243 shutdown_connections 2024-03-28T05:28:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.833+0000 7fcf937f7740 1 -- 172.21.15.23:0/1599207243 wait complete. 2024-03-28T05:28:31.837 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.833+0000 7fcf937f7740 20 client.15873 trim_cache size 0 max 16384 2024-03-28T05:28:31.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.847+0000 7fcf937f7740 1 Processor -- start 2024-03-28T05:28:31.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.847+0000 7fcf937f7740 1 -- start start 2024-03-28T05:28:31.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.847+0000 7fcf937f7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f47910 0x5624d9f50790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:31.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.847+0000 7fcf937f7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624d9f48230 0x5624d9f43d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:31.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.847+0000 7fcf937f7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f51000 0x5624d9f5c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:31.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.847+0000 7fcf937f7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c1ae940 con 0x5624d9f51000 2024-03-28T05:28:31.848 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.847+0000 7fcf937f7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c1b0f00 con 0x5624d9f47910 2024-03-28T05:28:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.847+0000 7fcf937f7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c1aef40 con 0x5624d9f48230 2024-03-28T05:28:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.847+0000 7fcf8ad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f51000 0x5624d9f5c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.847+0000 7fcf8ad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f51000 0x5624d9f5c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45862/0 (socket says 172.21.15.23:45862) 2024-03-28T05:28:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.847+0000 7fcf89d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624d9f48230 0x5624d9f43d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.847+0000 7fcf8ad9d640 1 -- 172.21.15.23:0/3382220655 learned_addr learned my addr 172.21.15.23:0/3382220655 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.847+0000 7fcf8a59c640 1 --2- 172.21.15.23:0/3382220655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f47910 0x5624d9f50790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.847+0000 7fcf8ad9d640 1 -- 172.21.15.23:0/3382220655 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624d9f48230 msgr2=0x5624d9f43d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.847+0000 7fcf8ad9d640 1 --2- 172.21.15.23:0/3382220655 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624d9f48230 0x5624d9f43d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.847+0000 7fcf8ad9d640 1 -- 172.21.15.23:0/3382220655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f47910 msgr2=0x5624d9f50790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.847+0000 7fcf8ad9d640 1 --2- 172.21.15.23:0/3382220655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f47910 0x5624d9f50790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.847+0000 7fcf8ad9d640 1 -- 172.21.15.23:0/3382220655 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624d9f48020 con 0x5624d9f51000 2024-03-28T05:28:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.848+0000 7fcf8a59c640 1 --2- 172.21.15.23:0/3382220655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f47910 0x5624d9f50790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:28:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.848+0000 7fcf8ad9d640 1 --2- 172.21.15.23:0/3382220655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f51000 0x5624d9f5c460 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fcf840106b0 tx=0x7fcf84015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5f2c400b27ca7a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.848+0000 7fcf73fff640 1 -- 172.21.15.23:0/3382220655 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf84010ea0 con 0x5624d9f51000 2024-03-28T05:28:31.849 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.848+0000 7fcf73fff640 1 -- 172.21.15.23:0/3382220655 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf84028b60 con 0x5624d9f51000 2024-03-28T05:28:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.848+0000 7fcf73fff640 1 -- 172.21.15.23:0/3382220655 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8401cd30 con 0x5624d9f51000 2024-03-28T05:28:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.848+0000 7fcf937f7740 1 -- 172.21.15.23:0/3382220655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f51000 msgr2=0x5624d9f5c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.848+0000 7fcf937f7740 1 --2- 172.21.15.23:0/3382220655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f51000 0x5624d9f5c460 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fcf840106b0 tx=0x7fcf84015000 comp rx=0 tx=0).stop 2024-03-28T05:28:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.849+0000 7fcf937f7740 1 -- 172.21.15.23:0/3382220655 shutdown_connections 2024-03-28T05:28:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.849+0000 7fcf937f7740 1 --2- 172.21.15.23:0/3382220655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f51000 0x5624d9f5c460 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.849+0000 7fcf937f7740 1 --2- 172.21.15.23:0/3382220655 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624d9f48230 0x5624d9f43d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.849+0000 7fcf937f7740 1 --2- 172.21.15.23:0/3382220655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f47910 0x5624d9f50790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.849+0000 7fcf937f7740 1 -- 172.21.15.23:0/3382220655 >> 172.21.15.23:0/3382220655 conn(0x5624d9f42230 msgr2=0x5624d9f43630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.849+0000 7fcf937f7740 1 -- 172.21.15.23:0/3382220655 shutdown_connections 2024-03-28T05:28:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.849+0000 7fcf937f7740 1 -- 172.21.15.23:0/3382220655 wait complete. 2024-03-28T05:28:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.849+0000 7fcf937f7740 1 -- start start 2024-03-28T05:28:31.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.849+0000 7fcf937f7740 1 Processor -- start 2024-03-28T05:28:31.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.849+0000 7fcf937f7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f47910 0x5624d9fbd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:31.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.849+0000 7fcf937f7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624d9f48230 0x5624d9fbd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:31.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.849+0000 7fcf937f7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f51000 0x5624d9f54c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:31.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.849+0000 7fcf937f7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c1af580 con 0x5624d9f47910 2024-03-28T05:28:31.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.849+0000 7fcf937f7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c1b08a0 con 0x5624d9f51000 2024-03-28T05:28:31.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.849+0000 7fcf937f7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c1afbe0 con 0x5624d9f48230 2024-03-28T05:28:31.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.849+0000 7fcf937f7740 10 client.0 _ll_register_callbacks cb 0x7fcf80009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:28:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.849+0000 7fcf8a59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f47910 0x5624d9fbd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.849+0000 7fcf8a59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f47910 0x5624d9fbd2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45866/0 (socket says 172.21.15.23:45866) 2024-03-28T05:28:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.849+0000 7fcf8a59c640 1 -- 172.21.15.23:0/3913333415 learned_addr learned my addr 172.21.15.23:0/3913333415 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.849+0000 7fcf89d9b640 1 --2- 172.21.15.23:0/3913333415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624d9f48230 0x5624d9fbd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.850+0000 7fcf8ad9d640 1 --2- 172.21.15.23:0/3913333415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f51000 0x5624d9f54c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.850+0000 7fcf8a59c640 1 -- 172.21.15.23:0/3913333415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624d9f48230 msgr2=0x5624d9fbd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.850+0000 7fcf8a59c640 1 --2- 172.21.15.23:0/3913333415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624d9f48230 0x5624d9fbd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.850+0000 7fcf8a59c640 1 -- 172.21.15.23:0/3913333415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f51000 msgr2=0x5624d9f54c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.850+0000 7fcf8a59c640 1 --2- 172.21.15.23:0/3913333415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f51000 0x5624d9f54c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.850+0000 7fcf8a59c640 1 -- 172.21.15.23:0/3913333415 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5624d9f55430 con 0x5624d9f47910 2024-03-28T05:28:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.850+0000 7fcf8ad9d640 1 --2- 172.21.15.23:0/3913333415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f51000 0x5624d9f54c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:28:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.850+0000 7fcf8a59c640 1 --2- 172.21.15.23:0/3913333415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f47910 0x5624d9fbd2f0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c09b0d0 tx=0x7fcf7c0d2c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7e09adef630a172 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.850+0000 7fcf71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:28:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.850+0000 7fcf71ffb640 1 -- 172.21.15.23:0/3913333415 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c020040 con 0x5624d9f47910 2024-03-28T05:28:31.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.850+0000 7fcf71ffb640 1 -- 172.21.15.23:0/3913333415 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fcf7c0d2d00 con 0x5624d9f47910 2024-03-28T05:28:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.850+0000 7fcf937f7740 1 -- 172.21.15.23:0/3913333415 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf84028b60 con 0x5624d9f47910 2024-03-28T05:28:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.850+0000 7fcf71ffb640 1 -- 172.21.15.23:0/3913333415 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf7c09ad30 con 0x5624d9f47910 2024-03-28T05:28:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.850+0000 7fcf937f7740 10 client.15879 fetch_fsmap learned FSMap version 20 2024-03-28T05:28:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.850+0000 7fcf937f7740 1 -- 172.21.15.23:0/3913333415 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5624d9fc5bf0 con 0x5624d9f47910 2024-03-28T05:28:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.850+0000 7fcf71ffb640 1 -- 172.21.15.23:0/3913333415 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf84028b60 con 0x5624d9f47910 2024-03-28T05:28:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.850+0000 7fcf71ffb640 1 -- 172.21.15.23:0/3913333415 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fcf7c0cc740 con 0x5624d9f47910 2024-03-28T05:28:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.850+0000 7fcf937f7740 10 client.15879 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:28:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.850+0000 7fcf937f7740 10 client.15879 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:28:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.850+0000 7fcf937f7740 20 client.15879 populate_metadata read hostname 'smithi023' 2024-03-28T05:28:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.851+0000 7fcf937f7740 1 --2- 172.21.15.23:0/3913333415 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5624d9fc62c0 0x5624d9fc86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.851+0000 7fcf937f7740 4 client.15879 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:28:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.851+0000 7fcf937f7740 1 -- 172.21.15.23:0/3913333415 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5624d9e31710 con 0x5624d9fc62c0 2024-03-28T05:28:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.851+0000 7fcf89d9b640 1 --2- 172.21.15.23:0/3913333415 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5624d9fc62c0 0x5624d9fc86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.851+0000 7fcf71ffb640 10 client.15879 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:28:31.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:31.851+0000 7fcf89d9b640 1 --2- 172.21.15.23:0/3913333415 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5624d9fc62c0 0x5624d9fc86e0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9b19b952ff79772 server_cookie=a9bacf662823e0c6 in_seq=0 out_seq=0 2024-03-28T05:28:32.316 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.316+0000 7fcf71ffb640 1 -- 172.21.15.23:0/3913333415 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5624d9e31710 con 0x5624d9fc62c0 2024-03-28T05:28:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.316+0000 7fcf71ffb640 10 client.15879 handle_command_reply: tid=0 2024-03-28T05:28:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.316+0000 7fcf937f7740 1 client.15879 shutdown 2024-03-28T05:28:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.317+0000 7fcf937f7740 1 -- 172.21.15.23:0/3913333415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f47910 msgr2=0x5624d9fbd2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.317+0000 7fcf937f7740 1 --2- 172.21.15.23:0/3913333415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f47910 0x5624d9fbd2f0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c09b0d0 tx=0x7fcf7c0d2c50 comp rx=0 tx=0).stop 2024-03-28T05:28:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.317+0000 7fcf937f7740 1 -- 172.21.15.23:0/3913333415 shutdown_connections 2024-03-28T05:28:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.317+0000 7fcf937f7740 1 --2- 172.21.15.23:0/3913333415 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5624d9fc62c0 0x5624d9fc86e0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.317+0000 7fcf937f7740 1 --2- 172.21.15.23:0/3913333415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5624d9f51000 0x5624d9f54c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.317+0000 7fcf937f7740 1 --2- 172.21.15.23:0/3913333415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624d9f48230 0x5624d9fbd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.317+0000 7fcf937f7740 1 --2- 172.21.15.23:0/3913333415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5624d9f47910 0x5624d9fbd2f0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.317+0000 7fcf937f7740 1 -- 172.21.15.23:0/3913333415 >> 172.21.15.23:0/3913333415 conn(0x5624d9f42230 msgr2=0x5624d9f4e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.317+0000 7fcf71ffb640 0 client.15879 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:28:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.317+0000 7fcf937f7740 1 -- 172.21.15.23:0/3913333415 shutdown_connections 2024-03-28T05:28:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.317+0000 7fcf937f7740 1 -- 172.21.15.23:0/3913333415 wait complete. 2024-03-28T05:28:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.317+0000 7fcf937f7740 20 client.15879 trim_cache size 0 max 16384 2024-03-28T05:28:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:28:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:28:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:28:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:28:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.318+0000 7fcf8b59e640 1 -- 172.21.15.23:0/1002641691 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fcf740a2960 msgr2=0x7fcf740a4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.318+0000 7fcf8b59e640 1 --2- 172.21.15.23:0/1002641691 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fcf740a2960 0x7fcf740a4e20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcf84005fd0 tx=0x7fcf84009500 comp rx=0 tx=0).stop 2024-03-28T05:28:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.318+0000 7fcf8b59e640 1 -- 172.21.15.23:0/1002641691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf8c1014a0 msgr2=0x7fcf8c0739a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.318+0000 7fcf8b59e640 1 --2- 172.21.15.23:0/1002641691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf8c1014a0 0x7fcf8c0739a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c005fd0 tx=0x7fcf7c00dde0 comp rx=0 tx=0).stop 2024-03-28T05:28:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.318+0000 7fcf8b59e640 1 -- 172.21.15.23:0/1002641691 shutdown_connections 2024-03-28T05:28:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.318+0000 7fcf8b59e640 1 --2- 172.21.15.23:0/1002641691 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fcf740a2960 0x7fcf740a4e20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.318+0000 7fcf8b59e640 1 --2- 172.21.15.23:0/1002641691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf8c107560 0x7fcf8c074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.318+0000 7fcf8b59e640 1 --2- 172.21.15.23:0/1002641691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf8c101e60 0x7fcf8c073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:32.320 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.318+0000 7fcf8b59e640 1 --2- 172.21.15.23:0/1002641691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcf8c1014a0 0x7fcf8c0739a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:32.320 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.318+0000 7fcf8b59e640 1 -- 172.21.15.23:0/1002641691 >> 172.21.15.23:0/1002641691 conn(0x7fcf8c0fcff0 msgr2=0x7fcf8c0fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:32.320 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.318+0000 7fcf8b59e640 1 -- 172.21.15.23:0/1002641691 shutdown_connections 2024-03-28T05:28:32.320 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:32.318+0000 7fcf8b59e640 1 -- 172.21.15.23:0/1002641691 wait complete. 2024-03-28T05:28:33.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:33 smithi113 ceph-mon[37164]: pgmap v152: 129 pgs: 129 active+clean; 174 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 2.1 MiB/s wr, 412 op/s 2024-03-28T05:28:33.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:33 smithi023 ceph-mon[31954]: pgmap v152: 129 pgs: 129 active+clean; 174 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 2.1 MiB/s wr, 412 op/s 2024-03-28T05:28:33.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:33 smithi098 ceph-mon[36087]: pgmap v152: 129 pgs: 129 active+clean; 174 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 2.1 MiB/s wr, 412 op/s 2024-03-28T05:28:35.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:35 smithi113 ceph-mon[37164]: pgmap v153: 129 pgs: 129 active+clean; 174 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 2.0 MiB/s wr, 178 op/s 2024-03-28T05:28:35.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:35 smithi023 ceph-mon[31954]: pgmap v153: 129 pgs: 129 active+clean; 174 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 2.0 MiB/s wr, 178 op/s 2024-03-28T05:28:35.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:35 smithi098 ceph-mon[36087]: pgmap v153: 129 pgs: 129 active+clean; 174 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 2.0 MiB/s wr, 178 op/s 2024-03-28T05:28:36.400 INFO:tasks.workunit.client.0.smithi023.stdout: 50 222581 106 152985 51 106239 203 2024-03-28T05:28:37.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:37 smithi113 ceph-mon[37164]: pgmap v154: 129 pgs: 129 active+clean; 170 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 2.5 MiB/s wr, 443 op/s 2024-03-28T05:28:37.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:37 smithi023 ceph-mon[31954]: pgmap v154: 129 pgs: 129 active+clean; 170 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 2.5 MiB/s wr, 443 op/s 2024-03-28T05:28:37.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:37 smithi098 ceph-mon[36087]: pgmap v154: 129 pgs: 129 active+clean; 170 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 2.5 MiB/s wr, 443 op/s 2024-03-28T05:28:39.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:39 smithi113 ceph-mon[37164]: pgmap v155: 129 pgs: 129 active+clean; 170 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 2.2 MiB/s wr, 402 op/s 2024-03-28T05:28:39.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:39 smithi023 ceph-mon[31954]: pgmap v155: 129 pgs: 129 active+clean; 170 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 2.2 MiB/s wr, 402 op/s 2024-03-28T05:28:39.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:39 smithi098 ceph-mon[36087]: pgmap v155: 129 pgs: 129 active+clean; 170 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 2.2 MiB/s wr, 402 op/s 2024-03-28T05:28:41.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:41 smithi113 ceph-mon[37164]: pgmap v156: 129 pgs: 129 active+clean; 172 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 2.4 MiB/s wr, 417 op/s 2024-03-28T05:28:41.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:41 smithi023 ceph-mon[31954]: pgmap v156: 129 pgs: 129 active+clean; 172 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 2.4 MiB/s wr, 417 op/s 2024-03-28T05:28:41.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:41 smithi098 ceph-mon[36087]: pgmap v156: 129 pgs: 129 active+clean; 172 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 2.4 MiB/s wr, 417 op/s 2024-03-28T05:28:43.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:43 smithi113 ceph-mon[37164]: pgmap v157: 129 pgs: 129 active+clean; 172 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 2.0 MiB/s wr, 408 op/s 2024-03-28T05:28:43.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:43 smithi023 ceph-mon[31954]: pgmap v157: 129 pgs: 129 active+clean; 172 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 2.0 MiB/s wr, 408 op/s 2024-03-28T05:28:43.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:43 smithi098 ceph-mon[36087]: pgmap v157: 129 pgs: 129 active+clean; 172 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 2.0 MiB/s wr, 408 op/s 2024-03-28T05:28:44.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:44 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:28:44.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:44 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:28:44.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:44 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:28:45.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.415+0000 7fc0baf77640 1 -- 172.21.15.23:0/944167902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0b41011b0 msgr2=0x7fc0b410d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:45.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.415+0000 7fc0baf77640 1 --2- 172.21.15.23:0/944167902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0b41011b0 0x7fc0b410d3d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac009ca0 tx=0x7fc0ac02c0f0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.415+0000 7fc0baf77640 1 -- 172.21.15.23:0/944167902 shutdown_connections 2024-03-28T05:28:45.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.415+0000 7fc0baf77640 1 --2- 172.21.15.23:0/944167902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0b41011b0 0x7fc0b410d3d0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.415+0000 7fc0baf77640 1 --2- 172.21.15.23:0/944167902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0b41007f0 0x7fc0b4100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.415+0000 7fc0baf77640 1 --2- 172.21.15.23:0/944167902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0b4105540 0x7fc0b4105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.415+0000 7fc0baf77640 1 -- 172.21.15.23:0/944167902 >> 172.21.15.23:0/944167902 conn(0x7fc0b40773f0 msgr2=0x7fc0b4077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:45.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.415+0000 7fc0baf77640 1 -- 172.21.15.23:0/944167902 shutdown_connections 2024-03-28T05:28:45.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.415+0000 7fc0baf77640 1 -- 172.21.15.23:0/944167902 wait complete. 2024-03-28T05:28:45.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.416+0000 7fc0baf77640 1 Processor -- start 2024-03-28T05:28:45.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.416+0000 7fc0baf77640 1 -- start start 2024-03-28T05:28:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.416+0000 7fc0baf77640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0b41007f0 0x7fc0b4198de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.416+0000 7fc0baf77640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0b41011b0 0x7fc0b4199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.416+0000 7fc0baf77640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0b4105540 0x7fc0b41a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.416+0000 7fc0baf77640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b4110a60 con 0x7fc0b41007f0 2024-03-28T05:28:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.416+0000 7fc0baf77640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b41108e0 con 0x7fc0b41011b0 2024-03-28T05:28:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.416+0000 7fc0baf77640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b4110be0 con 0x7fc0b4105540 2024-03-28T05:28:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.416+0000 7fc0b8d13640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0b41007f0 0x7fc0b4198de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.416+0000 7fc0b8d13640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0b41007f0 0x7fc0b4198de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41348/0 (socket says 172.21.15.23:41348) 2024-03-28T05:28:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.416+0000 7fc0b8d13640 1 -- 172.21.15.23:0/565403724 learned_addr learned my addr 172.21.15.23:0/565403724 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.416+0000 7fc0b9514640 1 --2- 172.21.15.23:0/565403724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0b4105540 0x7fc0b41a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.416+0000 7fc0b3fff640 1 --2- 172.21.15.23:0/565403724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0b41011b0 0x7fc0b4199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.416+0000 7fc0b8d13640 1 -- 172.21.15.23:0/565403724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0b4105540 msgr2=0x7fc0b41a3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:45.418 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.416+0000 7fc0b8d13640 1 --2- 172.21.15.23:0/565403724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0b4105540 0x7fc0b41a3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.416+0000 7fc0b8d13640 1 -- 172.21.15.23:0/565403724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0b41011b0 msgr2=0x7fc0b4199320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.416+0000 7fc0b8d13640 1 --2- 172.21.15.23:0/565403724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0b41011b0 0x7fc0b4199320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.416+0000 7fc0b8d13640 1 -- 172.21.15.23:0/565403724 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b41a43a0 con 0x7fc0b41007f0 2024-03-28T05:28:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.416+0000 7fc0b8d13640 1 --2- 172.21.15.23:0/565403724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0b41007f0 0x7fc0b4198de0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc0a800e9f0 tx=0x7fc0a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.417+0000 7fc0b1ffb640 1 -- 172.21.15.23:0/565403724 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a800cd80 con 0x7fc0b41007f0 2024-03-28T05:28:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.417+0000 7fc0b1ffb640 1 -- 172.21.15.23:0/565403724 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0a8010910 con 0x7fc0b41007f0 2024-03-28T05:28:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.417+0000 7fc0b1ffb640 1 -- 172.21.15.23:0/565403724 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a8018990 con 0x7fc0b41007f0 2024-03-28T05:28:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.417+0000 7fc0baf77640 1 -- 172.21.15.23:0/565403724 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b41a4570 con 0x7fc0b41007f0 2024-03-28T05:28:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.417+0000 7fc0baf77640 1 -- 172.21.15.23:0/565403724 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b41a4a40 con 0x7fc0b41007f0 2024-03-28T05:28:45.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.419+0000 7fc0b1ffb640 1 -- 172.21.15.23:0/565403724 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fc0a8010ac0 con 0x7fc0b41007f0 2024-03-28T05:28:45.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.420+0000 7fc0bdb64740 1 Processor -- start 2024-03-28T05:28:45.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.420+0000 7fc0bdb64740 1 -- start start 2024-03-28T05:28:45.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.420+0000 7fc0b1ffb640 1 --2- 172.21.15.23:0/565403724 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fc0a40a2a10 0x7fc0a40a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:45.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.420+0000 7fc0bdb64740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012c710 0x55cda012caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:45.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.420+0000 7fc0bdb64740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda012d030 0x55cda01358c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:45.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.420+0000 7fc0bdb64740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cda0135e00 0x55cda01381f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:45.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.420+0000 7fc0bdb64740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda0090b10 con 0x55cda012c710 2024-03-28T05:28:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.420+0000 7fc0bdb64740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda0039900 con 0x55cda0135e00 2024-03-28T05:28:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.420+0000 7fc0bdb64740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda008cbb0 con 0x55cda012d030 2024-03-28T05:28:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.420+0000 7fc0b9514640 1 --2- 172.21.15.23:0/565403724 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fc0a40a2a10 0x7fc0a40a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.420+0000 7fc0b1ffb640 1 -- 172.21.15.23:0/565403724 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fc0a80cf4b0 con 0x7fc0b41007f0 2024-03-28T05:28:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.420+0000 7fc0b8d13640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012c710 0x55cda012caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.420+0000 7fc0b8d13640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012c710 0x55cda012caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41358/0 (socket says 172.21.15.23:41358) 2024-03-28T05:28:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.420+0000 7fc0b8d13640 1 -- 172.21.15.23:0/1863923104 learned_addr learned my addr 172.21.15.23:0/1863923104 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.420+0000 7fc0b9514640 1 --2- 172.21.15.23:0/1863923104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cda0135e00 0x55cda01381f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.420+0000 7fc0b3fff640 1 --2- 172.21.15.23:0/1863923104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda012d030 0x55cda01358c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.421+0000 7fc0b9514640 1 -- 172.21.15.23:0/1863923104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda012d030 msgr2=0x55cda01358c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.421+0000 7fc0b9514640 1 --2- 172.21.15.23:0/1863923104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda012d030 0x55cda01358c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.421+0000 7fc0b9514640 1 -- 172.21.15.23:0/1863923104 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012c710 msgr2=0x55cda012caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.421+0000 7fc0b9514640 1 --2- 172.21.15.23:0/1863923104 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012c710 0x55cda012caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.421+0000 7fc0b9514640 1 -- 172.21.15.23:0/1863923104 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd9fda5210 con 0x55cda0135e00 2024-03-28T05:28:45.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.421+0000 7fc0b8d13640 1 --2- 172.21.15.23:0/1863923104 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012c710 0x55cda012caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:28:45.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.421+0000 7fc0b9514640 1 --2- 172.21.15.23:0/565403724 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fc0a40a2a10 0x7fc0a40a4ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac004af0 tx=0x7fc0ac032e90 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:45.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.421+0000 7fc0b9514640 1 --2- 172.21.15.23:0/1863923104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cda0135e00 0x55cda01381f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac02c700 tx=0x7fc0ac049000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=147f336f041701c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:45.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.421+0000 7fc09effd640 1 -- 172.21.15.23:0/1863923104 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac039040 con 0x55cda0135e00 2024-03-28T05:28:45.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.421+0000 7fc09effd640 1 -- 172.21.15.23:0/1863923104 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0ac052070 con 0x55cda0135e00 2024-03-28T05:28:45.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.421+0000 7fc09effd640 1 -- 172.21.15.23:0/1863923104 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac04d6e0 con 0x55cda0135e00 2024-03-28T05:28:45.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.422+0000 7fc0bdb64740 1 -- 172.21.15.23:0/1863923104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cda0135e00 msgr2=0x55cda01381f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:45.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.422+0000 7fc0bdb64740 1 --2- 172.21.15.23:0/1863923104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cda0135e00 0x55cda01381f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac02c700 tx=0x7fc0ac049000 comp rx=0 tx=0).stop 2024-03-28T05:28:45.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.425+0000 7fc0bdb64740 1 -- 172.21.15.23:0/1863923104 shutdown_connections 2024-03-28T05:28:45.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.425+0000 7fc0bdb64740 1 --2- 172.21.15.23:0/1863923104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cda0135e00 0x55cda01381f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.425+0000 7fc0bdb64740 1 --2- 172.21.15.23:0/1863923104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda012d030 0x55cda01358c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.425+0000 7fc0bdb64740 1 --2- 172.21.15.23:0/1863923104 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012c710 0x55cda012caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.425+0000 7fc0bdb64740 1 -- 172.21.15.23:0/1863923104 >> 172.21.15.23:0/1863923104 conn(0x55cda0127030 msgr2=0x55cda0127410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.433+0000 7fc0bdb64740 1 -- 172.21.15.23:0/1863923104 shutdown_connections 2024-03-28T05:28:45.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.433+0000 7fc0bdb64740 1 -- 172.21.15.23:0/1863923104 wait complete. 2024-03-28T05:28:45.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.433+0000 7fc0bdb64740 1 -- start start 2024-03-28T05:28:45.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.433+0000 7fc0bdb64740 1 Processor -- start 2024-03-28T05:28:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.434+0000 7fc0bdb64740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012c710 0x55cda0132ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.434+0000 7fc0bdb64740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cda012d030 0x55cda01333e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.434+0000 7fc0bdb64740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda0135e00 0x55cda0133920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.434+0000 7fc0bdb64740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda013cde0 con 0x55cda012c710 2024-03-28T05:28:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.434+0000 7fc0bdb64740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda004cd50 con 0x55cda012d030 2024-03-28T05:28:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.434+0000 7fc0bdb64740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda013cf60 con 0x55cda0135e00 2024-03-28T05:28:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.434+0000 7fc0bdb64740 10 client.0 _ll_register_callbacks cb 0x55cda004c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:28:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.434+0000 7fc0b9514640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda0135e00 0x55cda0133920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.434+0000 7fc0b9514640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda0135e00 0x55cda0133920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:55150/0 (socket says 172.21.15.23:55150) 2024-03-28T05:28:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.434+0000 7fc0b9514640 1 -- 172.21.15.23:0/548572922 learned_addr learned my addr 172.21.15.23:0/548572922 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.435+0000 7fc0b3fff640 1 --2- 172.21.15.23:0/548572922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cda012d030 0x55cda01333e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.435+0000 7fc0b8d13640 1 --2- 172.21.15.23:0/548572922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012c710 0x55cda0132ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.435+0000 7fc0b9514640 1 -- 172.21.15.23:0/548572922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cda012d030 msgr2=0x55cda01333e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.435+0000 7fc0b9514640 1 --2- 172.21.15.23:0/548572922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cda012d030 0x55cda01333e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.435+0000 7fc0b9514640 1 -- 172.21.15.23:0/548572922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012c710 msgr2=0x55cda0132ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.435+0000 7fc0b9514640 1 --2- 172.21.15.23:0/548572922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012c710 0x55cda0132ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.435+0000 7fc0b9514640 1 -- 172.21.15.23:0/548572922 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cd9fe71620 con 0x55cda0135e00 2024-03-28T05:28:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.435+0000 7fc0b8d13640 1 --2- 172.21.15.23:0/548572922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012c710 0x55cda0132ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:28:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.435+0000 7fc0b3fff640 1 --2- 172.21.15.23:0/548572922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cda012d030 0x55cda01333e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:28:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.435+0000 7fc0b9514640 1 --2- 172.21.15.23:0/548572922 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda0135e00 0x55cda0133920 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac009a60 tx=0x7fc0ac04d830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40470e5970e2f941 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.435+0000 7fc093fff640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-28T05:28:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.435+0000 7fc093fff640 1 -- 172.21.15.23:0/548572922 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac003e40 con 0x55cda0135e00 2024-03-28T05:28:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.435+0000 7fc093fff640 1 -- 172.21.15.23:0/548572922 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fc0ac04d9e0 con 0x55cda0135e00 2024-03-28T05:28:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.435+0000 7fc0bdb64740 1 -- 172.21.15.23:0/548572922 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc0ac052070 con 0x55cda0135e00 2024-03-28T05:28:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.436+0000 7fc093fff640 1 -- 172.21.15.23:0/548572922 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0ac04dbe0 con 0x55cda0135e00 2024-03-28T05:28:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.436+0000 7fc0bdb64740 10 client.25723 fetch_fsmap learned FSMap version 20 2024-03-28T05:28:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.436+0000 7fc0bdb64740 1 -- 172.21.15.23:0/548572922 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55cda013cde0 con 0x55cda0135e00 2024-03-28T05:28:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.436+0000 7fc093fff640 1 -- 172.21.15.23:0/548572922 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0ac052070 con 0x55cda0135e00 2024-03-28T05:28:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.436+0000 7fc093fff640 1 -- 172.21.15.23:0/548572922 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fc0ac03f050 con 0x55cda0135e00 2024-03-28T05:28:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.436+0000 7fc0bdb64740 10 client.25723 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:28:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.436+0000 7fc0bdb64740 10 client.25723 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:28:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.436+0000 7fc0bdb64740 20 client.25723 populate_metadata read hostname 'smithi023' 2024-03-28T05:28:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.436+0000 7fc0bdb64740 1 --2- 172.21.15.23:0/548572922 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55cda014a260 0x55cda014c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.436+0000 7fc0bdb64740 4 client.25723 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:28:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.436+0000 7fc0bdb64740 1 -- 172.21.15.23:0/548572922 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cda009e050 con 0x55cda014a260 2024-03-28T05:28:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.437+0000 7fc0b8d13640 1 --2- 172.21.15.23:0/548572922 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55cda014a260 0x55cda014c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.437+0000 7fc093fff640 10 client.25723 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:28:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.437+0000 7fc0b8d13640 1 --2- 172.21.15.23:0/548572922 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55cda014a260 0x55cda014c650 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d7b4e365eac4836 server_cookie=70d944da4d0b72a9 in_seq=0 out_seq=0 2024-03-28T05:28:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.438+0000 7fc093fff640 1 -- 172.21.15.23:0/548572922 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cda009e050 con 0x55cda014a260 2024-03-28T05:28:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.438+0000 7fc093fff640 10 client.25723 handle_command_reply: tid=0 2024-03-28T05:28:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.438+0000 7fc0bdb64740 1 client.25723 shutdown 2024-03-28T05:28:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.439+0000 7fc0bdb64740 1 -- 172.21.15.23:0/548572922 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda0135e00 msgr2=0x55cda0133920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.439+0000 7fc0bdb64740 1 --2- 172.21.15.23:0/548572922 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda0135e00 0x55cda0133920 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac009a60 tx=0x7fc0ac04d830 comp rx=0 tx=0).stop 2024-03-28T05:28:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.439+0000 7fc0bdb64740 1 -- 172.21.15.23:0/548572922 shutdown_connections 2024-03-28T05:28:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.439+0000 7fc0bdb64740 1 --2- 172.21.15.23:0/548572922 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda0135e00 0x55cda0133920 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.439+0000 7fc0bdb64740 1 --2- 172.21.15.23:0/548572922 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55cda014a260 0x55cda014c650 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.439+0000 7fc0bdb64740 1 --2- 172.21.15.23:0/548572922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cda012d030 0x55cda01333e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.439+0000 7fc0bdb64740 1 --2- 172.21.15.23:0/548572922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012c710 0x55cda0132ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.439+0000 7fc093fff640 0 client.25723 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:28:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.439+0000 7fc0bdb64740 1 -- 172.21.15.23:0/548572922 >> 172.21.15.23:0/548572922 conn(0x55cda0127030 msgr2=0x55cda0128d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.439+0000 7fc0bdb64740 1 -- 172.21.15.23:0/548572922 shutdown_connections 2024-03-28T05:28:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.439+0000 7fc0bdb64740 1 -- 172.21.15.23:0/548572922 wait complete. 2024-03-28T05:28:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.439+0000 7fc0bdb64740 20 client.25723 trim_cache size 0 max 16384 2024-03-28T05:28:45.456 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.454+0000 7fc0bdb64740 1 Processor -- start 2024-03-28T05:28:45.456 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.454+0000 7fc0bdb64740 1 -- start start 2024-03-28T05:28:45.456 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.455+0000 7fc0bdb64740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cda012c710 0x55cda01355d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:45.456 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.455+0000 7fc0bdb64740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012d030 0x55cda0133740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:45.456 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.455+0000 7fc0bdb64740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda0135e00 0x55cda0145670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:45.456 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.455+0000 7fc0bdb64740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b41aa870 con 0x55cda012d030 2024-03-28T05:28:45.456 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.455+0000 7fc0bdb64740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b41ace30 con 0x55cda012c710 2024-03-28T05:28:45.456 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.455+0000 7fc0bdb64740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b41aae70 con 0x55cda0135e00 2024-03-28T05:28:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.455+0000 7fc0b3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012d030 0x55cda0133740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.455+0000 7fc0b9514640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda0135e00 0x55cda0145670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.455+0000 7fc0b3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012d030 0x55cda0133740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41372/0 (socket says 172.21.15.23:41372) 2024-03-28T05:28:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.455+0000 7fc0b3fff640 1 -- 172.21.15.23:0/3644825693 learned_addr learned my addr 172.21.15.23:0/3644825693 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.455+0000 7fc0b8d13640 1 --2- 172.21.15.23:0/3644825693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cda012c710 0x55cda01355d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.455+0000 7fc0b3fff640 1 -- 172.21.15.23:0/3644825693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda0135e00 msgr2=0x55cda0145670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.455+0000 7fc0b3fff640 1 --2- 172.21.15.23:0/3644825693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda0135e00 0x55cda0145670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.455+0000 7fc0b3fff640 1 -- 172.21.15.23:0/3644825693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cda012c710 msgr2=0x55cda01355d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.455+0000 7fc0b3fff640 1 --2- 172.21.15.23:0/3644825693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cda012c710 0x55cda01355d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.455+0000 7fc0b3fff640 1 -- 172.21.15.23:0/3644825693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cda0131de0 con 0x55cda012d030 2024-03-28T05:28:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.455+0000 7fc0b9514640 1 --2- 172.21.15.23:0/3644825693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda0135e00 0x55cda0145670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:28:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.455+0000 7fc0b3fff640 1 --2- 172.21.15.23:0/3644825693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012d030 0x55cda0133740 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0009c60 tx=0x7fc0a0010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d897cb5dc8f551d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.456+0000 7fc09e7fc640 1 -- 172.21.15.23:0/3644825693 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a0021070 con 0x55cda012d030 2024-03-28T05:28:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.456+0000 7fc09e7fc640 1 -- 172.21.15.23:0/3644825693 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0a0009e10 con 0x55cda012d030 2024-03-28T05:28:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.456+0000 7fc09e7fc640 1 -- 172.21.15.23:0/3644825693 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a001c630 con 0x55cda012d030 2024-03-28T05:28:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.456+0000 7fc0bdb64740 1 -- 172.21.15.23:0/3644825693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012d030 msgr2=0x55cda0133740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.456+0000 7fc0bdb64740 1 --2- 172.21.15.23:0/3644825693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012d030 0x55cda0133740 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0009c60 tx=0x7fc0a0010800 comp rx=0 tx=0).stop 2024-03-28T05:28:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.456+0000 7fc0bdb64740 1 -- 172.21.15.23:0/3644825693 shutdown_connections 2024-03-28T05:28:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.456+0000 7fc0bdb64740 1 --2- 172.21.15.23:0/3644825693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda0135e00 0x55cda0145670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.456+0000 7fc0bdb64740 1 --2- 172.21.15.23:0/3644825693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012d030 0x55cda0133740 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.456+0000 7fc0bdb64740 1 --2- 172.21.15.23:0/3644825693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cda012c710 0x55cda01355d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.456+0000 7fc0bdb64740 1 -- 172.21.15.23:0/3644825693 >> 172.21.15.23:0/3644825693 conn(0x55cda0127030 msgr2=0x55cda012e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.456+0000 7fc0bdb64740 1 -- 172.21.15.23:0/3644825693 shutdown_connections 2024-03-28T05:28:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.456+0000 7fc0bdb64740 1 -- 172.21.15.23:0/3644825693 wait complete. 2024-03-28T05:28:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.457+0000 7fc0bdb64740 1 -- start start 2024-03-28T05:28:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.457+0000 7fc0bdb64740 1 Processor -- start 2024-03-28T05:28:45.461 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.457+0000 7fc0bdb64740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda012c710 0x55cda013d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:45.461 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.457+0000 7fc0bdb64740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012d030 0x55cda013dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:45.461 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.457+0000 7fc0bdb64740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cda0135e00 0x55cda013e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:45.461 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.457+0000 7fc0bdb64740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b41ab4b0 con 0x55cda012d030 2024-03-28T05:28:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.457+0000 7fc0bdb64740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b41ac7d0 con 0x55cda0135e00 2024-03-28T05:28:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.457+0000 7fc0bdb64740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b41abb10 con 0x55cda012c710 2024-03-28T05:28:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.457+0000 7fc0bdb64740 10 client.0 _ll_register_callbacks cb 0x7fc0a8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:28:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.457+0000 7fc0b3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012d030 0x55cda013dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.457+0000 7fc0b3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012d030 0x55cda013dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41388/0 (socket says 172.21.15.23:41388) 2024-03-28T05:28:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.457+0000 7fc0b3fff640 1 -- 172.21.15.23:0/1176707321 learned_addr learned my addr 172.21.15.23:0/1176707321 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.457+0000 7fc0b8d13640 1 --2- 172.21.15.23:0/1176707321 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda012c710 0x55cda013d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.457+0000 7fc0b9514640 1 --2- 172.21.15.23:0/1176707321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cda0135e00 0x55cda013e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.458+0000 7fc0b3fff640 1 -- 172.21.15.23:0/1176707321 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda012c710 msgr2=0x55cda013d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.458+0000 7fc0b3fff640 1 --2- 172.21.15.23:0/1176707321 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda012c710 0x55cda013d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.458+0000 7fc0b3fff640 1 -- 172.21.15.23:0/1176707321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cda0135e00 msgr2=0x55cda013e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.458+0000 7fc0b3fff640 1 --2- 172.21.15.23:0/1176707321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cda0135e00 0x55cda013e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.458+0000 7fc0b3fff640 1 -- 172.21.15.23:0/1176707321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cda01a4990 con 0x55cda012d030 2024-03-28T05:28:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.458+0000 7fc0b8d13640 1 --2- 172.21.15.23:0/1176707321 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda012c710 0x55cda013d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:28:45.463 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.458+0000 7fc0b3fff640 1 --2- 172.21.15.23:0/1176707321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012d030 0x55cda013dcb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0003dc0 tx=0x7fc0a0003ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c188012209e810d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:45.463 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.458+0000 7fc093fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:28:45.463 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.458+0000 7fc093fff640 1 -- 172.21.15.23:0/1176707321 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a0021040 con 0x55cda012d030 2024-03-28T05:28:45.463 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.458+0000 7fc0bdb64740 1 -- 172.21.15.23:0/1176707321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc0a0009e10 con 0x55cda012d030 2024-03-28T05:28:45.463 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.458+0000 7fc093fff640 1 -- 172.21.15.23:0/1176707321 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fc0a0005460 con 0x55cda012d030 2024-03-28T05:28:45.463 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.458+0000 7fc093fff640 1 -- 172.21.15.23:0/1176707321 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0a0033050 con 0x55cda012d030 2024-03-28T05:28:45.463 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.458+0000 7fc0bdb64740 10 client.15894 fetch_fsmap learned FSMap version 20 2024-03-28T05:28:45.463 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.458+0000 7fc0bdb64740 1 -- 172.21.15.23:0/1176707321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55cda01a5c60 con 0x55cda012d030 2024-03-28T05:28:45.463 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.459+0000 7fc093fff640 1 -- 172.21.15.23:0/1176707321 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0a0009e10 con 0x55cda012d030 2024-03-28T05:28:45.463 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.459+0000 7fc093fff640 1 -- 172.21.15.23:0/1176707321 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fc0a001b030 con 0x55cda012d030 2024-03-28T05:28:45.463 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.459+0000 7fc0bdb64740 10 client.15894 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:28:45.463 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.459+0000 7fc0bdb64740 10 client.15894 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:28:45.463 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.459+0000 7fc0bdb64740 20 client.15894 populate_metadata read hostname 'smithi023' 2024-03-28T05:28:45.463 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.459+0000 7fc0bdb64740 1 --2- 172.21.15.23:0/1176707321 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55cda01a6390 0x55cda01a87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:45.463 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.459+0000 7fc0bdb64740 4 client.15894 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:28:45.464 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.459+0000 7fc0bdb64740 1 -- 172.21.15.23:0/1176707321 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55cda00983a0 con 0x55cda01a6390 2024-03-28T05:28:45.464 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.459+0000 7fc0b8d13640 1 --2- 172.21.15.23:0/1176707321 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55cda01a6390 0x55cda01a87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:45.464 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.459+0000 7fc093fff640 10 client.15894 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:28:45.464 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.459+0000 7fc0b8d13640 1 --2- 172.21.15.23:0/1176707321 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55cda01a6390 0x55cda01a87b0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7e923b1f6ac37e2 server_cookie=91afbdfa8b067ae4 in_seq=0 out_seq=0 2024-03-28T05:28:45.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:45 smithi113 ceph-mon[37164]: pgmap v158: 129 pgs: 129 active+clean; 172 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 657 KiB/s wr, 296 op/s 2024-03-28T05:28:45.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:45 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:28:45.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:45 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:28:45.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:45 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:28:45.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:45 smithi023 ceph-mon[31954]: pgmap v158: 129 pgs: 129 active+clean; 172 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 657 KiB/s wr, 296 op/s 2024-03-28T05:28:45.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:45 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:28:45.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:45 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:28:45.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:45 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:28:45.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:45 smithi098 ceph-mon[36087]: pgmap v158: 129 pgs: 129 active+clean; 172 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 657 KiB/s wr, 296 op/s 2024-03-28T05:28:45.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:45 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:28:45.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:45 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:28:45.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:45 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:28:45.900 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.900+0000 7fc093fff640 1 -- 172.21.15.23:0/1176707321 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55cda00983a0 con 0x55cda01a6390 2024-03-28T05:28:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.900+0000 7fc093fff640 10 client.15894 handle_command_reply: tid=0 2024-03-28T05:28:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.900+0000 7fc0bdb64740 1 client.15894 shutdown 2024-03-28T05:28:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.900+0000 7fc0bdb64740 1 -- 172.21.15.23:0/1176707321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012d030 msgr2=0x55cda013dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.900+0000 7fc0bdb64740 1 --2- 172.21.15.23:0/1176707321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012d030 0x55cda013dcb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0003dc0 tx=0x7fc0a0003ac0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.900+0000 7fc0bdb64740 1 -- 172.21.15.23:0/1176707321 shutdown_connections 2024-03-28T05:28:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.900+0000 7fc0bdb64740 1 --2- 172.21.15.23:0/1176707321 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55cda01a6390 0x55cda01a87b0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.900+0000 7fc093fff640 0 client.15894 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:28:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.900+0000 7fc0bdb64740 1 --2- 172.21.15.23:0/1176707321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cda0135e00 0x55cda013e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.900+0000 7fc0bdb64740 1 --2- 172.21.15.23:0/1176707321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda012d030 0x55cda013dcb0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.900+0000 7fc0bdb64740 1 --2- 172.21.15.23:0/1176707321 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda012c710 0x55cda013d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.900+0000 7fc0bdb64740 1 -- 172.21.15.23:0/1176707321 >> 172.21.15.23:0/1176707321 conn(0x55cda0127030 msgr2=0x55cda0125790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.901+0000 7fc0bdb64740 1 -- 172.21.15.23:0/1176707321 shutdown_connections 2024-03-28T05:28:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.901+0000 7fc0bdb64740 1 -- 172.21.15.23:0/1176707321 wait complete. 2024-03-28T05:28:45.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.901+0000 7fc0bdb64740 20 client.15894 trim_cache size 0 max 16384 2024-03-28T05:28:45.902 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:28:45.902 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:28:45.902 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:28:45.902 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:28:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.902+0000 7fc0baf77640 1 -- 172.21.15.23:0/565403724 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fc0a40a2a10 msgr2=0x7fc0a40a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.902+0000 7fc0baf77640 1 --2- 172.21.15.23:0/565403724 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fc0a40a2a10 0x7fc0a40a4ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac004af0 tx=0x7fc0ac032e90 comp rx=0 tx=0).stop 2024-03-28T05:28:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.902+0000 7fc0baf77640 1 -- 172.21.15.23:0/565403724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0b41007f0 msgr2=0x7fc0b4198de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.902+0000 7fc0baf77640 1 --2- 172.21.15.23:0/565403724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0b41007f0 0x7fc0b4198de0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc0a800e9f0 tx=0x7fc0a800eeb0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.902+0000 7fc0baf77640 1 -- 172.21.15.23:0/565403724 shutdown_connections 2024-03-28T05:28:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.902+0000 7fc0baf77640 1 --2- 172.21.15.23:0/565403724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0b4105540 0x7fc0b41a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.902+0000 7fc0baf77640 1 --2- 172.21.15.23:0/565403724 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fc0a40a2a10 0x7fc0a40a4ed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.902+0000 7fc0baf77640 1 --2- 172.21.15.23:0/565403724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0b41011b0 0x7fc0b4199320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.902+0000 7fc0baf77640 1 --2- 172.21.15.23:0/565403724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0b41007f0 0x7fc0b4198de0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.902+0000 7fc0baf77640 1 -- 172.21.15.23:0/565403724 >> 172.21.15.23:0/565403724 conn(0x7fc0b40773f0 msgr2=0x7fc0b40fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.902+0000 7fc0baf77640 1 -- 172.21.15.23:0/565403724 shutdown_connections 2024-03-28T05:28:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:45.902+0000 7fc0baf77640 1 -- 172.21.15.23:0/565403724 wait complete. 2024-03-28T05:28:46.400 INFO:tasks.workunit.client.0.smithi023.stdout: 52 266234 116 184507 59 128104 263 2024-03-28T05:28:47.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:47 smithi113 ceph-mon[37164]: pgmap v159: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 853 KiB/s wr, 313 op/s 2024-03-28T05:28:47.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:47 smithi023 ceph-mon[31954]: pgmap v159: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 853 KiB/s wr, 313 op/s 2024-03-28T05:28:47.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:47 smithi098 ceph-mon[36087]: pgmap v159: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 853 KiB/s wr, 313 op/s 2024-03-28T05:28:49.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:49 smithi113 ceph-mon[37164]: pgmap v160: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 404 KiB/s wr, 48 op/s 2024-03-28T05:28:49.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:49 smithi023 ceph-mon[31954]: pgmap v160: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 404 KiB/s wr, 48 op/s 2024-03-28T05:28:49.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:49 smithi098 ceph-mon[36087]: pgmap v160: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 404 KiB/s wr, 48 op/s 2024-03-28T05:28:51.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:51 smithi113 ceph-mon[37164]: pgmap v161: 129 pgs: 129 active+clean; 168 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 580 KiB/s wr, 117 op/s 2024-03-28T05:28:51.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:51 smithi023 ceph-mon[31954]: pgmap v161: 129 pgs: 129 active+clean; 168 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 580 KiB/s wr, 117 op/s 2024-03-28T05:28:51.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:51 smithi098 ceph-mon[36087]: pgmap v161: 129 pgs: 129 active+clean; 168 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 580 KiB/s wr, 117 op/s 2024-03-28T05:28:53.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:53 smithi113 ceph-mon[37164]: pgmap v162: 129 pgs: 129 active+clean; 180 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 1.5 MiB/s wr, 380 op/s 2024-03-28T05:28:53.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:53 smithi023 ceph-mon[31954]: pgmap v162: 129 pgs: 129 active+clean; 180 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 1.5 MiB/s wr, 380 op/s 2024-03-28T05:28:53.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:53 smithi098 ceph-mon[36087]: pgmap v162: 129 pgs: 129 active+clean; 180 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 1.5 MiB/s wr, 380 op/s 2024-03-28T05:28:55.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:55 smithi113 ceph-mon[37164]: pgmap v163: 129 pgs: 129 active+clean; 180 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 1.5 MiB/s wr, 364 op/s 2024-03-28T05:28:55.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:55 smithi023 ceph-mon[31954]: pgmap v163: 129 pgs: 129 active+clean; 180 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 1.5 MiB/s wr, 364 op/s 2024-03-28T05:28:55.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:55 smithi098 ceph-mon[36087]: pgmap v163: 129 pgs: 129 active+clean; 180 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 1.5 MiB/s wr, 364 op/s 2024-03-28T05:28:56.400 INFO:tasks.workunit.client.0.smithi023.stdout: 54 292258 175 193178 95 145049 367 2024-03-28T05:28:57.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:57 smithi113 ceph-mon[37164]: pgmap v164: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 408 op/s 2024-03-28T05:28:57.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:57 smithi023 ceph-mon[31954]: pgmap v164: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 408 op/s 2024-03-28T05:28:57.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:57 smithi098 ceph-mon[36087]: pgmap v164: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 408 op/s 2024-03-28T05:28:58.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.993+0000 7fd3feba9640 1 -- 172.21.15.23:0/176002846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3f8105540 msgr2=0x7fd3f8105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:58.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.994+0000 7fd3feba9640 1 --2- 172.21.15.23:0/176002846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3f8105540 0x7fd3f8105920 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec009900 tx=0x7fd3ec02bc30 comp rx=0 tx=0).stop 2024-03-28T05:28:58.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.994+0000 7fd3feba9640 1 -- 172.21.15.23:0/176002846 shutdown_connections 2024-03-28T05:28:58.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.994+0000 7fd3feba9640 1 --2- 172.21.15.23:0/176002846 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3f81011b0 0x7fd3f810d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:58.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.994+0000 7fd3feba9640 1 --2- 172.21.15.23:0/176002846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3f81007f0 0x7fd3f8100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:58.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.994+0000 7fd3feba9640 1 --2- 172.21.15.23:0/176002846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3f8105540 0x7fd3f8105920 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:58.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.994+0000 7fd3feba9640 1 -- 172.21.15.23:0/176002846 >> 172.21.15.23:0/176002846 conn(0x7fd3f80773f0 msgr2=0x7fd3f8077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:58.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.994+0000 7fd3feba9640 1 -- 172.21.15.23:0/176002846 shutdown_connections 2024-03-28T05:28:58.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.994+0000 7fd3feba9640 1 -- 172.21.15.23:0/176002846 wait complete. 2024-03-28T05:28:58.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.994+0000 7fd3feba9640 1 Processor -- start 2024-03-28T05:28:58.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.994+0000 7fd3feba9640 1 -- start start 2024-03-28T05:28:58.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.994+0000 7fd3feba9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3f81007f0 0x7fd3f8197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:58.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.994+0000 7fd3feba9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3f81011b0 0x7fd3f8198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:58.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.994+0000 7fd3feba9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3f8105540 0x7fd3f819c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:58.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.994+0000 7fd3feba9640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f8110a70 con 0x7fd3f81011b0 2024-03-28T05:28:58.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.994+0000 7fd3feba9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f81108f0 con 0x7fd3f81007f0 2024-03-28T05:28:58.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.994+0000 7fd3feba9640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f8110bf0 con 0x7fd3f8105540 2024-03-28T05:28:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.995+0000 7fd3f7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3f81011b0 0x7fd3f8198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.995+0000 7fd3f7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3f81011b0 0x7fd3f8198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37550/0 (socket says 172.21.15.23:37550) 2024-03-28T05:28:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.995+0000 7fd3f7fff640 1 -- 172.21.15.23:0/2869475452 learned_addr learned my addr 172.21.15.23:0/2869475452 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.995+0000 7fd3fd146640 1 --2- 172.21.15.23:0/2869475452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3f8105540 0x7fd3f819c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.995+0000 7fd3fc945640 1 --2- 172.21.15.23:0/2869475452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3f81007f0 0x7fd3f8197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.995+0000 7fd3f7fff640 1 -- 172.21.15.23:0/2869475452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3f8105540 msgr2=0x7fd3f819c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.995+0000 7fd3f7fff640 1 --2- 172.21.15.23:0/2869475452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3f8105540 0x7fd3f819c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.995+0000 7fd3f7fff640 1 -- 172.21.15.23:0/2869475452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3f81007f0 msgr2=0x7fd3f8197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.995+0000 7fd3f7fff640 1 --2- 172.21.15.23:0/2869475452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3f81007f0 0x7fd3f8197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.995+0000 7fd3f7fff640 1 -- 172.21.15.23:0/2869475452 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f819cef0 con 0x7fd3f81011b0 2024-03-28T05:28:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.995+0000 7fd3f7fff640 1 --2- 172.21.15.23:0/2869475452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3f81011b0 0x7fd3f8198440 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd3e400e9f0 tx=0x7fd3e400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.995+0000 7fd3f5ffb640 1 -- 172.21.15.23:0/2869475452 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e400cd80 con 0x7fd3f81011b0 2024-03-28T05:28:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.995+0000 7fd3f5ffb640 1 -- 172.21.15.23:0/2869475452 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3e4010910 con 0x7fd3f81011b0 2024-03-28T05:28:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.995+0000 7fd3feba9640 1 -- 172.21.15.23:0/2869475452 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f819d1e0 con 0x7fd3f81011b0 2024-03-28T05:28:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.995+0000 7fd3f5ffb640 1 -- 172.21.15.23:0/2869475452 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e4018990 con 0x7fd3f81011b0 2024-03-28T05:28:58.999 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.995+0000 7fd3feba9640 1 -- 172.21.15.23:0/2869475452 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f81a4a70 con 0x7fd3f81011b0 2024-03-28T05:28:59.003 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.998+0000 7fd3f5ffb640 1 -- 172.21.15.23:0/2869475452 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fd3e4010ac0 con 0x7fd3f81011b0 2024-03-28T05:28:59.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.998+0000 7fd3f5ffb640 1 --2- 172.21.15.23:0/2869475452 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fd3e80a2a90 0x7fd3e80a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:59.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.998+0000 7fd3fc945640 1 --2- 172.21.15.23:0/2869475452 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fd3e80a2a90 0x7fd3e80a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:59.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.998+0000 7fd401790740 1 Processor -- start 2024-03-28T05:28:59.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.998+0000 7fd3f5ffb640 1 -- 172.21.15.23:0/2869475452 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fd3e40cf490 con 0x7fd3f81011b0 2024-03-28T05:28:59.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.998+0000 7fd401790740 1 -- start start 2024-03-28T05:28:59.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.999+0000 7fd401790740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55941f75a710 0x55941f75aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:59.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.999+0000 7fd401790740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55941f75b030 0x55941f7638c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:59.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.999+0000 7fd401790740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f763e00 0x55941f7661f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:59.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.999+0000 7fd401790740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55941f6beb10 con 0x55941f763e00 2024-03-28T05:28:59.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.999+0000 7fd401790740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55941f667900 con 0x55941f75a710 2024-03-28T05:28:59.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.999+0000 7fd401790740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55941f6babb0 con 0x55941f75b030 2024-03-28T05:28:59.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.999+0000 7fd3fd146640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f763e00 0x55941f7661f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:59.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.999+0000 7fd3f7fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55941f75b030 0x55941f7638c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:59.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.999+0000 7fd3fc945640 1 --2- 172.21.15.23:0/2869475452 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fd3e80a2a90 0x7fd3e80a4f50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec02efd0 tx=0x7fd3ec000bc0 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.999+0000 7fd3fd146640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f763e00 0x55941f7661f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37556/0 (socket says 172.21.15.23:37556) 2024-03-28T05:28:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.999+0000 7fd3fd146640 1 -- 172.21.15.23:0/1314995500 learned_addr learned my addr 172.21.15.23:0/1314995500 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.999+0000 7fd3fc945640 1 --2- 172.21.15.23:0/1314995500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55941f75a710 0x55941f75aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.999+0000 7fd3f7fff640 1 -- 172.21.15.23:0/1314995500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55941f75a710 msgr2=0x55941f75aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.999+0000 7fd3f7fff640 1 --2- 172.21.15.23:0/1314995500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55941f75a710 0x55941f75aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.999+0000 7fd3f7fff640 1 -- 172.21.15.23:0/1314995500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f763e00 msgr2=0x55941f7661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.999+0000 7fd3f7fff640 1 --2- 172.21.15.23:0/1314995500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f763e00 0x55941f7661f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.999+0000 7fd3f7fff640 1 -- 172.21.15.23:0/1314995500 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55941f3d3210 con 0x55941f75b030 2024-03-28T05:28:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.999+0000 7fd3fd146640 1 --2- 172.21.15.23:0/1314995500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f763e00 0x55941f7661f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:28:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.999+0000 7fd3f7fff640 1 --2- 172.21.15.23:0/1314995500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55941f75b030 0x55941f7638c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd3e40cc850 tx=0x7fd3e409b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b050cf985199cda1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.999+0000 7fd3e2ffd640 1 -- 172.21.15.23:0/1314995500 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e40860b0 con 0x55941f75b030 2024-03-28T05:28:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:58.999+0000 7fd3e2ffd640 1 -- 172.21.15.23:0/1314995500 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3e40cf170 con 0x55941f75b030 2024-03-28T05:28:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.000+0000 7fd3e2ffd640 1 -- 172.21.15.23:0/1314995500 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e4018d30 con 0x55941f75b030 2024-03-28T05:28:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.000+0000 7fd401790740 1 -- 172.21.15.23:0/1314995500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55941f75b030 msgr2=0x55941f7638c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.000+0000 7fd401790740 1 --2- 172.21.15.23:0/1314995500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55941f75b030 0x55941f7638c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd3e40cc850 tx=0x7fd3e409b8d0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.000+0000 7fd401790740 1 -- 172.21.15.23:0/1314995500 shutdown_connections 2024-03-28T05:28:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.000+0000 7fd401790740 1 --2- 172.21.15.23:0/1314995500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f763e00 0x55941f7661f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.000+0000 7fd401790740 1 --2- 172.21.15.23:0/1314995500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55941f75b030 0x55941f7638c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.000+0000 7fd401790740 1 --2- 172.21.15.23:0/1314995500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55941f75a710 0x55941f75aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.000+0000 7fd401790740 1 -- 172.21.15.23:0/1314995500 >> 172.21.15.23:0/1314995500 conn(0x55941f755030 msgr2=0x55941f755410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.000+0000 7fd401790740 1 -- 172.21.15.23:0/1314995500 shutdown_connections 2024-03-28T05:28:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.000+0000 7fd401790740 1 -- 172.21.15.23:0/1314995500 wait complete. 2024-03-28T05:28:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.000+0000 7fd401790740 1 -- start start 2024-03-28T05:28:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.001+0000 7fd401790740 1 Processor -- start 2024-03-28T05:28:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.001+0000 7fd401790740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55941f75a710 0x55941f760e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.001+0000 7fd401790740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f75b030 0x55941f7613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.001+0000 7fd401790740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55941f763e00 0x55941f761900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.001+0000 7fd401790740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55941f766970 con 0x55941f75b030 2024-03-28T05:28:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.001+0000 7fd401790740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55941f67ad50 con 0x55941f763e00 2024-03-28T05:28:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.001+0000 7fd401790740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55941f766af0 con 0x55941f75a710 2024-03-28T05:28:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.001+0000 7fd401790740 10 client.0 _ll_register_callbacks cb 0x55941f67a720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:28:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.001+0000 7fd3f7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f75b030 0x55941f7613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.001+0000 7fd3fc945640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55941f75a710 0x55941f760e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.001+0000 7fd3f7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f75b030 0x55941f7613c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37564/0 (socket says 172.21.15.23:37564) 2024-03-28T05:28:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.001+0000 7fd3f7fff640 1 -- 172.21.15.23:0/2471453624 learned_addr learned my addr 172.21.15.23:0/2471453624 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.001+0000 7fd3fd146640 1 --2- 172.21.15.23:0/2471453624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55941f763e00 0x55941f761900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.001+0000 7fd3f7fff640 1 -- 172.21.15.23:0/2471453624 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55941f75a710 msgr2=0x55941f760e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.001+0000 7fd3f7fff640 1 --2- 172.21.15.23:0/2471453624 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55941f75a710 0x55941f760e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.001+0000 7fd3f7fff640 1 -- 172.21.15.23:0/2471453624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55941f763e00 msgr2=0x55941f761900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.001+0000 7fd3f7fff640 1 --2- 172.21.15.23:0/2471453624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55941f763e00 0x55941f761900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.001+0000 7fd3f7fff640 1 -- 172.21.15.23:0/2471453624 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55941f49f620 con 0x55941f75b030 2024-03-28T05:28:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.002+0000 7fd3f7fff640 1 --2- 172.21.15.23:0/2471453624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f75b030 0x55941f7613c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4097c70 tx=0x7fd3e409fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=492ac7998a829085 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.002+0000 7fd3d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:28:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.002+0000 7fd3d7fff640 1 -- 172.21.15.23:0/2471453624 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e40b9060 con 0x55941f75b030 2024-03-28T05:28:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.002+0000 7fd3d7fff640 1 -- 172.21.15.23:0/2471453624 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fd3e40afd50 con 0x55941f75b030 2024-03-28T05:28:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.002+0000 7fd401790740 1 -- 172.21.15.23:0/2471453624 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd3e40cf170 con 0x55941f75b030 2024-03-28T05:28:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.002+0000 7fd3d7fff640 1 -- 172.21.15.23:0/2471453624 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd3e40b4070 con 0x55941f75b030 2024-03-28T05:28:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.002+0000 7fd401790740 10 client.15906 fetch_fsmap learned FSMap version 20 2024-03-28T05:28:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.002+0000 7fd401790740 1 -- 172.21.15.23:0/2471453624 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55941f772c20 con 0x55941f75b030 2024-03-28T05:28:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.002+0000 7fd3d7fff640 1 -- 172.21.15.23:0/2471453624 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3e40cf170 con 0x55941f75b030 2024-03-28T05:28:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.002+0000 7fd3d7fff640 1 -- 172.21.15.23:0/2471453624 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fd3e409acf0 con 0x55941f75b030 2024-03-28T05:28:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.002+0000 7fd401790740 10 client.15906 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:28:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.002+0000 7fd401790740 10 client.15906 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:28:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.002+0000 7fd401790740 20 client.15906 populate_metadata read hostname 'smithi023' 2024-03-28T05:28:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.002+0000 7fd401790740 1 --2- 172.21.15.23:0/2471453624 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55941f773f70 0x55941f776360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.002+0000 7fd401790740 4 client.15906 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:28:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.002+0000 7fd401790740 1 -- 172.21.15.23:0/2471453624 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55941f6cc050 con 0x55941f773f70 2024-03-28T05:28:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.002+0000 7fd3fc945640 1 --2- 172.21.15.23:0/2471453624 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55941f773f70 0x55941f776360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.003+0000 7fd3d7fff640 10 client.15906 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:28:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.003+0000 7fd3fc945640 1 --2- 172.21.15.23:0/2471453624 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55941f773f70 0x55941f776360 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d932bc95bf2eb4a4 server_cookie=3f4b000dacb5f245 in_seq=0 out_seq=0 2024-03-28T05:28:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.004+0000 7fd3d7fff640 1 -- 172.21.15.23:0/2471453624 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55941f6cc050 con 0x55941f773f70 2024-03-28T05:28:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.004+0000 7fd3d7fff640 10 client.15906 handle_command_reply: tid=0 2024-03-28T05:28:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.004+0000 7fd401790740 1 client.15906 shutdown 2024-03-28T05:28:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.004+0000 7fd401790740 1 -- 172.21.15.23:0/2471453624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f75b030 msgr2=0x55941f7613c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.004+0000 7fd401790740 1 --2- 172.21.15.23:0/2471453624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f75b030 0x55941f7613c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4097c70 tx=0x7fd3e409fa30 comp rx=0 tx=0).stop 2024-03-28T05:28:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.004+0000 7fd401790740 1 -- 172.21.15.23:0/2471453624 shutdown_connections 2024-03-28T05:28:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.004+0000 7fd401790740 1 --2- 172.21.15.23:0/2471453624 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55941f773f70 0x55941f776360 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.004+0000 7fd401790740 1 --2- 172.21.15.23:0/2471453624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55941f763e00 0x55941f761900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.004+0000 7fd401790740 1 --2- 172.21.15.23:0/2471453624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f75b030 0x55941f7613c0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.004+0000 7fd401790740 1 --2- 172.21.15.23:0/2471453624 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55941f75a710 0x55941f760e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.004+0000 7fd401790740 1 -- 172.21.15.23:0/2471453624 >> 172.21.15.23:0/2471453624 conn(0x55941f755030 msgr2=0x55941f756d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.004+0000 7fd3d7fff640 0 client.15906 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:28:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.005+0000 7fd401790740 1 -- 172.21.15.23:0/2471453624 shutdown_connections 2024-03-28T05:28:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.005+0000 7fd401790740 1 -- 172.21.15.23:0/2471453624 wait complete. 2024-03-28T05:28:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.005+0000 7fd401790740 20 client.15906 trim_cache size 0 max 16384 2024-03-28T05:28:59.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.019+0000 7fd401790740 1 Processor -- start 2024-03-28T05:28:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.019+0000 7fd401790740 1 -- start start 2024-03-28T05:28:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.019+0000 7fd401790740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55941f75a710 0x55941f7635b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.019+0000 7fd401790740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f75b030 0x55941f756b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.019+0000 7fd401790740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55941f763e00 0x55941f75e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.019+0000 7fd401790740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f81aca60 con 0x55941f75b030 2024-03-28T05:28:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.019+0000 7fd401790740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f81af020 con 0x55941f763e00 2024-03-28T05:28:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.019+0000 7fd401790740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f81ad060 con 0x55941f75a710 2024-03-28T05:28:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.019+0000 7fd3f7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f75b030 0x55941f756b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.019+0000 7fd3f7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f75b030 0x55941f756b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37578/0 (socket says 172.21.15.23:37578) 2024-03-28T05:28:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.019+0000 7fd3f7fff640 1 -- 172.21.15.23:0/2784318393 learned_addr learned my addr 172.21.15.23:0/2784318393 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.019+0000 7fd3fc945640 1 --2- 172.21.15.23:0/2784318393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55941f75a710 0x55941f7635b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.020+0000 7fd3f7fff640 1 -- 172.21.15.23:0/2784318393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55941f75a710 msgr2=0x55941f7635b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.020+0000 7fd3fd146640 1 --2- 172.21.15.23:0/2784318393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55941f763e00 0x55941f75e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.020+0000 7fd3f7fff640 1 --2- 172.21.15.23:0/2784318393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55941f75a710 0x55941f7635b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.020+0000 7fd3f7fff640 1 -- 172.21.15.23:0/2784318393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55941f763e00 msgr2=0x55941f75e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.020+0000 7fd3f7fff640 1 --2- 172.21.15.23:0/2784318393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55941f763e00 0x55941f75e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.020+0000 7fd3f7fff640 1 -- 172.21.15.23:0/2784318393 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55941f759c90 con 0x55941f75b030 2024-03-28T05:28:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.020+0000 7fd3fd146640 1 --2- 172.21.15.23:0/2784318393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55941f763e00 0x55941f75e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:28:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.020+0000 7fd3f7fff640 1 --2- 172.21.15.23:0/2784318393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f75b030 0x55941f756b90 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd3e409b510 tx=0x7fd3e409b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9710a331ae5428f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.020+0000 7fd3e27fc640 1 -- 172.21.15.23:0/2784318393 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e40b9040 con 0x55941f75b030 2024-03-28T05:28:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.020+0000 7fd3e27fc640 1 -- 172.21.15.23:0/2784318393 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3e4097f40 con 0x55941f75b030 2024-03-28T05:28:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.020+0000 7fd3e27fc640 1 -- 172.21.15.23:0/2784318393 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e409aec0 con 0x55941f75b030 2024-03-28T05:28:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.020+0000 7fd401790740 1 -- 172.21.15.23:0/2784318393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f75b030 msgr2=0x55941f756b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.020+0000 7fd401790740 1 --2- 172.21.15.23:0/2784318393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f75b030 0x55941f756b90 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd3e409b510 tx=0x7fd3e409b8d0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.021+0000 7fd401790740 1 -- 172.21.15.23:0/2784318393 shutdown_connections 2024-03-28T05:28:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.021+0000 7fd401790740 1 --2- 172.21.15.23:0/2784318393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55941f763e00 0x55941f75e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.021+0000 7fd401790740 1 --2- 172.21.15.23:0/2784318393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f75b030 0x55941f756b90 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.021+0000 7fd401790740 1 --2- 172.21.15.23:0/2784318393 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55941f75a710 0x55941f7635b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.025 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.021+0000 7fd401790740 1 -- 172.21.15.23:0/2784318393 >> 172.21.15.23:0/2784318393 conn(0x55941f755030 msgr2=0x55941f756480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:59.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.021+0000 7fd401790740 1 -- 172.21.15.23:0/2784318393 shutdown_connections 2024-03-28T05:28:59.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.021+0000 7fd401790740 1 -- 172.21.15.23:0/2784318393 wait complete. 2024-03-28T05:28:59.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.021+0000 7fd401790740 1 -- start start 2024-03-28T05:28:59.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.021+0000 7fd401790740 1 Processor -- start 2024-03-28T05:28:59.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.021+0000 7fd401790740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55941f75a710 0x55941f776e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:59.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.021+0000 7fd401790740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55941f75b030 0x55941f7773a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:59.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.021+0000 7fd401790740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f763e00 0x55941f7ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:59.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.021+0000 7fd401790740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f81ad6a0 con 0x55941f763e00 2024-03-28T05:28:59.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.021+0000 7fd401790740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f81ae9c0 con 0x55941f75a710 2024-03-28T05:28:59.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.021+0000 7fd401790740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f81add00 con 0x55941f75b030 2024-03-28T05:28:59.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.021+0000 7fd401790740 10 client.0 _ll_register_callbacks cb 0x7fd3e400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:28:59.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.021+0000 7fd3fd146640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f763e00 0x55941f7ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:59.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.021+0000 7fd3fd146640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f763e00 0x55941f7ce4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37586/0 (socket says 172.21.15.23:37586) 2024-03-28T05:28:59.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.022+0000 7fd3fd146640 1 -- 172.21.15.23:0/2940030282 learned_addr learned my addr 172.21.15.23:0/2940030282 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:28:59.026 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.022+0000 7fd3f7fff640 1 --2- 172.21.15.23:0/2940030282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55941f75b030 0x55941f7773a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.022+0000 7fd3fc945640 1 --2- 172.21.15.23:0/2940030282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55941f75a710 0x55941f776e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.022+0000 7fd3fd146640 1 -- 172.21.15.23:0/2940030282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55941f75b030 msgr2=0x55941f7773a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.022+0000 7fd3fd146640 1 --2- 172.21.15.23:0/2940030282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55941f75b030 0x55941f7773a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.022+0000 7fd3fd146640 1 -- 172.21.15.23:0/2940030282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55941f75a710 msgr2=0x55941f776e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:28:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.022+0000 7fd3fd146640 1 --2- 172.21.15.23:0/2940030282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55941f75a710 0x55941f776e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.022+0000 7fd3fd146640 1 -- 172.21.15.23:0/2940030282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55941f7ce9e0 con 0x55941f763e00 2024-03-28T05:28:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.022+0000 7fd3fd146640 1 --2- 172.21.15.23:0/2940030282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f763e00 0x55941f7ce4a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd3f00042f0 tx=0x7fd3f0003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d3dc2d9ae9a516 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:28:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.022+0000 7fd3d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:28:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.022+0000 7fd3d7fff640 1 -- 172.21.15.23:0/2940030282 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f0022060 con 0x55941f763e00 2024-03-28T05:28:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.022+0000 7fd3d7fff640 1 -- 172.21.15.23:0/2940030282 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fd3f0010da0 con 0x55941f763e00 2024-03-28T05:28:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.022+0000 7fd401790740 1 -- 172.21.15.23:0/2940030282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd3e4097f40 con 0x55941f763e00 2024-03-28T05:28:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.022+0000 7fd3d7fff640 1 -- 172.21.15.23:0/2940030282 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd3f0029050 con 0x55941f763e00 2024-03-28T05:28:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.022+0000 7fd401790740 10 client.15912 fetch_fsmap learned FSMap version 20 2024-03-28T05:28:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.022+0000 7fd401790740 1 -- 172.21.15.23:0/2940030282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55941f7cfc60 con 0x55941f763e00 2024-03-28T05:28:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.023+0000 7fd3d7fff640 1 -- 172.21.15.23:0/2940030282 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3e4097f40 con 0x55941f763e00 2024-03-28T05:28:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.023+0000 7fd3d7fff640 1 -- 172.21.15.23:0/2940030282 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fd3f001b500 con 0x55941f763e00 2024-03-28T05:28:59.027 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.023+0000 7fd401790740 10 client.15912 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:28:59.028 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.023+0000 7fd401790740 10 client.15912 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:28:59.028 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.023+0000 7fd401790740 20 client.15912 populate_metadata read hostname 'smithi023' 2024-03-28T05:28:59.028 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.023+0000 7fd401790740 1 --2- 172.21.15.23:0/2940030282 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55941f7d03c0 0x55941f7d27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:28:59.028 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.023+0000 7fd401790740 4 client.15912 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:28:59.028 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.023+0000 7fd401790740 1 -- 172.21.15.23:0/2940030282 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55941f6c63a0 con 0x55941f7d03c0 2024-03-28T05:28:59.028 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.023+0000 7fd3fc945640 1 --2- 172.21.15.23:0/2940030282 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55941f7d03c0 0x55941f7d27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:28:59.028 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.023+0000 7fd3d7fff640 10 client.15912 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:28:59.028 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.023+0000 7fd3fc945640 1 --2- 172.21.15.23:0/2940030282 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55941f7d03c0 0x55941f7d27e0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95938ec9bebb095d server_cookie=65293371d33da351 in_seq=0 out_seq=0 2024-03-28T05:28:59.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:28:59 smithi113 ceph-mon[37164]: pgmap v165: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 1.8 MiB/s wr, 391 op/s 2024-03-28T05:28:59.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:28:59 smithi023 ceph-mon[31954]: pgmap v165: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 1.8 MiB/s wr, 391 op/s 2024-03-28T05:28:59.601 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.601+0000 7fd3d7fff640 1 -- 172.21.15.23:0/2940030282 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55941f6c63a0 con 0x55941f7d03c0 2024-03-28T05:28:59.601 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.601+0000 7fd3d7fff640 10 client.15912 handle_command_reply: tid=0 2024-03-28T05:28:59.601 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.601+0000 7fd401790740 1 client.15912 shutdown 2024-03-28T05:28:59.602 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.601+0000 7fd401790740 1 -- 172.21.15.23:0/2940030282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f763e00 msgr2=0x55941f7ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:59.602 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.601+0000 7fd401790740 1 --2- 172.21.15.23:0/2940030282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f763e00 0x55941f7ce4a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd3f00042f0 tx=0x7fd3f0003f90 comp rx=0 tx=0).stop 2024-03-28T05:28:59.603 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.603+0000 7fd401790740 1 -- 172.21.15.23:0/2940030282 shutdown_connections 2024-03-28T05:28:59.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.603+0000 7fd401790740 1 --2- 172.21.15.23:0/2940030282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55941f763e00 0x55941f7ce4a0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.603+0000 7fd401790740 1 --2- 172.21.15.23:0/2940030282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55941f75b030 0x55941f7773a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.603+0000 7fd401790740 1 --2- 172.21.15.23:0/2940030282 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55941f7d03c0 0x55941f7d27e0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.603+0000 7fd401790740 1 --2- 172.21.15.23:0/2940030282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55941f75a710 0x55941f776e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.603+0000 7fd3d7fff640 0 client.15912 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:28:59.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.603+0000 7fd401790740 1 -- 172.21.15.23:0/2940030282 >> 172.21.15.23:0/2940030282 conn(0x55941f755030 msgr2=0x55941f7616b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:59.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.603+0000 7fd401790740 1 -- 172.21.15.23:0/2940030282 shutdown_connections 2024-03-28T05:28:59.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.603+0000 7fd401790740 1 -- 172.21.15.23:0/2940030282 wait complete. 2024-03-28T05:28:59.604 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.603+0000 7fd401790740 20 client.15912 trim_cache size 0 max 16384 2024-03-28T05:28:59.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:28:59.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:28:59.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:28:59.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:28:59.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.604+0000 7fd3feba9640 1 -- 172.21.15.23:0/2869475452 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fd3e80a2a90 msgr2=0x7fd3e80a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:59.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.604+0000 7fd3feba9640 1 --2- 172.21.15.23:0/2869475452 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fd3e80a2a90 0x7fd3e80a4f50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec02efd0 tx=0x7fd3ec000bc0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.604+0000 7fd3feba9640 1 -- 172.21.15.23:0/2869475452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3f81011b0 msgr2=0x7fd3f8198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:28:59.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.604+0000 7fd3feba9640 1 --2- 172.21.15.23:0/2869475452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3f81011b0 0x7fd3f8198440 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd3e400e9f0 tx=0x7fd3e400eeb0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.604+0000 7fd3feba9640 1 -- 172.21.15.23:0/2869475452 shutdown_connections 2024-03-28T05:28:59.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.604+0000 7fd3feba9640 1 --2- 172.21.15.23:0/2869475452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3f8105540 0x7fd3f819c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.604+0000 7fd3feba9640 1 --2- 172.21.15.23:0/2869475452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3f81011b0 0x7fd3f8198440 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.604+0000 7fd3feba9640 1 --2- 172.21.15.23:0/2869475452 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fd3e80a2a90 0x7fd3e80a4f50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.604+0000 7fd3feba9640 1 --2- 172.21.15.23:0/2869475452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd3f81007f0 0x7fd3f8197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:28:59.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.604+0000 7fd3feba9640 1 -- 172.21.15.23:0/2869475452 >> 172.21.15.23:0/2869475452 conn(0x7fd3f80773f0 msgr2=0x7fd3f8068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:28:59.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.604+0000 7fd3feba9640 1 -- 172.21.15.23:0/2869475452 shutdown_connections 2024-03-28T05:28:59.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:28:59.604+0000 7fd3feba9640 1 -- 172.21.15.23:0/2869475452 wait complete. 2024-03-28T05:28:59.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:28:59 smithi098 ceph-mon[36087]: pgmap v165: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 1.8 MiB/s wr, 391 op/s 2024-03-28T05:29:01.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:01 smithi113 ceph-mon[37164]: pgmap v166: 129 pgs: 129 active+clean; 181 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 1.8 MiB/s wr, 538 op/s 2024-03-28T05:29:01.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:01 smithi023 ceph-mon[31954]: pgmap v166: 129 pgs: 129 active+clean; 181 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 1.8 MiB/s wr, 538 op/s 2024-03-28T05:29:01.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:01 smithi098 ceph-mon[36087]: pgmap v166: 129 pgs: 129 active+clean; 181 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 1.8 MiB/s wr, 538 op/s 2024-03-28T05:29:03.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:03 smithi113 ceph-mon[37164]: pgmap v167: 129 pgs: 129 active+clean; 177 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 687 op/s 2024-03-28T05:29:03.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:03 smithi023 ceph-mon[31954]: pgmap v167: 129 pgs: 129 active+clean; 177 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 687 op/s 2024-03-28T05:29:03.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:03 smithi098 ceph-mon[36087]: pgmap v167: 129 pgs: 129 active+clean; 177 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 687 op/s 2024-03-28T05:29:05.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:05 smithi113 ceph-mon[37164]: pgmap v168: 129 pgs: 129 active+clean; 177 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 924 KiB/s wr, 409 op/s 2024-03-28T05:29:05.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:05 smithi023 ceph-mon[31954]: pgmap v168: 129 pgs: 129 active+clean; 177 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 924 KiB/s wr, 409 op/s 2024-03-28T05:29:05.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:05 smithi098 ceph-mon[36087]: pgmap v168: 129 pgs: 129 active+clean; 177 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 924 KiB/s wr, 409 op/s 2024-03-28T05:29:06.400 INFO:tasks.workunit.client.0.smithi023.stdout: 56 145620 151 95661 145 70928 709 2024-03-28T05:29:07.494 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:07 smithi113 ceph-mon[37164]: pgmap v169: 129 pgs: 129 active+clean; 183 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 514 op/s 2024-03-28T05:29:07.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:07 smithi023 ceph-mon[31954]: pgmap v169: 129 pgs: 129 active+clean; 183 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 514 op/s 2024-03-28T05:29:07.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:07 smithi098 ceph-mon[36087]: pgmap v169: 129 pgs: 129 active+clean; 183 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 514 op/s 2024-03-28T05:29:09.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:09 smithi113 ceph-mon[37164]: pgmap v170: 129 pgs: 129 active+clean; 183 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 1.1 MiB/s wr, 470 op/s 2024-03-28T05:29:09.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:09 smithi023 ceph-mon[31954]: pgmap v170: 129 pgs: 129 active+clean; 183 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 1.1 MiB/s wr, 470 op/s 2024-03-28T05:29:09.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:09 smithi098 ceph-mon[36087]: pgmap v170: 129 pgs: 129 active+clean; 183 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 1.1 MiB/s wr, 470 op/s 2024-03-28T05:29:11.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:11 smithi113 ceph-mon[37164]: pgmap v171: 129 pgs: 129 active+clean; 188 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 509 op/s 2024-03-28T05:29:11.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:11 smithi023 ceph-mon[31954]: pgmap v171: 129 pgs: 129 active+clean; 188 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 509 op/s 2024-03-28T05:29:11.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:11 smithi098 ceph-mon[36087]: pgmap v171: 129 pgs: 129 active+clean; 188 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 509 op/s 2024-03-28T05:29:12.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.696+0000 7f0daee4d640 1 -- 172.21.15.23:0/2272762908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0da8102000 msgr2=0x7f0da810e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:12.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.696+0000 7f0daee4d640 1 --2- 172.21.15.23:0/2272762908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0da8102000 0x7f0da810e510 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f0da0009cd0 tx=0x7f0da002c0f0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.696+0000 7f0daee4d640 1 -- 172.21.15.23:0/2272762908 shutdown_connections 2024-03-28T05:29:12.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.696+0000 7f0daee4d640 1 --2- 172.21.15.23:0/2272762908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0da8102000 0x7f0da810e510 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.696+0000 7f0daee4d640 1 --2- 172.21.15.23:0/2272762908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0da8101640 0x7f0da8101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.696+0000 7f0daee4d640 1 --2- 172.21.15.23:0/2272762908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0da8107650 0x7f0da8107a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.696+0000 7f0daee4d640 1 -- 172.21.15.23:0/2272762908 >> 172.21.15.23:0/2272762908 conn(0x7f0da80fd130 msgr2=0x7f0da80ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:12.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.696+0000 7f0daee4d640 1 -- 172.21.15.23:0/2272762908 shutdown_connections 2024-03-28T05:29:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.696+0000 7f0daee4d640 1 -- 172.21.15.23:0/2272762908 wait complete. 2024-03-28T05:29:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.697+0000 7f0daee4d640 1 Processor -- start 2024-03-28T05:29:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.697+0000 7f0daee4d640 1 -- start start 2024-03-28T05:29:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.697+0000 7f0daee4d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0da8101640 0x7f0da8197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.697+0000 7f0daee4d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0da8102000 0x7f0da81983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.697+0000 7f0daee4d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0da8107650 0x7f0da819c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.697+0000 7f0daee4d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da8111a60 con 0x7f0da8107650 2024-03-28T05:29:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.697+0000 7f0daee4d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da81118e0 con 0x7f0da8101640 2024-03-28T05:29:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.697+0000 7f0daee4d640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da8111be0 con 0x7f0da8102000 2024-03-28T05:29:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.697+0000 7f0dad3ea640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0da8107650 0x7f0da819c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.697+0000 7f0da7fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0da8102000 0x7f0da81983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.697+0000 7f0dad3ea640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0da8107650 0x7f0da819c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55414/0 (socket says 172.21.15.23:55414) 2024-03-28T05:29:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.697+0000 7f0da7fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0da8102000 0x7f0da81983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:46536/0 (socket says 172.21.15.23:46536) 2024-03-28T05:29:12.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.697+0000 7f0dad3ea640 1 -- 172.21.15.23:0/226440771 learned_addr learned my addr 172.21.15.23:0/226440771 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:29:12.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.697+0000 7f0dacbe9640 1 --2- 172.21.15.23:0/226440771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0da8101640 0x7f0da8197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:12.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.697+0000 7f0da7fff640 1 -- 172.21.15.23:0/226440771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0da8101640 msgr2=0x7f0da8197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:12.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.697+0000 7f0da7fff640 1 --2- 172.21.15.23:0/226440771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0da8101640 0x7f0da8197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.697+0000 7f0da7fff640 1 -- 172.21.15.23:0/226440771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0da8107650 msgr2=0x7f0da819c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:12.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.697+0000 7f0da7fff640 1 --2- 172.21.15.23:0/226440771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0da8107650 0x7f0da819c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.697+0000 7f0da7fff640 1 -- 172.21.15.23:0/226440771 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da819ce50 con 0x7f0da8102000 2024-03-28T05:29:12.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.697+0000 7f0dad3ea640 1 --2- 172.21.15.23:0/226440771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0da8107650 0x7f0da819c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:29:12.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.698+0000 7f0da7fff640 1 --2- 172.21.15.23:0/226440771 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0da8102000 0x7f0da81983a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0d9400d6d0 tx=0x7f0d9400db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:12.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.698+0000 7f0da5ffb640 1 -- 172.21.15.23:0/226440771 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d94017070 con 0x7f0da8102000 2024-03-28T05:29:12.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.698+0000 7f0daee4d640 1 -- 172.21.15.23:0/226440771 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0da819d140 con 0x7f0da8102000 2024-03-28T05:29:12.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.698+0000 7f0da5ffb640 1 -- 172.21.15.23:0/226440771 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d94009820 con 0x7f0da8102000 2024-03-28T05:29:12.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.698+0000 7f0da5ffb640 1 -- 172.21.15.23:0/226440771 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d94004cd0 con 0x7f0da8102000 2024-03-28T05:29:12.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.698+0000 7f0daee4d640 1 -- 172.21.15.23:0/226440771 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0da81a49d0 con 0x7f0da8102000 2024-03-28T05:29:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.700+0000 7f0da5ffb640 1 -- 172.21.15.23:0/226440771 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0d940099d0 con 0x7f0da8102000 2024-03-28T05:29:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.701+0000 7f0db1a30740 1 Processor -- start 2024-03-28T05:29:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.701+0000 7f0db1a30740 1 -- start start 2024-03-28T05:29:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.701+0000 7f0db1a30740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55588678b710 0x55588678baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.701+0000 7f0db1a30740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55588678c030 0x5558867948c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.701+0000 7f0db1a30740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555886794e00 0x5558867971f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.701+0000 7f0db1a30740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558866efb10 con 0x55588678c030 2024-03-28T05:29:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.701+0000 7f0db1a30740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x555886698900 con 0x555886794e00 2024-03-28T05:29:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.701+0000 7f0db1a30740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558866ebbb0 con 0x55588678b710 2024-03-28T05:29:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.701+0000 7f0da5ffb640 1 --2- 172.21.15.23:0/226440771 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f0d980a2a90 0x7f0d980a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.701+0000 7f0dacbe9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55588678c030 0x5558867948c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.701+0000 7f0dacbe9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55588678c030 0x5558867948c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55416/0 (socket says 172.21.15.23:55416) 2024-03-28T05:29:12.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.701+0000 7f0dacbe9640 1 -- 172.21.15.23:0/868606883 learned_addr learned my addr 172.21.15.23:0/868606883 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:29:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.701+0000 7f0dad3ea640 1 --2- 172.21.15.23:0/868606883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55588678b710 0x55588678baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.701+0000 7f0da7fff640 1 --2- 172.21.15.23:0/868606883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555886794e00 0x5558867971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.701+0000 7f0dad3ea640 1 --2- 172.21.15.23:0/226440771 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f0d980a2a90 0x7f0d980a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.701+0000 7f0da5ffb640 1 -- 172.21.15.23:0/226440771 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f0d940ced20 con 0x7f0da8102000 2024-03-28T05:29:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.701+0000 7f0dacbe9640 1 -- 172.21.15.23:0/868606883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55588678b710 msgr2=0x55588678baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.701+0000 7f0dacbe9640 1 --2- 172.21.15.23:0/868606883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55588678b710 0x55588678baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.701+0000 7f0dacbe9640 1 -- 172.21.15.23:0/868606883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555886794e00 msgr2=0x5558867971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.701+0000 7f0dacbe9640 1 --2- 172.21.15.23:0/868606883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555886794e00 0x5558867971f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.701+0000 7f0dacbe9640 1 -- 172.21.15.23:0/868606883 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555886404210 con 0x55588678c030 2024-03-28T05:29:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.702+0000 7f0dacbe9640 1 --2- 172.21.15.23:0/868606883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55588678c030 0x5558867948c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c009ca0 tx=0x7f0d9c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec2fdc2ec3853558 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.702+0000 7f0dad3ea640 1 --2- 172.21.15.23:0/226440771 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f0d980a2a90 0x7f0d980a4f50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0da0005fe0 tx=0x7f0da0005d80 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.702+0000 7f0d92ffd640 1 -- 172.21.15.23:0/868606883 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c01e070 con 0x55588678c030 2024-03-28T05:29:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.702+0000 7f0d92ffd640 1 -- 172.21.15.23:0/868606883 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d9c019b10 con 0x55588678c030 2024-03-28T05:29:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.702+0000 7f0d92ffd640 1 -- 172.21.15.23:0/868606883 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c00ea80 con 0x55588678c030 2024-03-28T05:29:12.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.702+0000 7f0db1a30740 1 -- 172.21.15.23:0/868606883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55588678c030 msgr2=0x5558867948c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:12.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.702+0000 7f0db1a30740 1 --2- 172.21.15.23:0/868606883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55588678c030 0x5558867948c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c009ca0 tx=0x7f0d9c011810 comp rx=0 tx=0).stop 2024-03-28T05:29:12.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.703+0000 7f0db1a30740 1 -- 172.21.15.23:0/868606883 shutdown_connections 2024-03-28T05:29:12.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.703+0000 7f0db1a30740 1 --2- 172.21.15.23:0/868606883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555886794e00 0x5558867971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.703+0000 7f0db1a30740 1 --2- 172.21.15.23:0/868606883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55588678c030 0x5558867948c0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.703+0000 7f0db1a30740 1 --2- 172.21.15.23:0/868606883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55588678b710 0x55588678baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.703+0000 7f0db1a30740 1 -- 172.21.15.23:0/868606883 >> 172.21.15.23:0/868606883 conn(0x555886786030 msgr2=0x555886786410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:12.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.703+0000 7f0db1a30740 1 -- 172.21.15.23:0/868606883 shutdown_connections 2024-03-28T05:29:12.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.703+0000 7f0db1a30740 1 -- 172.21.15.23:0/868606883 wait complete. 2024-03-28T05:29:12.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.703+0000 7f0db1a30740 1 -- start start 2024-03-28T05:29:12.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.703+0000 7f0db1a30740 1 Processor -- start 2024-03-28T05:29:12.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.703+0000 7f0db1a30740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588678b710 0x555886791ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:12.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.703+0000 7f0db1a30740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55588678c030 0x555886792430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:12.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.703+0000 7f0db1a30740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555886794e00 0x555886792970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:12.709 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.703+0000 7f0db1a30740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555886797a30 con 0x555886794e00 2024-03-28T05:29:12.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.703+0000 7f0db1a30740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558866abd50 con 0x55588678b710 2024-03-28T05:29:12.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.703+0000 7f0db1a30740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x555886797bb0 con 0x55588678c030 2024-03-28T05:29:12.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.703+0000 7f0db1a30740 10 client.0 _ll_register_callbacks cb 0x5558866ab720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:29:12.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.703+0000 7f0dad3ea640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555886794e00 0x555886792970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:12.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.703+0000 7f0da7fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55588678c030 0x555886792430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:12.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.703+0000 7f0da7fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55588678c030 0x555886792430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:46550/0 (socket says 172.21.15.23:46550) 2024-03-28T05:29:12.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.704+0000 7f0da7fff640 1 -- 172.21.15.23:0/1862231994 learned_addr learned my addr 172.21.15.23:0/1862231994 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:29:12.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.703+0000 7f0dad3ea640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555886794e00 0x555886792970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55426/0 (socket says 172.21.15.23:55426) 2024-03-28T05:29:12.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.704+0000 7f0dacbe9640 1 --2- 172.21.15.23:0/1862231994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588678b710 0x555886791ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:12.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.704+0000 7f0da7fff640 1 -- 172.21.15.23:0/1862231994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588678b710 msgr2=0x555886791ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:12.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.704+0000 7f0da7fff640 1 --2- 172.21.15.23:0/1862231994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588678b710 0x555886791ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.704+0000 7f0da7fff640 1 -- 172.21.15.23:0/1862231994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555886794e00 msgr2=0x555886792970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:12.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.704+0000 7f0da7fff640 1 --2- 172.21.15.23:0/1862231994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555886794e00 0x555886792970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.710 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.704+0000 7f0da7fff640 1 -- 172.21.15.23:0/1862231994 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558864d0620 con 0x55588678c030 2024-03-28T05:29:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.704+0000 7f0da7fff640 1 --2- 172.21.15.23:0/1862231994 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55588678c030 0x555886792430 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0d94009610 tx=0x7f0d9409a9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b96e03a4e5b2ec6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.704+0000 7f0d87fff640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-28T05:29:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.704+0000 7f0d87fff640 1 -- 172.21.15.23:0/1862231994 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d94020040 con 0x55588678c030 2024-03-28T05:29:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.704+0000 7f0db1a30740 1 -- 172.21.15.23:0/1862231994 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0d9c019b10 con 0x55588678c030 2024-03-28T05:29:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.704+0000 7f0d87fff640 1 -- 172.21.15.23:0/1862231994 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f0d940cc300 con 0x55588678c030 2024-03-28T05:29:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.704+0000 7f0d87fff640 1 -- 172.21.15.23:0/1862231994 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d9407e050 con 0x55588678c030 2024-03-28T05:29:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.704+0000 7f0db1a30740 10 client.25759 fetch_fsmap learned FSMap version 20 2024-03-28T05:29:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.704+0000 7f0db1a30740 1 -- 172.21.15.23:0/1862231994 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5558867a3db0 con 0x55588678c030 2024-03-28T05:29:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.705+0000 7f0d87fff640 1 -- 172.21.15.23:0/1862231994 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d9c019b10 con 0x55588678c030 2024-03-28T05:29:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.705+0000 7f0d87fff640 1 -- 172.21.15.23:0/1862231994 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f0d9409a490 con 0x55588678c030 2024-03-28T05:29:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.705+0000 7f0db1a30740 10 client.25759 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:29:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.705+0000 7f0db1a30740 10 client.25759 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:29:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.705+0000 7f0db1a30740 20 client.25759 populate_metadata read hostname 'smithi023' 2024-03-28T05:29:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.705+0000 7f0db1a30740 1 --2- 172.21.15.23:0/1862231994 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5558867a5270 0x5558867a7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:12.711 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.705+0000 7f0db1a30740 4 client.25759 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:29:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.705+0000 7f0db1a30740 1 -- 172.21.15.23:0/1862231994 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5558866fd050 con 0x5558867a5270 2024-03-28T05:29:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.705+0000 7f0dacbe9640 1 --2- 172.21.15.23:0/1862231994 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5558867a5270 0x5558867a7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.705+0000 7f0dacbe9640 1 --2- 172.21.15.23:0/1862231994 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5558867a5270 0x5558867a7660 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93d76eb7f9f08c61 server_cookie=82da9d69b1c0cec3 in_seq=0 out_seq=0 2024-03-28T05:29:12.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.705+0000 7f0d87fff640 10 client.25759 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:29:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.707+0000 7f0d87fff640 1 -- 172.21.15.23:0/1862231994 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5558866fd050 con 0x5558867a5270 2024-03-28T05:29:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.707+0000 7f0d87fff640 10 client.25759 handle_command_reply: tid=0 2024-03-28T05:29:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.707+0000 7f0db1a30740 1 client.25759 shutdown 2024-03-28T05:29:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.707+0000 7f0db1a30740 1 -- 172.21.15.23:0/1862231994 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55588678c030 msgr2=0x555886792430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.707+0000 7f0db1a30740 1 --2- 172.21.15.23:0/1862231994 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55588678c030 0x555886792430 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0d94009610 tx=0x7f0d9409a9a0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.707+0000 7f0db1a30740 1 -- 172.21.15.23:0/1862231994 shutdown_connections 2024-03-28T05:29:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.707+0000 7f0db1a30740 1 --2- 172.21.15.23:0/1862231994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555886794e00 0x555886792970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.707+0000 7f0db1a30740 1 --2- 172.21.15.23:0/1862231994 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55588678c030 0x555886792430 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.707+0000 7f0db1a30740 1 --2- 172.21.15.23:0/1862231994 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5558867a5270 0x5558867a7660 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.707+0000 7f0db1a30740 1 --2- 172.21.15.23:0/1862231994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588678b710 0x555886791ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.707+0000 7f0d87fff640 0 client.25759 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:29:12.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.707+0000 7f0db1a30740 1 -- 172.21.15.23:0/1862231994 >> 172.21.15.23:0/1862231994 conn(0x555886786030 msgr2=0x555886787d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:12.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.708+0000 7f0db1a30740 1 -- 172.21.15.23:0/1862231994 shutdown_connections 2024-03-28T05:29:12.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.708+0000 7f0db1a30740 1 -- 172.21.15.23:0/1862231994 wait complete. 2024-03-28T05:29:12.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.708+0000 7f0db1a30740 20 client.25759 trim_cache size 0 max 16384 2024-03-28T05:29:12.724 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.722+0000 7f0db1a30740 1 Processor -- start 2024-03-28T05:29:12.724 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.722+0000 7f0db1a30740 1 -- start start 2024-03-28T05:29:12.724 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.722+0000 7f0db1a30740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588678b710 0x555886794620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:12.724 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.722+0000 7f0db1a30740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55588678c030 0x555886784ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:12.724 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.722+0000 7f0db1a30740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555886794e00 0x5558867a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:12.724 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.722+0000 7f0db1a30740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da81ac9f0 con 0x555886794e00 2024-03-28T05:29:12.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.722+0000 7f0db1a30740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da81aefb0 con 0x55588678b710 2024-03-28T05:29:12.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.722+0000 7f0db1a30740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da81acff0 con 0x55588678c030 2024-03-28T05:29:12.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.722+0000 7f0dad3ea640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555886794e00 0x5558867a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:12.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.722+0000 7f0dad3ea640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555886794e00 0x5558867a02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55428/0 (socket says 172.21.15.23:55428) 2024-03-28T05:29:12.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.722+0000 7f0dad3ea640 1 -- 172.21.15.23:0/1620057351 learned_addr learned my addr 172.21.15.23:0/1620057351 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:29:12.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.722+0000 7f0da7fff640 1 --2- 172.21.15.23:0/1620057351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55588678c030 0x555886784ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:12.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.722+0000 7f0dad3ea640 1 -- 172.21.15.23:0/1620057351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55588678c030 msgr2=0x555886784ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:12.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.722+0000 7f0dad3ea640 1 --2- 172.21.15.23:0/1620057351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55588678c030 0x555886784ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.722+0000 7f0dacbe9640 1 --2- 172.21.15.23:0/1620057351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588678b710 0x555886794620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:12.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.722+0000 7f0dad3ea640 1 -- 172.21.15.23:0/1620057351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588678b710 msgr2=0x555886794620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:12.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.723+0000 7f0dad3ea640 1 --2- 172.21.15.23:0/1620057351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588678b710 0x555886794620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.723+0000 7f0dad3ea640 1 -- 172.21.15.23:0/1620057351 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558867a7cc0 con 0x555886794e00 2024-03-28T05:29:12.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.723+0000 7f0dad3ea640 1 --2- 172.21.15.23:0/1620057351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555886794e00 0x5558867a02c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0da0009bf0 tx=0x7f0da002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a894513ca084530 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:12.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.723+0000 7f0d927fc640 1 -- 172.21.15.23:0/1620057351 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da0039040 con 0x555886794e00 2024-03-28T05:29:12.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.723+0000 7f0d927fc640 1 -- 172.21.15.23:0/1620057351 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0da0003e50 con 0x555886794e00 2024-03-28T05:29:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.723+0000 7f0d927fc640 1 -- 172.21.15.23:0/1620057351 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da0034e50 con 0x555886794e00 2024-03-28T05:29:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.724+0000 7f0db1a30740 1 -- 172.21.15.23:0/1620057351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555886794e00 msgr2=0x5558867a02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.724+0000 7f0db1a30740 1 --2- 172.21.15.23:0/1620057351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555886794e00 0x5558867a02c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0da0009bf0 tx=0x7f0da002e040 comp rx=0 tx=0).stop 2024-03-28T05:29:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.724+0000 7f0db1a30740 1 -- 172.21.15.23:0/1620057351 shutdown_connections 2024-03-28T05:29:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.724+0000 7f0db1a30740 1 --2- 172.21.15.23:0/1620057351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555886794e00 0x5558867a02c0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.724+0000 7f0db1a30740 1 --2- 172.21.15.23:0/1620057351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55588678c030 0x555886784ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.724+0000 7f0db1a30740 1 --2- 172.21.15.23:0/1620057351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588678b710 0x555886794620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.724+0000 7f0db1a30740 1 -- 172.21.15.23:0/1620057351 >> 172.21.15.23:0/1620057351 conn(0x555886786030 msgr2=0x555886784790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.724+0000 7f0db1a30740 1 -- 172.21.15.23:0/1620057351 shutdown_connections 2024-03-28T05:29:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.724+0000 7f0db1a30740 1 -- 172.21.15.23:0/1620057351 wait complete. 2024-03-28T05:29:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.724+0000 7f0db1a30740 1 -- start start 2024-03-28T05:29:12.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.724+0000 7f0db1a30740 1 Processor -- start 2024-03-28T05:29:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.724+0000 7f0db1a30740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55588678b710 0x55588678f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:12.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.724+0000 7f0db1a30740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55588678c030 0x55588678f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.724+0000 7f0db1a30740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555886794e00 0x5558867f8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.724+0000 7f0db1a30740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da81ad630 con 0x55588678c030 2024-03-28T05:29:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.725+0000 7f0db1a30740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da81ae950 con 0x555886794e00 2024-03-28T05:29:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.725+0000 7f0db1a30740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da81adc90 con 0x55588678b710 2024-03-28T05:29:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.725+0000 7f0db1a30740 10 client.0 _ll_register_callbacks cb 0x7f0d9409aec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:29:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.725+0000 7f0da7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55588678c030 0x55588678f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.725+0000 7f0da7fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55588678c030 0x55588678f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55434/0 (socket says 172.21.15.23:55434) 2024-03-28T05:29:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.725+0000 7f0da7fff640 1 -- 172.21.15.23:0/784107478 learned_addr learned my addr 172.21.15.23:0/784107478 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:29:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.725+0000 7f0dacbe9640 1 --2- 172.21.15.23:0/784107478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55588678b710 0x55588678f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.725+0000 7f0da7fff640 1 -- 172.21.15.23:0/784107478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55588678b710 msgr2=0x55588678f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.725+0000 7f0da7fff640 1 --2- 172.21.15.23:0/784107478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55588678b710 0x55588678f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.725+0000 7f0dad3ea640 1 --2- 172.21.15.23:0/784107478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555886794e00 0x5558867f8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.725+0000 7f0da7fff640 1 -- 172.21.15.23:0/784107478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555886794e00 msgr2=0x5558867f8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:12.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.725+0000 7f0da7fff640 1 --2- 172.21.15.23:0/784107478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555886794e00 0x5558867f8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.725+0000 7f0da7fff640 1 -- 172.21.15.23:0/784107478 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558867f92e0 con 0x55588678c030 2024-03-28T05:29:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.725+0000 7f0da7fff640 1 --2- 172.21.15.23:0/784107478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55588678c030 0x55588678f650 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f0d94097a10 tx=0x7f0d940a0290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6344b39bd9c62b84 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.725+0000 7f0d87fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:29:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.725+0000 7f0d87fff640 1 -- 172.21.15.23:0/784107478 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d94020070 con 0x55588678c030 2024-03-28T05:29:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.725+0000 7f0d87fff640 1 -- 172.21.15.23:0/784107478 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f0d9409a490 con 0x55588678c030 2024-03-28T05:29:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.725+0000 7f0db1a30740 1 -- 172.21.15.23:0/784107478 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0da0003e50 con 0x55588678c030 2024-03-28T05:29:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.725+0000 7f0d87fff640 1 -- 172.21.15.23:0/784107478 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d94079e60 con 0x55588678c030 2024-03-28T05:29:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.725+0000 7f0db1a30740 10 client.15930 fetch_fsmap learned FSMap version 20 2024-03-28T05:29:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.725+0000 7f0db1a30740 1 -- 172.21.15.23:0/784107478 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x555886801660 con 0x55588678c030 2024-03-28T05:29:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.726+0000 7f0d87fff640 1 -- 172.21.15.23:0/784107478 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0da0003e50 con 0x55588678c030 2024-03-28T05:29:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.726+0000 7f0d87fff640 1 -- 172.21.15.23:0/784107478 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f0d94084120 con 0x55588678c030 2024-03-28T05:29:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.726+0000 7f0db1a30740 10 client.15930 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:29:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.726+0000 7f0db1a30740 10 client.15930 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:29:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.726+0000 7f0db1a30740 20 client.15930 populate_metadata read hostname 'smithi023' 2024-03-28T05:29:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.726+0000 7f0db1a30740 1 --2- 172.21.15.23:0/784107478 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x555886801b40 0x555886803f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.726+0000 7f0db1a30740 4 client.15930 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:29:12.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.726+0000 7f0db1a30740 1 -- 172.21.15.23:0/784107478 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5558866f73a0 con 0x555886801b40 2024-03-28T05:29:12.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.726+0000 7f0dacbe9640 1 --2- 172.21.15.23:0/784107478 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x555886801b40 0x555886803f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:12.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.726+0000 7f0dacbe9640 1 --2- 172.21.15.23:0/784107478 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x555886801b40 0x555886803f60 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af5e387ae82605a1 server_cookie=d8df68b1832a86c1 in_seq=0 out_seq=0 2024-03-28T05:29:12.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:12.726+0000 7f0d87fff640 10 client.15930 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:29:13.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:13 smithi113 ceph-mon[37164]: pgmap v172: 129 pgs: 129 active+clean; 193 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 415 op/s 2024-03-28T05:29:13.531 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:13 smithi023 ceph-mon[31954]: pgmap v172: 129 pgs: 129 active+clean; 193 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 415 op/s 2024-03-28T05:29:13.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:13 smithi098 ceph-mon[36087]: pgmap v172: 129 pgs: 129 active+clean; 193 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 415 op/s 2024-03-28T05:29:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.696+0000 7f0d87fff640 1 -- 172.21.15.23:0/784107478 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5558866f73a0 con 0x555886801b40 2024-03-28T05:29:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.696+0000 7f0d87fff640 10 client.15930 handle_command_reply: tid=0 2024-03-28T05:29:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.696+0000 7f0db1a30740 1 client.15930 shutdown 2024-03-28T05:29:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.696+0000 7f0db1a30740 1 -- 172.21.15.23:0/784107478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55588678c030 msgr2=0x55588678f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.696+0000 7f0db1a30740 1 --2- 172.21.15.23:0/784107478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55588678c030 0x55588678f650 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f0d94097a10 tx=0x7f0d940a0290 comp rx=0 tx=0).stop 2024-03-28T05:29:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.697+0000 7f0db1a30740 1 -- 172.21.15.23:0/784107478 shutdown_connections 2024-03-28T05:29:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.697+0000 7f0db1a30740 1 --2- 172.21.15.23:0/784107478 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x555886801b40 0x555886803f60 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.697+0000 7f0db1a30740 1 --2- 172.21.15.23:0/784107478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555886794e00 0x5558867f8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.697+0000 7f0db1a30740 1 --2- 172.21.15.23:0/784107478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55588678c030 0x55588678f650 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.697+0000 7f0db1a30740 1 --2- 172.21.15.23:0/784107478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55588678b710 0x55588678f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.697+0000 7f0db1a30740 1 -- 172.21.15.23:0/784107478 >> 172.21.15.23:0/784107478 conn(0x555886786030 msgr2=0x555886785020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.697+0000 7f0d87fff640 0 client.15930 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:29:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.697+0000 7f0db1a30740 1 -- 172.21.15.23:0/784107478 shutdown_connections 2024-03-28T05:29:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.697+0000 7f0db1a30740 1 -- 172.21.15.23:0/784107478 wait complete. 2024-03-28T05:29:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.697+0000 7f0db1a30740 20 client.15930 trim_cache size 0 max 16384 2024-03-28T05:29:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:29:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:29:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:29:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:29:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.698+0000 7f0daee4d640 1 -- 172.21.15.23:0/226440771 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f0d980a2a90 msgr2=0x7f0d980a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.698+0000 7f0daee4d640 1 --2- 172.21.15.23:0/226440771 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f0d980a2a90 0x7f0d980a4f50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0da0005fe0 tx=0x7f0da0005d80 comp rx=0 tx=0).stop 2024-03-28T05:29:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.698+0000 7f0daee4d640 1 -- 172.21.15.23:0/226440771 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0da8102000 msgr2=0x7f0da81983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.698+0000 7f0daee4d640 1 --2- 172.21.15.23:0/226440771 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0da8102000 0x7f0da81983a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0d9400d6d0 tx=0x7f0d9400db90 comp rx=0 tx=0).stop 2024-03-28T05:29:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.698+0000 7f0daee4d640 1 -- 172.21.15.23:0/226440771 shutdown_connections 2024-03-28T05:29:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.698+0000 7f0daee4d640 1 --2- 172.21.15.23:0/226440771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0da8107650 0x7f0da819c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.698+0000 7f0daee4d640 1 --2- 172.21.15.23:0/226440771 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0da8102000 0x7f0da81983a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.698+0000 7f0daee4d640 1 --2- 172.21.15.23:0/226440771 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f0d980a2a90 0x7f0d980a4f50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.698+0000 7f0daee4d640 1 --2- 172.21.15.23:0/226440771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0da8101640 0x7f0da8197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.698+0000 7f0daee4d640 1 -- 172.21.15.23:0/226440771 >> 172.21.15.23:0/226440771 conn(0x7f0da80fd130 msgr2=0x7f0da80fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.698+0000 7f0daee4d640 1 -- 172.21.15.23:0/226440771 shutdown_connections 2024-03-28T05:29:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:13.698+0000 7f0daee4d640 1 -- 172.21.15.23:0/226440771 wait complete. 2024-03-28T05:29:16.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:15 smithi023 ceph-mon[31954]: pgmap v173: 129 pgs: 129 active+clean; 193 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 197 op/s 2024-03-28T05:29:16.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:15 smithi098 ceph-mon[36087]: pgmap v173: 129 pgs: 129 active+clean; 193 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 197 op/s 2024-03-28T05:29:16.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:15 smithi113 ceph-mon[37164]: pgmap v173: 129 pgs: 129 active+clean; 193 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 197 op/s 2024-03-28T05:29:16.401 INFO:tasks.workunit.client.0.smithi023.stdout: 60 23236 151 14995 151 9470 564 2024-03-28T05:29:18.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:17 smithi023 ceph-mon[31954]: pgmap v174: 129 pgs: 129 active+clean; 202 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.5 MiB/s wr, 852 op/s 2024-03-28T05:29:18.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:17 smithi098 ceph-mon[36087]: pgmap v174: 129 pgs: 129 active+clean; 202 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.5 MiB/s wr, 852 op/s 2024-03-28T05:29:18.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:17 smithi113 ceph-mon[37164]: pgmap v174: 129 pgs: 129 active+clean; 202 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.5 MiB/s wr, 852 op/s 2024-03-28T05:29:20.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:19 smithi023 ceph-mon[31954]: pgmap v175: 129 pgs: 129 active+clean; 202 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 747 op/s 2024-03-28T05:29:20.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:19 smithi098 ceph-mon[36087]: pgmap v175: 129 pgs: 129 active+clean; 202 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 747 op/s 2024-03-28T05:29:20.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:19 smithi113 ceph-mon[37164]: pgmap v175: 129 pgs: 129 active+clean; 202 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 747 op/s 2024-03-28T05:29:22.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:21 smithi023 ceph-mon[31954]: pgmap v176: 129 pgs: 129 active+clean; 205 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.2 MiB/s wr, 788 op/s 2024-03-28T05:29:22.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:21 smithi098 ceph-mon[36087]: pgmap v176: 129 pgs: 129 active+clean; 205 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.2 MiB/s wr, 788 op/s 2024-03-28T05:29:22.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:21 smithi113 ceph-mon[37164]: pgmap v176: 129 pgs: 129 active+clean; 205 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.2 MiB/s wr, 788 op/s 2024-03-28T05:29:24.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:23 smithi023 ceph-mon[31954]: pgmap v177: 129 pgs: 129 active+clean; 204 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 783 op/s 2024-03-28T05:29:24.045 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:23 smithi113 ceph-mon[37164]: pgmap v177: 129 pgs: 129 active+clean; 204 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 783 op/s 2024-03-28T05:29:24.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:23 smithi098 ceph-mon[36087]: pgmap v177: 129 pgs: 129 active+clean; 204 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.8 MiB/s wr, 783 op/s 2024-03-28T05:29:26.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:25 smithi023 ceph-mon[31954]: pgmap v178: 129 pgs: 129 active+clean; 204 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 730 op/s 2024-03-28T05:29:26.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:25 smithi098 ceph-mon[36087]: pgmap v178: 129 pgs: 129 active+clean; 204 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 730 op/s 2024-03-28T05:29:26.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:25 smithi113 ceph-mon[37164]: pgmap v178: 129 pgs: 129 active+clean; 204 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 730 op/s 2024-03-28T05:29:26.401 INFO:tasks.workunit.client.0.smithi023.stdout: 63 110738 102 73969 104 58236 226 2024-03-28T05:29:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.792+0000 7f275e09a640 1 -- 172.21.15.23:0/2177206011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27581011b0 msgr2=0x7f275810d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.792+0000 7f275e09a640 1 --2- 172.21.15.23:0/2177206011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27581011b0 0x7f275810d3d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f2750009ca0 tx=0x7f275002c100 comp rx=0 tx=0).stop 2024-03-28T05:29:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.792+0000 7f275e09a640 1 -- 172.21.15.23:0/2177206011 shutdown_connections 2024-03-28T05:29:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.792+0000 7f275e09a640 1 --2- 172.21.15.23:0/2177206011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27581011b0 0x7f275810d3d0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.795 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.792+0000 7f275e09a640 1 --2- 172.21.15.23:0/2177206011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27581007f0 0x7f2758100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.792+0000 7f275e09a640 1 --2- 172.21.15.23:0/2177206011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2758105540 0x7f2758105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.792+0000 7f275e09a640 1 -- 172.21.15.23:0/2177206011 >> 172.21.15.23:0/2177206011 conn(0x7f27580773f0 msgr2=0x7f2758077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.792+0000 7f275e09a640 1 -- 172.21.15.23:0/2177206011 shutdown_connections 2024-03-28T05:29:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.792+0000 7f275e09a640 1 -- 172.21.15.23:0/2177206011 wait complete. 2024-03-28T05:29:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.793+0000 7f275e09a640 1 Processor -- start 2024-03-28T05:29:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.793+0000 7f275e09a640 1 -- start start 2024-03-28T05:29:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.793+0000 7f275e09a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27581007f0 0x7f2758197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.793+0000 7f275e09a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27581011b0 0x7f2758198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.793+0000 7f275e09a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2758105540 0x7f275819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.793+0000 7f275e09a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2758110a80 con 0x7f2758105540 2024-03-28T05:29:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.793+0000 7f275e09a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2758110900 con 0x7f27581011b0 2024-03-28T05:29:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.793+0000 7f275e09a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2758110c00 con 0x7f27581007f0 2024-03-28T05:29:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.793+0000 7f2757fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2758105540 0x7f275819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.793+0000 7f27577fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27581007f0 0x7f2758197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.793+0000 7f2757fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2758105540 0x7f275819c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40964/0 (socket says 172.21.15.23:40964) 2024-03-28T05:29:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.793+0000 7f2757fff640 1 -- 172.21.15.23:0/641877695 learned_addr learned my addr 172.21.15.23:0/641877695 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:29:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.794+0000 7f2756ffd640 1 --2- 172.21.15.23:0/641877695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27581011b0 0x7f2758198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.794+0000 7f27577fe640 1 -- 172.21.15.23:0/641877695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27581011b0 msgr2=0x7f2758198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.794+0000 7f27577fe640 1 --2- 172.21.15.23:0/641877695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27581011b0 0x7f2758198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.794+0000 7f27577fe640 1 -- 172.21.15.23:0/641877695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2758105540 msgr2=0x7f275819c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.794+0000 7f27577fe640 1 --2- 172.21.15.23:0/641877695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2758105540 0x7f275819c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.794+0000 7f27577fe640 1 -- 172.21.15.23:0/641877695 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f275819cfc0 con 0x7f27581007f0 2024-03-28T05:29:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.794+0000 7f2757fff640 1 --2- 172.21.15.23:0/641877695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2758105540 0x7f275819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:29:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.794+0000 7f27577fe640 1 --2- 172.21.15.23:0/641877695 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27581007f0 0x7f2758197fd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f274c00e9f0 tx=0x7f274c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.794+0000 7f2754ff9640 1 -- 172.21.15.23:0/641877695 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c00ccc0 con 0x7f27581007f0 2024-03-28T05:29:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.794+0000 7f275e09a640 1 -- 172.21.15.23:0/641877695 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f275819d2b0 con 0x7f27581007f0 2024-03-28T05:29:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.794+0000 7f2754ff9640 1 -- 172.21.15.23:0/641877695 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f274c005020 con 0x7f27581007f0 2024-03-28T05:29:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.794+0000 7f275e09a640 1 -- 172.21.15.23:0/641877695 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27581a4b40 con 0x7f27581007f0 2024-03-28T05:29:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.794+0000 7f2754ff9640 1 -- 172.21.15.23:0/641877695 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c004a00 con 0x7f27581007f0 2024-03-28T05:29:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.796+0000 7f2754ff9640 1 -- 172.21.15.23:0/641877695 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f274c004ba0 con 0x7f27581007f0 2024-03-28T05:29:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.797+0000 7f2754ff9640 1 --2- 172.21.15.23:0/641877695 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f27400a2a10 0x7f27400a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.797+0000 7f2756ffd640 1 --2- 172.21.15.23:0/641877695 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f27400a2a10 0x7f27400a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.797+0000 7f2760d65740 1 Processor -- start 2024-03-28T05:29:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.797+0000 7f2754ff9640 1 -- 172.21.15.23:0/641877695 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f274c0cf080 con 0x7f27581007f0 2024-03-28T05:29:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.797+0000 7f2760d65740 1 -- start start 2024-03-28T05:29:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.797+0000 7f2760d65740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c39710 0x555c80c39af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.797+0000 7f2760d65740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555c80c3a030 0x555c80c428c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.797+0000 7f2760d65740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c80c42e00 0x555c80c451f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.797+0000 7f2760d65740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c80b9db10 con 0x555c80c39710 2024-03-28T05:29:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.797+0000 7f2760d65740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c80b46900 con 0x555c80c42e00 2024-03-28T05:29:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.797+0000 7f2760d65740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c80b99bb0 con 0x555c80c3a030 2024-03-28T05:29:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.797+0000 7f2756ffd640 1 --2- 172.21.15.23:0/641877695 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f27400a2a10 0x7f27400a4ed0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2758077a40 tx=0x7f2748009110 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.797+0000 7f27577fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c39710 0x555c80c39af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.797+0000 7f27577fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c39710 0x555c80c39af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40980/0 (socket says 172.21.15.23:40980) 2024-03-28T05:29:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.798+0000 7f27577fe640 1 -- 172.21.15.23:0/3276131978 learned_addr learned my addr 172.21.15.23:0/3276131978 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:29:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.798+0000 7f2756ffd640 1 --2- 172.21.15.23:0/3276131978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555c80c3a030 0x555c80c428c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.798+0000 7f27577fe640 1 -- 172.21.15.23:0/3276131978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555c80c3a030 msgr2=0x555c80c428c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.798+0000 7f27577fe640 1 --2- 172.21.15.23:0/3276131978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555c80c3a030 0x555c80c428c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.798+0000 7f27577fe640 1 -- 172.21.15.23:0/3276131978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c80c42e00 msgr2=0x555c80c451f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.798+0000 7f27577fe640 1 --2- 172.21.15.23:0/3276131978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c80c42e00 0x555c80c451f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.798+0000 7f27577fe640 1 -- 172.21.15.23:0/3276131978 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c808b2210 con 0x555c80c39710 2024-03-28T05:29:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.798+0000 7f27577fe640 1 --2- 172.21.15.23:0/3276131978 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c39710 0x555c80c39af0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f274c097d20 tx=0x7f274c09bb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b293c9254046bd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.798+0000 7f273dffb640 1 -- 172.21.15.23:0/3276131978 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c01c040 con 0x555c80c39710 2024-03-28T05:29:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.798+0000 7f273dffb640 1 -- 172.21.15.23:0/3276131978 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f274c0b9e50 con 0x555c80c39710 2024-03-28T05:29:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.798+0000 7f273dffb640 1 -- 172.21.15.23:0/3276131978 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c0a3660 con 0x555c80c39710 2024-03-28T05:29:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.799+0000 7f2760d65740 1 -- 172.21.15.23:0/3276131978 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c39710 msgr2=0x555c80c39af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.799+0000 7f2760d65740 1 --2- 172.21.15.23:0/3276131978 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c39710 0x555c80c39af0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f274c097d20 tx=0x7f274c09bb90 comp rx=0 tx=0).stop 2024-03-28T05:29:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.799+0000 7f2760d65740 1 -- 172.21.15.23:0/3276131978 shutdown_connections 2024-03-28T05:29:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.799+0000 7f2760d65740 1 --2- 172.21.15.23:0/3276131978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c80c42e00 0x555c80c451f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.799+0000 7f2760d65740 1 --2- 172.21.15.23:0/3276131978 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555c80c3a030 0x555c80c428c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.799+0000 7f2760d65740 1 --2- 172.21.15.23:0/3276131978 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c39710 0x555c80c39af0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.799+0000 7f2760d65740 1 -- 172.21.15.23:0/3276131978 >> 172.21.15.23:0/3276131978 conn(0x555c80c34030 msgr2=0x555c80c34410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.799+0000 7f2760d65740 1 -- 172.21.15.23:0/3276131978 shutdown_connections 2024-03-28T05:29:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.799+0000 7f2760d65740 1 -- 172.21.15.23:0/3276131978 wait complete. 2024-03-28T05:29:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.799+0000 7f2760d65740 1 -- start start 2024-03-28T05:29:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.799+0000 7f2760d65740 1 Processor -- start 2024-03-28T05:29:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.799+0000 7f2760d65740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c80c39710 0x555c80c3fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.799+0000 7f2760d65740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555c80c3a030 0x555c80c403a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.799+0000 7f2760d65740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c42e00 0x555c80c408e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.799+0000 7f2760d65740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c80c45a30 con 0x555c80c42e00 2024-03-28T05:29:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.800+0000 7f2760d65740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c80b59d50 con 0x555c80c39710 2024-03-28T05:29:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.800+0000 7f2760d65740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c80c45bb0 con 0x555c80c3a030 2024-03-28T05:29:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.800+0000 7f2760d65740 10 client.0 _ll_register_callbacks cb 0x555c80b59720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:29:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.800+0000 7f2757fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c42e00 0x555c80c408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.800+0000 7f2757fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c42e00 0x555c80c408e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40984/0 (socket says 172.21.15.23:40984) 2024-03-28T05:29:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.800+0000 7f2757fff640 1 -- 172.21.15.23:0/1123833054 learned_addr learned my addr 172.21.15.23:0/1123833054 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:29:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.800+0000 7f2756ffd640 1 --2- 172.21.15.23:0/1123833054 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555c80c3a030 0x555c80c403a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.800+0000 7f27577fe640 1 --2- 172.21.15.23:0/1123833054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c80c39710 0x555c80c3fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.800+0000 7f2757fff640 1 -- 172.21.15.23:0/1123833054 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555c80c3a030 msgr2=0x555c80c403a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.800+0000 7f2757fff640 1 --2- 172.21.15.23:0/1123833054 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555c80c3a030 0x555c80c403a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.800+0000 7f2757fff640 1 -- 172.21.15.23:0/1123833054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c80c39710 msgr2=0x555c80c3fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.800+0000 7f2757fff640 1 --2- 172.21.15.23:0/1123833054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c80c39710 0x555c80c3fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.800+0000 7f2757fff640 1 -- 172.21.15.23:0/1123833054 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555c8097e620 con 0x555c80c42e00 2024-03-28T05:29:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.800+0000 7f27577fe640 1 --2- 172.21.15.23:0/1123833054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c80c39710 0x555c80c3fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:29:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.800+0000 7f2757fff640 1 --2- 172.21.15.23:0/1123833054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c42e00 0x555c80c408e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f2750009430 tx=0x7f2750009930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eda0e6e406ad2fca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.800+0000 7f2736ffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:29:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.800+0000 7f2736ffd640 1 -- 172.21.15.23:0/1123833054 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2750045020 con 0x555c80c42e00 2024-03-28T05:29:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.800+0000 7f2736ffd640 1 -- 172.21.15.23:0/1123833054 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f2750034440 con 0x555c80c42e00 2024-03-28T05:29:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.800+0000 7f2760d65740 1 -- 172.21.15.23:0/1123833054 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f274c0b9e50 con 0x555c80c42e00 2024-03-28T05:29:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.801+0000 7f2736ffd640 1 -- 172.21.15.23:0/1123833054 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2750050050 con 0x555c80c42e00 2024-03-28T05:29:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.801+0000 7f2760d65740 10 client.15942 fetch_fsmap learned FSMap version 20 2024-03-28T05:29:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.801+0000 7f2760d65740 1 -- 172.21.15.23:0/1123833054 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x555c80c51d30 con 0x555c80c42e00 2024-03-28T05:29:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.801+0000 7f2736ffd640 1 -- 172.21.15.23:0/1123833054 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f274c0b9e50 con 0x555c80c42e00 2024-03-28T05:29:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.801+0000 7f2736ffd640 1 -- 172.21.15.23:0/1123833054 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f2750004160 con 0x555c80c42e00 2024-03-28T05:29:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.801+0000 7f2760d65740 10 client.15942 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:29:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.801+0000 7f2760d65740 10 client.15942 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:29:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.801+0000 7f2760d65740 20 client.15942 populate_metadata read hostname 'smithi023' 2024-03-28T05:29:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.801+0000 7f2760d65740 1 --2- 172.21.15.23:0/1123833054 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x555c80c53240 0x555c80c55680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.801+0000 7f2760d65740 4 client.15942 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:29:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.801+0000 7f2760d65740 1 -- 172.21.15.23:0/1123833054 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555c80bab050 con 0x555c80c53240 2024-03-28T05:29:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.801+0000 7f27577fe640 1 --2- 172.21.15.23:0/1123833054 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x555c80c53240 0x555c80c55680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.802+0000 7f27577fe640 1 --2- 172.21.15.23:0/1123833054 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x555c80c53240 0x555c80c55680 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4040ad072b3de187 server_cookie=5e0ea37d2de5e1db in_seq=0 out_seq=0 2024-03-28T05:29:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.802+0000 7f2736ffd640 10 client.15942 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:29:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.803+0000 7f2736ffd640 1 -- 172.21.15.23:0/1123833054 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555c80bab050 con 0x555c80c53240 2024-03-28T05:29:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.803+0000 7f2736ffd640 10 client.15942 handle_command_reply: tid=0 2024-03-28T05:29:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.803+0000 7f2760d65740 1 client.15942 shutdown 2024-03-28T05:29:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.803+0000 7f2760d65740 1 -- 172.21.15.23:0/1123833054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c42e00 msgr2=0x555c80c408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.803+0000 7f2760d65740 1 --2- 172.21.15.23:0/1123833054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c42e00 0x555c80c408e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f2750009430 tx=0x7f2750009930 comp rx=0 tx=0).stop 2024-03-28T05:29:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.803+0000 7f2760d65740 1 -- 172.21.15.23:0/1123833054 shutdown_connections 2024-03-28T05:29:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.803+0000 7f2760d65740 1 --2- 172.21.15.23:0/1123833054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c42e00 0x555c80c408e0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.803+0000 7f2760d65740 1 --2- 172.21.15.23:0/1123833054 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555c80c3a030 0x555c80c403a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.803+0000 7f2760d65740 1 --2- 172.21.15.23:0/1123833054 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x555c80c53240 0x555c80c55680 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.803+0000 7f2760d65740 1 --2- 172.21.15.23:0/1123833054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c80c39710 0x555c80c3fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.803+0000 7f2736ffd640 0 client.15942 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:29:26.811 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.803+0000 7f2760d65740 1 -- 172.21.15.23:0/1123833054 >> 172.21.15.23:0/1123833054 conn(0x555c80c34030 msgr2=0x555c80c35d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:26.811 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.803+0000 7f2760d65740 1 -- 172.21.15.23:0/1123833054 shutdown_connections 2024-03-28T05:29:26.811 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.803+0000 7f2760d65740 1 -- 172.21.15.23:0/1123833054 wait complete. 2024-03-28T05:29:26.811 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.803+0000 7f2760d65740 20 client.15942 trim_cache size 0 max 16384 2024-03-28T05:29:26.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.818+0000 7f2760d65740 1 Processor -- start 2024-03-28T05:29:26.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.818+0000 7f2760d65740 1 -- start start 2024-03-28T05:29:26.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.818+0000 7f2760d65740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c39710 0x555c80c42590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:26.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.818+0000 7f2760d65740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555c80c3a030 0x555c80c4e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:26.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.818+0000 7f2760d65740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c80c42e00 0x555c80c4e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:26.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.818+0000 7f2760d65740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274400df00 con 0x555c80c39710 2024-03-28T05:29:26.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.818+0000 7f2760d65740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27440104c0 con 0x555c80c42e00 2024-03-28T05:29:26.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.818+0000 7f2760d65740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274400e500 con 0x555c80c3a030 2024-03-28T05:29:26.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.818+0000 7f27577fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c39710 0x555c80c42590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:26.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.818+0000 7f27577fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c39710 0x555c80c42590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40986/0 (socket says 172.21.15.23:40986) 2024-03-28T05:29:26.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.818+0000 7f27577fe640 1 -- 172.21.15.23:0/3327396285 learned_addr learned my addr 172.21.15.23:0/3327396285 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:29:26.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.818+0000 7f2756ffd640 1 --2- 172.21.15.23:0/3327396285 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555c80c3a030 0x555c80c4e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:26.821 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.818+0000 7f2757fff640 1 --2- 172.21.15.23:0/3327396285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c80c42e00 0x555c80c4e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.819+0000 7f27577fe640 1 -- 172.21.15.23:0/3327396285 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555c80c3a030 msgr2=0x555c80c4e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.819+0000 7f27577fe640 1 --2- 172.21.15.23:0/3327396285 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555c80c3a030 0x555c80c4e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.819+0000 7f27577fe640 1 -- 172.21.15.23:0/3327396285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c80c42e00 msgr2=0x555c80c4e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.819+0000 7f27577fe640 1 --2- 172.21.15.23:0/3327396285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c80c42e00 0x555c80c4e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.819+0000 7f27577fe640 1 -- 172.21.15.23:0/3327396285 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c80c39e60 con 0x555c80c39710 2024-03-28T05:29:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.819+0000 7f2757fff640 1 --2- 172.21.15.23:0/3327396285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c80c42e00 0x555c80c4e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:29:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.819+0000 7f27577fe640 1 --2- 172.21.15.23:0/3327396285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c39710 0x555c80c42590 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f274c0cc4f0 tx=0x7f274c09f5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e286fbbe8dde0ee server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.819+0000 7f273d7fa640 1 -- 172.21.15.23:0/3327396285 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c01c070 con 0x555c80c39710 2024-03-28T05:29:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.819+0000 7f273d7fa640 1 -- 172.21.15.23:0/3327396285 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f274c0b9e50 con 0x555c80c39710 2024-03-28T05:29:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.819+0000 7f273d7fa640 1 -- 172.21.15.23:0/3327396285 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c0804b0 con 0x555c80c39710 2024-03-28T05:29:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.820+0000 7f2760d65740 1 -- 172.21.15.23:0/3327396285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c39710 msgr2=0x555c80c42590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.820+0000 7f2760d65740 1 --2- 172.21.15.23:0/3327396285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c39710 0x555c80c42590 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f274c0cc4f0 tx=0x7f274c09f5e0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.820+0000 7f2760d65740 1 -- 172.21.15.23:0/3327396285 shutdown_connections 2024-03-28T05:29:26.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.820+0000 7f2760d65740 1 --2- 172.21.15.23:0/3327396285 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c80c42e00 0x555c80c4e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.820+0000 7f2760d65740 1 --2- 172.21.15.23:0/3327396285 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555c80c3a030 0x555c80c4e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.820+0000 7f2760d65740 1 --2- 172.21.15.23:0/3327396285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c39710 0x555c80c42590 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.820+0000 7f2760d65740 1 -- 172.21.15.23:0/3327396285 >> 172.21.15.23:0/3327396285 conn(0x555c80c34030 msgr2=0x555c80c34410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.820+0000 7f2760d65740 1 -- 172.21.15.23:0/3327396285 shutdown_connections 2024-03-28T05:29:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.820+0000 7f2760d65740 1 -- 172.21.15.23:0/3327396285 wait complete. 2024-03-28T05:29:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.820+0000 7f2760d65740 1 -- start start 2024-03-28T05:29:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.821+0000 7f2760d65740 1 Processor -- start 2024-03-28T05:29:26.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.821+0000 7f2760d65740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555c80c39710 0x555c80c464d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.821+0000 7f2760d65740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c3a030 0x555c80c46a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.821+0000 7f2760d65740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c80c42e00 0x555c80cadb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.821+0000 7f2760d65740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274400eb40 con 0x555c80c3a030 2024-03-28T05:29:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.821+0000 7f2760d65740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274400fe60 con 0x555c80c42e00 2024-03-28T05:29:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.821+0000 7f2760d65740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274400f1a0 con 0x555c80c39710 2024-03-28T05:29:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.821+0000 7f2760d65740 10 client.0 _ll_register_callbacks cb 0x7f2750009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:29:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.821+0000 7f2756ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c3a030 0x555c80c46a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.821+0000 7f2756ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c3a030 0x555c80c46a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41000/0 (socket says 172.21.15.23:41000) 2024-03-28T05:29:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.821+0000 7f2756ffd640 1 -- 172.21.15.23:0/1425718210 learned_addr learned my addr 172.21.15.23:0/1425718210 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:29:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.821+0000 7f27577fe640 1 --2- 172.21.15.23:0/1425718210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555c80c39710 0x555c80c464d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.821+0000 7f2757fff640 1 --2- 172.21.15.23:0/1425718210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c80c42e00 0x555c80cadb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.821+0000 7f2756ffd640 1 -- 172.21.15.23:0/1425718210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555c80c39710 msgr2=0x555c80c464d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.821+0000 7f2756ffd640 1 --2- 172.21.15.23:0/1425718210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555c80c39710 0x555c80c464d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.821+0000 7f2756ffd640 1 -- 172.21.15.23:0/1425718210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c80c42e00 msgr2=0x555c80cadb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.821+0000 7f2756ffd640 1 --2- 172.21.15.23:0/1425718210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c80c42e00 0x555c80cadb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.821+0000 7f2756ffd640 1 -- 172.21.15.23:0/1425718210 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555c80c472a0 con 0x555c80c3a030 2024-03-28T05:29:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.821+0000 7f2756ffd640 1 --2- 172.21.15.23:0/1425718210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c3a030 0x555c80c46a10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f2748009490 tx=0x7f27480047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6603ffaf893be69 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.821+0000 7f2736ffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:29:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.822+0000 7f2736ffd640 1 -- 172.21.15.23:0/1425718210 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274801b040 con 0x555c80c3a030 2024-03-28T05:29:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.822+0000 7f2736ffd640 1 -- 172.21.15.23:0/1425718210 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f27480139a0 con 0x555c80c3a030 2024-03-28T05:29:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.822+0000 7f2760d65740 1 -- 172.21.15.23:0/1425718210 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f274c0b9e50 con 0x555c80c3a030 2024-03-28T05:29:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.822+0000 7f2736ffd640 1 -- 172.21.15.23:0/1425718210 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2748013ba0 con 0x555c80c3a030 2024-03-28T05:29:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.822+0000 7f2760d65740 10 client.15948 fetch_fsmap learned FSMap version 20 2024-03-28T05:29:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.822+0000 7f2760d65740 1 -- 172.21.15.23:0/1425718210 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x555c80caf190 con 0x555c80c3a030 2024-03-28T05:29:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.822+0000 7f2736ffd640 1 -- 172.21.15.23:0/1425718210 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f274c0b9e50 con 0x555c80c3a030 2024-03-28T05:29:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.822+0000 7f2736ffd640 1 -- 172.21.15.23:0/1425718210 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f274802a070 con 0x555c80c3a030 2024-03-28T05:29:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.822+0000 7f2760d65740 10 client.15948 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:29:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.822+0000 7f2760d65740 10 client.15948 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:29:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.822+0000 7f2760d65740 20 client.15948 populate_metadata read hostname 'smithi023' 2024-03-28T05:29:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.822+0000 7f2760d65740 1 --2- 172.21.15.23:0/1425718210 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x555c80caf970 0x555c80cb1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.822+0000 7f2760d65740 4 client.15948 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:29:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.822+0000 7f2760d65740 1 -- 172.21.15.23:0/1425718210 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555c80ba53a0 con 0x555c80caf970 2024-03-28T05:29:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.822+0000 7f27577fe640 1 --2- 172.21.15.23:0/1425718210 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x555c80caf970 0x555c80cb1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.823+0000 7f2736ffd640 10 client.15948 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:29:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:26.823+0000 7f27577fe640 1 --2- 172.21.15.23:0/1425718210 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x555c80caf970 0x555c80cb1d90 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45aca6637d0f99a server_cookie=4778b3a1ecd122bd in_seq=0 out_seq=0 2024-03-28T05:29:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.270+0000 7f2736ffd640 1 -- 172.21.15.23:0/1425718210 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555c80ba53a0 con 0x555c80caf970 2024-03-28T05:29:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.270+0000 7f2736ffd640 10 client.15948 handle_command_reply: tid=0 2024-03-28T05:29:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.270+0000 7f2760d65740 1 client.15948 shutdown 2024-03-28T05:29:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.271+0000 7f2760d65740 1 -- 172.21.15.23:0/1425718210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c3a030 msgr2=0x555c80c46a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.271+0000 7f2760d65740 1 --2- 172.21.15.23:0/1425718210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c3a030 0x555c80c46a10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f2748009490 tx=0x7f27480047d0 comp rx=0 tx=0).stop 2024-03-28T05:29:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.271+0000 7f2760d65740 1 -- 172.21.15.23:0/1425718210 shutdown_connections 2024-03-28T05:29:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.271+0000 7f2760d65740 1 --2- 172.21.15.23:0/1425718210 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x555c80caf970 0x555c80cb1d90 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.271+0000 7f2760d65740 1 --2- 172.21.15.23:0/1425718210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555c80c42e00 0x555c80cadb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.271+0000 7f2736ffd640 0 client.15948 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:29:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.271+0000 7f2760d65740 1 --2- 172.21.15.23:0/1425718210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c80c3a030 0x555c80c46a10 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.271+0000 7f2760d65740 1 --2- 172.21.15.23:0/1425718210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555c80c39710 0x555c80c464d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.271+0000 7f2760d65740 1 -- 172.21.15.23:0/1425718210 >> 172.21.15.23:0/1425718210 conn(0x555c80c34030 msgr2=0x555c80c35410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.271+0000 7f2760d65740 1 -- 172.21.15.23:0/1425718210 shutdown_connections 2024-03-28T05:29:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.271+0000 7f2760d65740 1 -- 172.21.15.23:0/1425718210 wait complete. 2024-03-28T05:29:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.271+0000 7f2760d65740 20 client.15948 trim_cache size 0 max 16384 2024-03-28T05:29:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:29:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:29:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:29:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:29:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.272+0000 7f275e09a640 1 -- 172.21.15.23:0/641877695 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f27400a2a10 msgr2=0x7f27400a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.272+0000 7f275e09a640 1 --2- 172.21.15.23:0/641877695 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f27400a2a10 0x7f27400a4ed0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2758077a40 tx=0x7f2748009110 comp rx=0 tx=0).stop 2024-03-28T05:29:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.272+0000 7f275e09a640 1 -- 172.21.15.23:0/641877695 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27581007f0 msgr2=0x7f2758197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.272+0000 7f275e09a640 1 --2- 172.21.15.23:0/641877695 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27581007f0 0x7f2758197fd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f274c00e9f0 tx=0x7f274c00edf0 comp rx=0 tx=0).stop 2024-03-28T05:29:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.272+0000 7f275e09a640 1 -- 172.21.15.23:0/641877695 shutdown_connections 2024-03-28T05:29:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.272+0000 7f275e09a640 1 --2- 172.21.15.23:0/641877695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2758105540 0x7f275819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.272+0000 7f275e09a640 1 --2- 172.21.15.23:0/641877695 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f27400a2a10 0x7f27400a4ed0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.272+0000 7f275e09a640 1 --2- 172.21.15.23:0/641877695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27581011b0 0x7f2758198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.272+0000 7f275e09a640 1 --2- 172.21.15.23:0/641877695 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27581007f0 0x7f2758197fd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.272+0000 7f275e09a640 1 -- 172.21.15.23:0/641877695 >> 172.21.15.23:0/641877695 conn(0x7f27580773f0 msgr2=0x7f2758068f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.273+0000 7f275e09a640 1 -- 172.21.15.23:0/641877695 shutdown_connections 2024-03-28T05:29:27.275 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:27.273+0000 7f275e09a640 1 -- 172.21.15.23:0/641877695 wait complete. 2024-03-28T05:29:28.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:27 smithi023 ceph-mon[31954]: pgmap v179: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 772 op/s 2024-03-28T05:29:28.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:27 smithi098 ceph-mon[36087]: pgmap v179: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 772 op/s 2024-03-28T05:29:28.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:27 smithi113 ceph-mon[37164]: pgmap v179: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.6 MiB/s wr, 772 op/s 2024-03-28T05:29:30.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:29 smithi023 ceph-mon[31954]: pgmap v180: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 555 KiB/s wr, 117 op/s 2024-03-28T05:29:30.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:29 smithi098 ceph-mon[36087]: pgmap v180: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 555 KiB/s wr, 117 op/s 2024-03-28T05:29:30.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:29 smithi113 ceph-mon[37164]: pgmap v180: 129 pgs: 129 active+clean; 207 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 555 KiB/s wr, 117 op/s 2024-03-28T05:29:32.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:31 smithi023 ceph-mon[31954]: pgmap v181: 129 pgs: 129 active+clean; 206 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 848 KiB/s wr, 252 op/s 2024-03-28T05:29:32.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:31 smithi098 ceph-mon[36087]: pgmap v181: 129 pgs: 129 active+clean; 206 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 848 KiB/s wr, 252 op/s 2024-03-28T05:29:32.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:31 smithi113 ceph-mon[37164]: pgmap v181: 129 pgs: 129 active+clean; 206 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 848 KiB/s wr, 252 op/s 2024-03-28T05:29:34.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:33 smithi023 ceph-mon[31954]: pgmap v182: 129 pgs: 129 active+clean; 202 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 562 KiB/s wr, 378 op/s 2024-03-28T05:29:34.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:33 smithi098 ceph-mon[36087]: pgmap v182: 129 pgs: 129 active+clean; 202 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 562 KiB/s wr, 378 op/s 2024-03-28T05:29:34.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:33 smithi113 ceph-mon[37164]: pgmap v182: 129 pgs: 129 active+clean; 202 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 562 KiB/s wr, 378 op/s 2024-03-28T05:29:36.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:35 smithi023 ceph-mon[31954]: pgmap v183: 129 pgs: 129 active+clean; 202 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 555 KiB/s wr, 344 op/s 2024-03-28T05:29:36.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:35 smithi098 ceph-mon[36087]: pgmap v183: 129 pgs: 129 active+clean; 202 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 555 KiB/s wr, 344 op/s 2024-03-28T05:29:36.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:35 smithi113 ceph-mon[37164]: pgmap v183: 129 pgs: 129 active+clean; 202 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 555 KiB/s wr, 344 op/s 2024-03-28T05:29:36.401 INFO:tasks.workunit.client.0.smithi023.stdout: 64 357852 163 238044 65 182858 242 2024-03-28T05:29:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:37 smithi098 ceph-mon[36087]: pgmap v184: 129 pgs: 129 active+clean; 204 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 826 KiB/s wr, 376 op/s 2024-03-28T05:29:38.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:37 smithi113 ceph-mon[37164]: pgmap v184: 129 pgs: 129 active+clean; 204 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 826 KiB/s wr, 376 op/s 2024-03-28T05:29:38.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:37 smithi023 ceph-mon[31954]: pgmap v184: 129 pgs: 129 active+clean; 204 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 826 KiB/s wr, 376 op/s 2024-03-28T05:29:40.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:39 smithi098 ceph-mon[36087]: pgmap v185: 129 pgs: 129 active+clean; 204 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 597 KiB/s wr, 334 op/s 2024-03-28T05:29:40.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:39 smithi113 ceph-mon[37164]: pgmap v185: 129 pgs: 129 active+clean; 204 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 597 KiB/s wr, 334 op/s 2024-03-28T05:29:40.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:39 smithi023 ceph-mon[31954]: pgmap v185: 129 pgs: 129 active+clean; 204 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 597 KiB/s wr, 334 op/s 2024-03-28T05:29:40.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.367+0000 7f7cf57c4640 1 -- 172.21.15.23:0/3794783120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf01007f0 msgr2=0x7f7cf0100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.367+0000 7f7cf57c4640 1 --2- 172.21.15.23:0/3794783120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf01007f0 0x7f7cf0100c70 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8009900 tx=0x7f7cd802bc30 comp rx=0 tx=0).stop 2024-03-28T05:29:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.367+0000 7f7cf57c4640 1 -- 172.21.15.23:0/3794783120 shutdown_connections 2024-03-28T05:29:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.367+0000 7f7cf57c4640 1 --2- 172.21.15.23:0/3794783120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cf01011b0 0x7f7cf010d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.367+0000 7f7cf57c4640 1 --2- 172.21.15.23:0/3794783120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf01007f0 0x7f7cf0100c70 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.367+0000 7f7cf57c4640 1 --2- 172.21.15.23:0/3794783120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7cf0105540 0x7f7cf0105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.367+0000 7f7cf57c4640 1 -- 172.21.15.23:0/3794783120 >> 172.21.15.23:0/3794783120 conn(0x7f7cf00773f0 msgr2=0x7f7cf0077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.367+0000 7f7cf57c4640 1 -- 172.21.15.23:0/3794783120 shutdown_connections 2024-03-28T05:29:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.368+0000 7f7cf57c4640 1 -- 172.21.15.23:0/3794783120 wait complete. 2024-03-28T05:29:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.368+0000 7f7cf57c4640 1 Processor -- start 2024-03-28T05:29:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.368+0000 7f7cf57c4640 1 -- start start 2024-03-28T05:29:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.368+0000 7f7cf57c4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cf01007f0 0x7f7cf0197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.369+0000 7f7cf57c4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf01011b0 0x7f7cf0198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.369+0000 7f7cf57c4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7cf0105540 0x7f7cf019c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.369+0000 7f7cf57c4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf0110a60 con 0x7f7cf01011b0 2024-03-28T05:29:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.369+0000 7f7cf57c4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf01108e0 con 0x7f7cf01007f0 2024-03-28T05:29:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.369+0000 7f7cf57c4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf0110be0 con 0x7f7cf0105540 2024-03-28T05:29:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.369+0000 7f7cee7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf01011b0 0x7f7cf0198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.369+0000 7f7cee7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf01011b0 0x7f7cf0198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45784/0 (socket says 172.21.15.23:45784) 2024-03-28T05:29:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.369+0000 7f7cee7fc640 1 -- 172.21.15.23:0/1532860109 learned_addr learned my addr 172.21.15.23:0/1532860109 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:29:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.369+0000 7f7cef7fe640 1 --2- 172.21.15.23:0/1532860109 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7cf0105540 0x7f7cf019c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.369+0000 7f7ceeffd640 1 --2- 172.21.15.23:0/1532860109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cf01007f0 0x7f7cf0197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.369+0000 7f7cef7fe640 1 -- 172.21.15.23:0/1532860109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cf01007f0 msgr2=0x7f7cf0197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.369+0000 7f7cef7fe640 1 --2- 172.21.15.23:0/1532860109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cf01007f0 0x7f7cf0197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.369+0000 7f7cef7fe640 1 -- 172.21.15.23:0/1532860109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf01011b0 msgr2=0x7f7cf0198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.369+0000 7f7cef7fe640 1 --2- 172.21.15.23:0/1532860109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf01011b0 0x7f7cf0198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.369+0000 7f7cef7fe640 1 -- 172.21.15.23:0/1532860109 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf019cf80 con 0x7f7cf0105540 2024-03-28T05:29:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.369+0000 7f7cee7fc640 1 --2- 172.21.15.23:0/1532860109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf01011b0 0x7f7cf0198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:29:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.369+0000 7f7cef7fe640 1 --2- 172.21.15.23:0/1532860109 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7cf0105540 0x7f7cf019c850 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0005fd0 tx=0x7f7ce000ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.369+0000 7f7ce7fff640 1 -- 172.21.15.23:0/1532860109 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce000cd80 con 0x7f7cf0105540 2024-03-28T05:29:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.369+0000 7f7cf57c4640 1 -- 172.21.15.23:0/1532860109 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf019d270 con 0x7f7cf0105540 2024-03-28T05:29:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.369+0000 7f7ce7fff640 1 -- 172.21.15.23:0/1532860109 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ce0005020 con 0x7f7cf0105540 2024-03-28T05:29:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.370+0000 7f7ce7fff640 1 -- 172.21.15.23:0/1532860109 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce00106c0 con 0x7f7cf0105540 2024-03-28T05:29:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.370+0000 7f7cf57c4640 1 -- 172.21.15.23:0/1532860109 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf01a4b00 con 0x7f7cf0105540 2024-03-28T05:29:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.372+0000 7f7ce7fff640 1 -- 172.21.15.23:0/1532860109 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f7ce00051d0 con 0x7f7cf0105540 2024-03-28T05:29:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.372+0000 7f7cf83ad740 1 Processor -- start 2024-03-28T05:29:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.372+0000 7f7cf83ad740 1 -- start start 2024-03-28T05:29:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.372+0000 7f7cf83ad740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac89710 0x555d7ac89af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7cf83ad740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d7ac8a030 0x555d7ac928c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7cf83ad740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d7ac92e00 0x555d7ac951f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7cf83ad740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d7abedb10 con 0x555d7ac89710 2024-03-28T05:29:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7ce7fff640 1 --2- 172.21.15.23:0/1532860109 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f7cdc0a2a10 0x7f7cdc0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7cf83ad740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d7ab96900 con 0x555d7ac92e00 2024-03-28T05:29:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7cf83ad740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d7abe9bb0 con 0x555d7ac8a030 2024-03-28T05:29:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7cee7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac89710 0x555d7ac89af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7cee7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac89710 0x555d7ac89af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45786/0 (socket says 172.21.15.23:45786) 2024-03-28T05:29:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7cee7fc640 1 -- 172.21.15.23:0/2756185843 learned_addr learned my addr 172.21.15.23:0/2756185843 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:29:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7ceeffd640 1 --2- 172.21.15.23:0/2756185843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d7ac8a030 0x555d7ac928c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7cef7fe640 1 --2- 172.21.15.23:0/1532860109 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f7cdc0a2a10 0x7f7cdc0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7ce7fff640 1 -- 172.21.15.23:0/1532860109 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f7ce00cf7b0 con 0x7f7cf0105540 2024-03-28T05:29:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7cee7fc640 1 --2- 172.21.15.23:0/2756185843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d7ac92e00 0x555d7ac951f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7cee7fc640 1 -- 172.21.15.23:0/2756185843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d7ac8a030 msgr2=0x555d7ac928c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7cee7fc640 1 --2- 172.21.15.23:0/2756185843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d7ac8a030 0x555d7ac928c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7cee7fc640 1 -- 172.21.15.23:0/2756185843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d7ac92e00 msgr2=0x555d7ac951f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7cee7fc640 1 --2- 172.21.15.23:0/2756185843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d7ac92e00 0x555d7ac951f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7cee7fc640 1 -- 172.21.15.23:0/2756185843 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d7a902210 con 0x555d7ac89710 2024-03-28T05:29:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7cee7fc640 1 --2- 172.21.15.23:0/2756185843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac89710 0x555d7ac89af0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8005c40 tx=0x7f7cd8005b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608090492f00bb55 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7cef7fe640 1 --2- 172.21.15.23:0/1532860109 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f7cdc0a2a10 0x7f7cdc0a4ed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7ce00cd170 tx=0x7f7ce006d6b0 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7ce57fa640 1 -- 172.21.15.23:0/2756185843 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8049070 con 0x555d7ac89710 2024-03-28T05:29:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7ce57fa640 1 -- 172.21.15.23:0/2756185843 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7cd802c8b0 con 0x555d7ac89710 2024-03-28T05:29:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.373+0000 7f7ce57fa640 1 -- 172.21.15.23:0/2756185843 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd80053c0 con 0x555d7ac89710 2024-03-28T05:29:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.374+0000 7f7cf83ad740 1 -- 172.21.15.23:0/2756185843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac89710 msgr2=0x555d7ac89af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.374+0000 7f7cf83ad740 1 --2- 172.21.15.23:0/2756185843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac89710 0x555d7ac89af0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8005c40 tx=0x7f7cd8005b70 comp rx=0 tx=0).stop 2024-03-28T05:29:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.374+0000 7f7cf83ad740 1 -- 172.21.15.23:0/2756185843 shutdown_connections 2024-03-28T05:29:40.382 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.374+0000 7f7cf83ad740 1 --2- 172.21.15.23:0/2756185843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d7ac92e00 0x555d7ac951f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.382 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.374+0000 7f7cf83ad740 1 --2- 172.21.15.23:0/2756185843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d7ac8a030 0x555d7ac928c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.382 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.374+0000 7f7cf83ad740 1 --2- 172.21.15.23:0/2756185843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac89710 0x555d7ac89af0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.382 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.374+0000 7f7cf83ad740 1 -- 172.21.15.23:0/2756185843 >> 172.21.15.23:0/2756185843 conn(0x555d7ac84030 msgr2=0x555d7ac84410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:40.382 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.374+0000 7f7cf83ad740 1 -- 172.21.15.23:0/2756185843 shutdown_connections 2024-03-28T05:29:40.382 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.374+0000 7f7cf83ad740 1 -- 172.21.15.23:0/2756185843 wait complete. 2024-03-28T05:29:40.382 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.375+0000 7f7cf83ad740 1 -- start start 2024-03-28T05:29:40.382 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.375+0000 7f7cf83ad740 1 Processor -- start 2024-03-28T05:29:40.382 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.375+0000 7f7cf83ad740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d7ac89710 0x555d7ac8fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:40.382 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.375+0000 7f7cf83ad740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d7ac8a030 0x555d7ac903a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:40.382 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.375+0000 7f7cf83ad740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac92e00 0x555d7ac908e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:40.382 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.375+0000 7f7cf83ad740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d7ac959d0 con 0x555d7ac92e00 2024-03-28T05:29:40.382 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.375+0000 7f7cf83ad740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d7aba9d50 con 0x555d7ac89710 2024-03-28T05:29:40.382 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.375+0000 7f7cf83ad740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d7ac95b50 con 0x555d7ac8a030 2024-03-28T05:29:40.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.375+0000 7f7cf83ad740 10 client.0 _ll_register_callbacks cb 0x555d7aba9720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:29:40.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.375+0000 7f7cee7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac92e00 0x555d7ac908e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:40.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.375+0000 7f7cee7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac92e00 0x555d7ac908e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45790/0 (socket says 172.21.15.23:45790) 2024-03-28T05:29:40.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.375+0000 7f7cee7fc640 1 -- 172.21.15.23:0/3425933147 learned_addr learned my addr 172.21.15.23:0/3425933147 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:29:40.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.375+0000 7f7ceeffd640 1 --2- 172.21.15.23:0/3425933147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d7ac8a030 0x555d7ac903a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:40.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.375+0000 7f7cee7fc640 1 --2- 172.21.15.23:0/3425933147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d7ac89710 0x555d7ac8fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:40.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.375+0000 7f7cee7fc640 1 -- 172.21.15.23:0/3425933147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d7ac8a030 msgr2=0x555d7ac903a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:40.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.375+0000 7f7cee7fc640 1 --2- 172.21.15.23:0/3425933147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d7ac8a030 0x555d7ac903a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.375+0000 7f7cee7fc640 1 -- 172.21.15.23:0/3425933147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d7ac89710 msgr2=0x555d7ac8fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:40.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.375+0000 7f7cee7fc640 1 --2- 172.21.15.23:0/3425933147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d7ac89710 0x555d7ac8fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.375+0000 7f7cee7fc640 1 -- 172.21.15.23:0/3425933147 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555d7a9ce620 con 0x555d7ac92e00 2024-03-28T05:29:40.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.375+0000 7f7cee7fc640 1 --2- 172.21.15.23:0/3425933147 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac92e00 0x555d7ac908e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8009850 tx=0x7f7cd802e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=183750844e316e4b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:40.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.375+0000 7f7cd67fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:29:40.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.376+0000 7f7cd67fc640 1 -- 172.21.15.23:0/3425933147 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8049040 con 0x555d7ac92e00 2024-03-28T05:29:40.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.376+0000 7f7cd67fc640 1 -- 172.21.15.23:0/3425933147 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f7cd8005550 con 0x555d7ac92e00 2024-03-28T05:29:40.383 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.376+0000 7f7cf83ad740 1 -- 172.21.15.23:0/3425933147 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7cd802c8b0 con 0x555d7ac92e00 2024-03-28T05:29:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.376+0000 7f7cd67fc640 1 -- 172.21.15.23:0/3425933147 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7cd803d450 con 0x555d7ac92e00 2024-03-28T05:29:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.376+0000 7f7cf83ad740 10 client.15960 fetch_fsmap learned FSMap version 20 2024-03-28T05:29:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.376+0000 7f7cf83ad740 1 -- 172.21.15.23:0/3425933147 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x555d7aca1ca0 con 0x555d7ac92e00 2024-03-28T05:29:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.376+0000 7f7cd67fc640 1 -- 172.21.15.23:0/3425933147 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7cd802c8b0 con 0x555d7ac92e00 2024-03-28T05:29:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.376+0000 7f7cd67fc640 1 -- 172.21.15.23:0/3425933147 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f7cd80346a0 con 0x555d7ac92e00 2024-03-28T05:29:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.376+0000 7f7cf83ad740 10 client.15960 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:29:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.376+0000 7f7cf83ad740 10 client.15960 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:29:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.376+0000 7f7cf83ad740 20 client.15960 populate_metadata read hostname 'smithi023' 2024-03-28T05:29:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.376+0000 7f7cf83ad740 1 --2- 172.21.15.23:0/3425933147 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x555d7aca2ff0 0x555d7aca53e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.376+0000 7f7cf83ad740 4 client.15960 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:29:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.376+0000 7f7cf83ad740 1 -- 172.21.15.23:0/3425933147 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555d7abfb050 con 0x555d7aca2ff0 2024-03-28T05:29:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.376+0000 7f7ceeffd640 1 --2- 172.21.15.23:0/3425933147 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x555d7aca2ff0 0x555d7aca53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.377+0000 7f7ceeffd640 1 --2- 172.21.15.23:0/3425933147 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x555d7aca2ff0 0x555d7aca53e0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=461e0f1fa66ecc20 server_cookie=4e3f8bd122ebb446 in_seq=0 out_seq=0 2024-03-28T05:29:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.377+0000 7f7cd67fc640 10 client.15960 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:29:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.378+0000 7f7cd67fc640 1 -- 172.21.15.23:0/3425933147 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555d7abfb050 con 0x555d7aca2ff0 2024-03-28T05:29:40.384 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.378+0000 7f7cd67fc640 10 client.15960 handle_command_reply: tid=0 2024-03-28T05:29:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.378+0000 7f7cf83ad740 1 client.15960 shutdown 2024-03-28T05:29:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.378+0000 7f7cf83ad740 1 -- 172.21.15.23:0/3425933147 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac92e00 msgr2=0x555d7ac908e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.378+0000 7f7cf83ad740 1 --2- 172.21.15.23:0/3425933147 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac92e00 0x555d7ac908e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8009850 tx=0x7f7cd802e040 comp rx=0 tx=0).stop 2024-03-28T05:29:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.378+0000 7f7cf83ad740 1 -- 172.21.15.23:0/3425933147 shutdown_connections 2024-03-28T05:29:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.378+0000 7f7cf83ad740 1 --2- 172.21.15.23:0/3425933147 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac92e00 0x555d7ac908e0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.378+0000 7f7cf83ad740 1 --2- 172.21.15.23:0/3425933147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d7ac8a030 0x555d7ac903a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.378+0000 7f7cf83ad740 1 --2- 172.21.15.23:0/3425933147 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x555d7aca2ff0 0x555d7aca53e0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.378+0000 7f7cf83ad740 1 --2- 172.21.15.23:0/3425933147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d7ac89710 0x555d7ac8fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.378+0000 7f7cd67fc640 0 client.15960 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:29:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.378+0000 7f7cf83ad740 1 -- 172.21.15.23:0/3425933147 >> 172.21.15.23:0/3425933147 conn(0x555d7ac84030 msgr2=0x555d7ac85d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.378+0000 7f7cf83ad740 1 -- 172.21.15.23:0/3425933147 shutdown_connections 2024-03-28T05:29:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.379+0000 7f7cf83ad740 1 -- 172.21.15.23:0/3425933147 wait complete. 2024-03-28T05:29:40.385 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.379+0000 7f7cf83ad740 20 client.15960 trim_cache size 0 max 16384 2024-03-28T05:29:40.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.393+0000 7f7cf83ad740 1 Processor -- start 2024-03-28T05:29:40.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.393+0000 7f7cf83ad740 1 -- start start 2024-03-28T05:29:40.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.394+0000 7f7cf83ad740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d7ac89710 0x555d7ac92590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:40.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.394+0000 7f7cf83ad740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d7ac8a030 0x555d7ac8d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:40.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.394+0000 7f7cf83ad740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac92e00 0x555d7ac8d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:40.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.394+0000 7f7cf83ad740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf01ac8d0 con 0x555d7ac92e00 2024-03-28T05:29:40.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.394+0000 7f7cf83ad740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf01aee90 con 0x555d7ac89710 2024-03-28T05:29:40.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.394+0000 7f7cf83ad740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf01aced0 con 0x555d7ac8a030 2024-03-28T05:29:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.394+0000 7f7cee7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac92e00 0x555d7ac8d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.394+0000 7f7ceeffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d7ac8a030 0x555d7ac8d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.394+0000 7f7ceeffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d7ac8a030 0x555d7ac8d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:58272/0 (socket says 172.21.15.23:58272) 2024-03-28T05:29:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.394+0000 7f7ceeffd640 1 -- 172.21.15.23:0/2722781584 learned_addr learned my addr 172.21.15.23:0/2722781584 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:29:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.394+0000 7f7cee7fc640 1 --2- 172.21.15.23:0/2722781584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d7ac89710 0x555d7ac92590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.394+0000 7f7cee7fc640 1 -- 172.21.15.23:0/2722781584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d7ac8a030 msgr2=0x555d7ac8d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.394+0000 7f7cee7fc640 1 --2- 172.21.15.23:0/2722781584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d7ac8a030 0x555d7ac8d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.394+0000 7f7cee7fc640 1 -- 172.21.15.23:0/2722781584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d7ac89710 msgr2=0x555d7ac92590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.394+0000 7f7cee7fc640 1 --2- 172.21.15.23:0/2722781584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d7ac89710 0x555d7ac92590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.394+0000 7f7cee7fc640 1 -- 172.21.15.23:0/2722781584 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d7ac951f0 con 0x555d7ac92e00 2024-03-28T05:29:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.394+0000 7f7ceeffd640 1 --2- 172.21.15.23:0/2722781584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d7ac8a030 0x555d7ac8d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:29:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.394+0000 7f7cee7fc640 1 --2- 172.21.15.23:0/2722781584 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac92e00 0x555d7ac8d600 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8034aa0 tx=0x7f7cd8004e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e200fe5dd69141a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.395+0000 7f7ce4ff9640 1 -- 172.21.15.23:0/2722781584 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8065070 con 0x555d7ac92e00 2024-03-28T05:29:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.395+0000 7f7ce4ff9640 1 -- 172.21.15.23:0/2722781584 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7cd802c8b0 con 0x555d7ac92e00 2024-03-28T05:29:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.395+0000 7f7ce4ff9640 1 -- 172.21.15.23:0/2722781584 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8049040 con 0x555d7ac92e00 2024-03-28T05:29:40.400 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.395+0000 7f7cf83ad740 1 -- 172.21.15.23:0/2722781584 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac92e00 msgr2=0x555d7ac8d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:40.400 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.395+0000 7f7cf83ad740 1 --2- 172.21.15.23:0/2722781584 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac92e00 0x555d7ac8d600 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8034aa0 tx=0x7f7cd8004e30 comp rx=0 tx=0).stop 2024-03-28T05:29:40.400 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.395+0000 7f7cf83ad740 1 -- 172.21.15.23:0/2722781584 shutdown_connections 2024-03-28T05:29:40.400 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.395+0000 7f7cf83ad740 1 --2- 172.21.15.23:0/2722781584 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac92e00 0x555d7ac8d600 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.400 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.395+0000 7f7cf83ad740 1 --2- 172.21.15.23:0/2722781584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d7ac8a030 0x555d7ac8d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.400 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.395+0000 7f7cf83ad740 1 --2- 172.21.15.23:0/2722781584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d7ac89710 0x555d7ac92590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.400 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.395+0000 7f7cf83ad740 1 -- 172.21.15.23:0/2722781584 >> 172.21.15.23:0/2722781584 conn(0x555d7ac84030 msgr2=0x555d7ac8a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:40.400 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.396+0000 7f7cf83ad740 1 -- 172.21.15.23:0/2722781584 shutdown_connections 2024-03-28T05:29:40.400 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.396+0000 7f7cf83ad740 1 -- 172.21.15.23:0/2722781584 wait complete. 2024-03-28T05:29:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.396+0000 7f7cf83ad740 1 -- start start 2024-03-28T05:29:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.396+0000 7f7cf83ad740 1 Processor -- start 2024-03-28T05:29:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.396+0000 7f7cf83ad740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d7ac89710 0x555d7aca8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.396+0000 7f7cf83ad740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d7ac8a030 0x555d7aca54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.396+0000 7f7cf83ad740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac92e00 0x555d7aca59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.396+0000 7f7cf83ad740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf01ad510 con 0x555d7ac92e00 2024-03-28T05:29:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.396+0000 7f7cf83ad740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf01ae830 con 0x555d7ac89710 2024-03-28T05:29:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.396+0000 7f7cf83ad740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf01adb70 con 0x555d7ac8a030 2024-03-28T05:29:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.396+0000 7f7cf83ad740 10 client.0 _ll_register_callbacks cb 0x7f7cd8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:29:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.396+0000 7f7cee7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac92e00 0x555d7aca59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.396+0000 7f7cee7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac92e00 0x555d7aca59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45816/0 (socket says 172.21.15.23:45816) 2024-03-28T05:29:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.396+0000 7f7cee7fc640 1 -- 172.21.15.23:0/1671186728 learned_addr learned my addr 172.21.15.23:0/1671186728 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:29:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.396+0000 7f7ceeffd640 1 --2- 172.21.15.23:0/1671186728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d7ac8a030 0x555d7aca54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.396+0000 7f7cee7fc640 1 --2- 172.21.15.23:0/1671186728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d7ac89710 0x555d7aca8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.397+0000 7f7cee7fc640 1 -- 172.21.15.23:0/1671186728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d7ac8a030 msgr2=0x555d7aca54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.397+0000 7f7cee7fc640 1 --2- 172.21.15.23:0/1671186728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d7ac8a030 0x555d7aca54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.397+0000 7f7cee7fc640 1 -- 172.21.15.23:0/1671186728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d7ac89710 msgr2=0x555d7aca8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.397+0000 7f7cee7fc640 1 --2- 172.21.15.23:0/1671186728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d7ac89710 0x555d7aca8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.397+0000 7f7cee7fc640 1 -- 172.21.15.23:0/1671186728 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555d7aca61e0 con 0x555d7ac92e00 2024-03-28T05:29:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.397+0000 7f7cee7fc640 1 --2- 172.21.15.23:0/1671186728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac92e00 0x555d7aca59e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f7cd802c3f0 tx=0x7f7cd8005950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efae837315b9a03b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.397+0000 7f7cd67fc640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:29:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.397+0000 7f7cd67fc640 1 -- 172.21.15.23:0/1671186728 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8065070 con 0x555d7ac92e00 2024-03-28T05:29:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.397+0000 7f7cd67fc640 1 -- 172.21.15.23:0/1671186728 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f7cd8034640 con 0x555d7ac92e00 2024-03-28T05:29:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.397+0000 7f7cf83ad740 1 -- 172.21.15.23:0/1671186728 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7cd802c8b0 con 0x555d7ac92e00 2024-03-28T05:29:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.397+0000 7f7cd67fc640 1 -- 172.21.15.23:0/1671186728 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7cd8049030 con 0x555d7ac92e00 2024-03-28T05:29:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.397+0000 7f7cf83ad740 10 client.15966 fetch_fsmap learned FSMap version 20 2024-03-28T05:29:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.397+0000 7f7cf83ad740 1 -- 172.21.15.23:0/1671186728 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x555d7acfe4a0 con 0x555d7ac92e00 2024-03-28T05:29:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.397+0000 7f7cd67fc640 1 -- 172.21.15.23:0/1671186728 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7cd802c8b0 con 0x555d7ac92e00 2024-03-28T05:29:40.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.397+0000 7f7cd67fc640 1 -- 172.21.15.23:0/1671186728 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f7cd803d7b0 con 0x555d7ac92e00 2024-03-28T05:29:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.397+0000 7f7cf83ad740 10 client.15966 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:29:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.397+0000 7f7cf83ad740 10 client.15966 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:29:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.397+0000 7f7cf83ad740 20 client.15966 populate_metadata read hostname 'smithi023' 2024-03-28T05:29:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.397+0000 7f7cf83ad740 1 --2- 172.21.15.23:0/1671186728 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x555d7acfe8b0 0x555d7ad00cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.397+0000 7f7cf83ad740 4 client.15966 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:29:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.397+0000 7f7cf83ad740 1 -- 172.21.15.23:0/1671186728 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555d7abf53a0 con 0x555d7acfe8b0 2024-03-28T05:29:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.398+0000 7f7ceeffd640 1 --2- 172.21.15.23:0/1671186728 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x555d7acfe8b0 0x555d7ad00cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.398+0000 7f7cd67fc640 10 client.15966 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:29:40.403 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.398+0000 7f7ceeffd640 1 --2- 172.21.15.23:0/1671186728 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x555d7acfe8b0 0x555d7ad00cd0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac13055e7941a683 server_cookie=d4c43b36eb09ada2 in_seq=0 out_seq=0 2024-03-28T05:29:40.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.869+0000 7f7cd67fc640 1 -- 172.21.15.23:0/1671186728 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555d7abf53a0 con 0x555d7acfe8b0 2024-03-28T05:29:40.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.870+0000 7f7cd67fc640 10 client.15966 handle_command_reply: tid=0 2024-03-28T05:29:40.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.870+0000 7f7cf83ad740 1 client.15966 shutdown 2024-03-28T05:29:40.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.870+0000 7f7cf83ad740 1 -- 172.21.15.23:0/1671186728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac92e00 msgr2=0x555d7aca59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:40.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.870+0000 7f7cf83ad740 1 --2- 172.21.15.23:0/1671186728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac92e00 0x555d7aca59e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f7cd802c3f0 tx=0x7f7cd8005950 comp rx=0 tx=0).stop 2024-03-28T05:29:40.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.870+0000 7f7cf83ad740 1 -- 172.21.15.23:0/1671186728 shutdown_connections 2024-03-28T05:29:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.870+0000 7f7cf83ad740 1 --2- 172.21.15.23:0/1671186728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d7ac92e00 0x555d7aca59e0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.870+0000 7f7cf83ad740 1 --2- 172.21.15.23:0/1671186728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d7ac8a030 0x555d7aca54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.870+0000 7f7cf83ad740 1 --2- 172.21.15.23:0/1671186728 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x555d7acfe8b0 0x555d7ad00cd0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.870+0000 7f7cf83ad740 1 --2- 172.21.15.23:0/1671186728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d7ac89710 0x555d7aca8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.870+0000 7f7cd67fc640 0 client.15966 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:29:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.870+0000 7f7cf83ad740 1 -- 172.21.15.23:0/1671186728 >> 172.21.15.23:0/1671186728 conn(0x555d7ac84030 msgr2=0x555d7ac8fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.870+0000 7f7cf83ad740 1 -- 172.21.15.23:0/1671186728 shutdown_connections 2024-03-28T05:29:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.870+0000 7f7cf83ad740 1 -- 172.21.15.23:0/1671186728 wait complete. 2024-03-28T05:29:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.870+0000 7f7cf83ad740 20 client.15966 trim_cache size 0 max 16384 2024-03-28T05:29:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:29:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:29:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:29:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:29:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.871+0000 7f7cf57c4640 1 -- 172.21.15.23:0/1532860109 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f7cdc0a2a10 msgr2=0x7f7cdc0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:40.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.871+0000 7f7cf57c4640 1 --2- 172.21.15.23:0/1532860109 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f7cdc0a2a10 0x7f7cdc0a4ed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7ce00cd170 tx=0x7f7ce006d6b0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.871+0000 7f7cf57c4640 1 -- 172.21.15.23:0/1532860109 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7cf0105540 msgr2=0x7f7cf019c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:40.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.871+0000 7f7cf57c4640 1 --2- 172.21.15.23:0/1532860109 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7cf0105540 0x7f7cf019c850 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0005fd0 tx=0x7f7ce000ee60 comp rx=0 tx=0).stop 2024-03-28T05:29:40.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.871+0000 7f7cf57c4640 1 -- 172.21.15.23:0/1532860109 shutdown_connections 2024-03-28T05:29:40.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.871+0000 7f7cf57c4640 1 --2- 172.21.15.23:0/1532860109 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7cf0105540 0x7f7cf019c850 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.871+0000 7f7cf57c4640 1 --2- 172.21.15.23:0/1532860109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf01011b0 0x7f7cf0198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.871+0000 7f7cf57c4640 1 --2- 172.21.15.23:0/1532860109 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f7cdc0a2a10 0x7f7cdc0a4ed0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.871+0000 7f7cf57c4640 1 --2- 172.21.15.23:0/1532860109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cf01007f0 0x7f7cf0197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:40.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.871+0000 7f7cf57c4640 1 -- 172.21.15.23:0/1532860109 >> 172.21.15.23:0/1532860109 conn(0x7f7cf00773f0 msgr2=0x7f7cf0104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:40.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.871+0000 7f7cf57c4640 1 -- 172.21.15.23:0/1532860109 shutdown_connections 2024-03-28T05:29:40.874 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:40.871+0000 7f7cf57c4640 1 -- 172.21.15.23:0/1532860109 wait complete. 2024-03-28T05:29:42.084 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:41 smithi023 ceph-mon[31954]: pgmap v186: 129 pgs: 129 active+clean; 214 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 384 op/s 2024-03-28T05:29:42.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:41 smithi098 ceph-mon[36087]: pgmap v186: 129 pgs: 129 active+clean; 214 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 384 op/s 2024-03-28T05:29:42.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:41 smithi113 ceph-mon[37164]: pgmap v186: 129 pgs: 129 active+clean; 214 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 384 op/s 2024-03-28T05:29:44.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:43 smithi098 ceph-mon[36087]: pgmap v187: 129 pgs: 129 active+clean; 226 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 2.2 MiB/s wr, 328 op/s 2024-03-28T05:29:44.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:43 smithi113 ceph-mon[37164]: pgmap v187: 129 pgs: 129 active+clean; 226 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 2.2 MiB/s wr, 328 op/s 2024-03-28T05:29:44.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:43 smithi023 ceph-mon[31954]: pgmap v187: 129 pgs: 129 active+clean; 226 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 2.2 MiB/s wr, 328 op/s 2024-03-28T05:29:46.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:45 smithi098 ceph-mon[36087]: pgmap v188: 129 pgs: 129 active+clean; 226 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 2.2 MiB/s wr, 160 op/s 2024-03-28T05:29:46.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:45 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:29:46.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:45 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:29:46.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:45 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:29:46.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:45 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:29:46.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:45 smithi113 ceph-mon[37164]: pgmap v188: 129 pgs: 129 active+clean; 226 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 2.2 MiB/s wr, 160 op/s 2024-03-28T05:29:46.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:45 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:29:46.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:45 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:29:46.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:45 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:29:46.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:45 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:29:46.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:45 smithi023 ceph-mon[31954]: pgmap v188: 129 pgs: 129 active+clean; 226 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 2.2 MiB/s wr, 160 op/s 2024-03-28T05:29:46.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:45 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:29:46.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:45 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:29:46.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:45 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:29:46.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:45 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:29:46.401 INFO:tasks.workunit.client.0.smithi023.stdout: 66 266478 86 176081 147 140869 302 2024-03-28T05:29:48.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:47 smithi098 ceph-mon[36087]: pgmap v189: 129 pgs: 129 active+clean; 217 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 424 op/s 2024-03-28T05:29:48.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:47 smithi113 ceph-mon[37164]: pgmap v189: 129 pgs: 129 active+clean; 217 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 424 op/s 2024-03-28T05:29:48.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:47 smithi023 ceph-mon[31954]: pgmap v189: 129 pgs: 129 active+clean; 217 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 424 op/s 2024-03-28T05:29:50.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:49 smithi098 ceph-mon[36087]: pgmap v190: 129 pgs: 129 active+clean; 217 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 2.3 MiB/s wr, 392 op/s 2024-03-28T05:29:50.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:49 smithi113 ceph-mon[37164]: pgmap v190: 129 pgs: 129 active+clean; 217 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 2.3 MiB/s wr, 392 op/s 2024-03-28T05:29:50.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:49 smithi023 ceph-mon[31954]: pgmap v190: 129 pgs: 129 active+clean; 217 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 2.3 MiB/s wr, 392 op/s 2024-03-28T05:29:52.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:51 smithi098 ceph-mon[36087]: pgmap v191: 129 pgs: 129 active+clean; 220 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 419 op/s 2024-03-28T05:29:52.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:51 smithi113 ceph-mon[37164]: pgmap v191: 129 pgs: 129 active+clean; 220 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 419 op/s 2024-03-28T05:29:52.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:51 smithi023 ceph-mon[31954]: pgmap v191: 129 pgs: 129 active+clean; 220 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 419 op/s 2024-03-28T05:29:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.963+0000 7f599b08a640 1 -- 172.21.15.23:0/3642672963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5994100880 msgr2=0x7f5994100d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.963+0000 7f599b08a640 1 --2- 172.21.15.23:0/3642672963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5994100880 0x7f5994100d00 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5980009930 tx=0x7f598002bc30 comp rx=0 tx=0).stop 2024-03-28T05:29:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.963+0000 7f599b08a640 1 -- 172.21.15.23:0/3642672963 shutdown_connections 2024-03-28T05:29:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.963+0000 7f599b08a640 1 --2- 172.21.15.23:0/3642672963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5994101240 0x7f599410d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.963+0000 7f599b08a640 1 --2- 172.21.15.23:0/3642672963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5994100880 0x7f5994100d00 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.963+0000 7f599b08a640 1 --2- 172.21.15.23:0/3642672963 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59941066e0 0x7f5994106ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.963+0000 7f599b08a640 1 -- 172.21.15.23:0/3642672963 >> 172.21.15.23:0/3642672963 conn(0x7f5994077380 msgr2=0x7f5994077790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.963+0000 7f599b08a640 1 -- 172.21.15.23:0/3642672963 shutdown_connections 2024-03-28T05:29:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.963+0000 7f599b08a640 1 -- 172.21.15.23:0/3642672963 wait complete. 2024-03-28T05:29:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.964+0000 7f599b08a640 1 Processor -- start 2024-03-28T05:29:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.964+0000 7f599b08a640 1 -- start start 2024-03-28T05:29:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.964+0000 7f599b08a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5994100880 0x7f5994197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.964+0000 7f599b08a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5994101240 0x7f59941983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.964+0000 7f599b08a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59941066e0 0x7f599419c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.964+0000 7f599b08a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5994110a60 con 0x7f5994100880 2024-03-28T05:29:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.964+0000 7f599b08a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59941108e0 con 0x7f59941066e0 2024-03-28T05:29:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.964+0000 7f599b08a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5994110be0 con 0x7f5994101240 2024-03-28T05:29:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.964+0000 7f5998e26640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5994100880 0x7f5994197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.964+0000 7f5998e26640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5994100880 0x7f5994197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53136/0 (socket says 172.21.15.23:53136) 2024-03-28T05:29:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.964+0000 7f5998e26640 1 -- 172.21.15.23:0/3847190076 learned_addr learned my addr 172.21.15.23:0/3847190076 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:29:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.964+0000 7f5993fff640 1 --2- 172.21.15.23:0/3847190076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5994101240 0x7f59941983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.964+0000 7f5999627640 1 --2- 172.21.15.23:0/3847190076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59941066e0 0x7f599419c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.964+0000 7f5998e26640 1 -- 172.21.15.23:0/3847190076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5994101240 msgr2=0x7f59941983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.964+0000 7f5998e26640 1 --2- 172.21.15.23:0/3847190076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5994101240 0x7f59941983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.964+0000 7f5998e26640 1 -- 172.21.15.23:0/3847190076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59941066e0 msgr2=0x7f599419c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.964+0000 7f5998e26640 1 --2- 172.21.15.23:0/3847190076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59941066e0 0x7f599419c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.964+0000 7f5998e26640 1 -- 172.21.15.23:0/3847190076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f599419ce80 con 0x7f5994100880 2024-03-28T05:29:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.965+0000 7f5998e26640 1 --2- 172.21.15.23:0/3847190076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5994100880 0x7f5994197e90 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f598800e9f0 tx=0x7f598800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.965+0000 7f5991ffb640 1 -- 172.21.15.23:0/3847190076 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598800cd80 con 0x7f5994100880 2024-03-28T05:29:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.965+0000 7f599b08a640 1 -- 172.21.15.23:0/3847190076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f599419d170 con 0x7f5994100880 2024-03-28T05:29:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.965+0000 7f5991ffb640 1 -- 172.21.15.23:0/3847190076 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5988005500 con 0x7f5994100880 2024-03-28T05:29:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.965+0000 7f5991ffb640 1 -- 172.21.15.23:0/3847190076 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5988010690 con 0x7f5994100880 2024-03-28T05:29:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.965+0000 7f599b08a640 1 -- 172.21.15.23:0/3847190076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59941a4a00 con 0x7f5994100880 2024-03-28T05:29:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.967+0000 7f5991ffb640 1 -- 172.21.15.23:0/3847190076 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f59880046c0 con 0x7f5994100880 2024-03-28T05:29:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.968+0000 7f5991ffb640 1 --2- 172.21.15.23:0/3847190076 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f59840a2a10 0x7f59840a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.968+0000 7f599dc70740 1 Processor -- start 2024-03-28T05:29:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.968+0000 7f599dc70740 1 -- start start 2024-03-28T05:29:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.968+0000 7f5991ffb640 1 -- 172.21.15.23:0/3847190076 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f59880cf690 con 0x7f5994100880 2024-03-28T05:29:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.968+0000 7f5993fff640 1 --2- 172.21.15.23:0/3847190076 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f59840a2a10 0x7f59840a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.968+0000 7f599dc70740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecb15f1230 0x55ecb15f1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.968+0000 7f599dc70740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15f1b50 0x55ecb15fecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.968+0000 7f599dc70740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb15ff210 0x55ecb1601600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.968+0000 7f599dc70740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecb1555b10 con 0x55ecb15f1b50 2024-03-28T05:29:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.968+0000 7f599dc70740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecb14fe900 con 0x55ecb15f1230 2024-03-28T05:29:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.968+0000 7f599dc70740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecb1551bb0 con 0x55ecb15ff210 2024-03-28T05:29:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.968+0000 7f5993fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15f1b50 0x55ecb15fecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.968+0000 7f5993fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15f1b50 0x55ecb15fecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53142/0 (socket says 172.21.15.23:53142) 2024-03-28T05:29:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.968+0000 7f5999627640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb15ff210 0x55ecb1601600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.968+0000 7f5998e26640 1 --2- 172.21.15.23:0/1292713503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecb15f1230 0x55ecb15f1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.968+0000 7f5993fff640 1 -- 172.21.15.23:0/1292713503 learned_addr learned my addr 172.21.15.23:0/1292713503 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:29:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.968+0000 7f5999627640 1 -- 172.21.15.23:0/1292713503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecb15f1230 msgr2=0x55ecb15f1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.968+0000 7f5999627640 1 --2- 172.21.15.23:0/1292713503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecb15f1230 0x55ecb15f1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.968+0000 7f5999627640 1 -- 172.21.15.23:0/1292713503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15f1b50 msgr2=0x55ecb15fecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.968+0000 7f5999627640 1 --2- 172.21.15.23:0/1292713503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15f1b50 0x55ecb15fecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.969+0000 7f5999627640 1 -- 172.21.15.23:0/1292713503 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ecb126a210 con 0x55ecb15ff210 2024-03-28T05:29:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.969+0000 7f5993fff640 1 --2- 172.21.15.23:0/3847190076 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f59840a2a10 0x7f59840a4ed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5980009900 tx=0x7f598002ca00 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.969+0000 7f5999627640 1 --2- 172.21.15.23:0/1292713503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb15ff210 0x55ecb1601600 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f598c008cd0 tx=0x7f598c00d710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0841a177f8c4322 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.969+0000 7f597effd640 1 -- 172.21.15.23:0/1292713503 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598c018680 con 0x55ecb15ff210 2024-03-28T05:29:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.969+0000 7f597effd640 1 -- 172.21.15.23:0/1292713503 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f598c00f900 con 0x55ecb15ff210 2024-03-28T05:29:53.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.969+0000 7f597effd640 1 -- 172.21.15.23:0/1292713503 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598c00e930 con 0x55ecb15ff210 2024-03-28T05:29:53.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.970+0000 7f599dc70740 1 -- 172.21.15.23:0/1292713503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb15ff210 msgr2=0x55ecb1601600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:53.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.970+0000 7f599dc70740 1 --2- 172.21.15.23:0/1292713503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb15ff210 0x55ecb1601600 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f598c008cd0 tx=0x7f598c00d710 comp rx=0 tx=0).stop 2024-03-28T05:29:53.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.970+0000 7f599dc70740 1 -- 172.21.15.23:0/1292713503 shutdown_connections 2024-03-28T05:29:53.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.970+0000 7f599dc70740 1 --2- 172.21.15.23:0/1292713503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb15ff210 0x55ecb1601600 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.970+0000 7f599dc70740 1 --2- 172.21.15.23:0/1292713503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15f1b50 0x55ecb15fecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.970+0000 7f599dc70740 1 --2- 172.21.15.23:0/1292713503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecb15f1230 0x55ecb15f1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.970+0000 7f599dc70740 1 -- 172.21.15.23:0/1292713503 >> 172.21.15.23:0/1292713503 conn(0x55ecb15ec030 msgr2=0x55ecb15ec410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:53.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.970+0000 7f599dc70740 1 -- 172.21.15.23:0/1292713503 shutdown_connections 2024-03-28T05:29:53.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.970+0000 7f599dc70740 1 -- 172.21.15.23:0/1292713503 wait complete. 2024-03-28T05:29:53.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.970+0000 7f599dc70740 1 -- start start 2024-03-28T05:29:53.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.970+0000 7f599dc70740 1 Processor -- start 2024-03-28T05:29:53.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.970+0000 7f599dc70740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15f1230 0x55ecb15fc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:53.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f599dc70740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb15f1b50 0x55ecb15fc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:53.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f599dc70740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecb15ff210 0x55ecb15fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:53.974 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f599dc70740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecb1601dd0 con 0x55ecb15f1230 2024-03-28T05:29:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f599dc70740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecb1511d50 con 0x55ecb15ff210 2024-03-28T05:29:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f599dc70740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecb1601f50 con 0x55ecb15f1b50 2024-03-28T05:29:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f599dc70740 10 client.0 _ll_register_callbacks cb 0x55ecb1511720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:29:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f5998e26640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15f1230 0x55ecb15fc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f5998e26640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15f1230 0x55ecb15fc200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53144/0 (socket says 172.21.15.23:53144) 2024-03-28T05:29:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f5998e26640 1 -- 172.21.15.23:0/2881174148 learned_addr learned my addr 172.21.15.23:0/2881174148 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:29:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f5993fff640 1 --2- 172.21.15.23:0/2881174148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb15f1b50 0x55ecb15fc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f5999627640 1 --2- 172.21.15.23:0/2881174148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecb15ff210 0x55ecb15fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f5998e26640 1 -- 172.21.15.23:0/2881174148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb15f1b50 msgr2=0x55ecb15fc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f5998e26640 1 --2- 172.21.15.23:0/2881174148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb15f1b50 0x55ecb15fc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f5998e26640 1 -- 172.21.15.23:0/2881174148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecb15ff210 msgr2=0x55ecb15fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f5998e26640 1 --2- 172.21.15.23:0/2881174148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecb15ff210 0x55ecb15fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f5998e26640 1 -- 172.21.15.23:0/2881174148 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ecb1336620 con 0x55ecb15f1230 2024-03-28T05:29:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f5998e26640 1 --2- 172.21.15.23:0/2881174148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15f1230 0x55ecb15fc200 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f598809b660 tx=0x7f598809c060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac78291b57fd4c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f5973fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:29:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f5973fff640 1 -- 172.21.15.23:0/2881174148 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59880b3070 con 0x55ecb15f1230 2024-03-28T05:29:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f5973fff640 1 -- 172.21.15.23:0/2881174148 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f59880a3e70 con 0x55ecb15f1230 2024-03-28T05:29:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f599dc70740 1 -- 172.21.15.23:0/2881174148 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f598c00f900 con 0x55ecb15f1230 2024-03-28T05:29:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f5973fff640 1 -- 172.21.15.23:0/2881174148 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59880a4100 con 0x55ecb15f1230 2024-03-28T05:29:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f599dc70740 10 client.15978 fetch_fsmap learned FSMap version 20 2024-03-28T05:29:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.971+0000 7f599dc70740 1 -- 172.21.15.23:0/2881174148 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55ecb160e0f0 con 0x55ecb15f1230 2024-03-28T05:29:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.972+0000 7f5973fff640 1 -- 172.21.15.23:0/2881174148 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f598c00f900 con 0x55ecb15f1230 2024-03-28T05:29:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.972+0000 7f5973fff640 1 -- 172.21.15.23:0/2881174148 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f598809adb0 con 0x55ecb15f1230 2024-03-28T05:29:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.972+0000 7f599dc70740 10 client.15978 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:29:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.972+0000 7f599dc70740 10 client.15978 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:29:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.972+0000 7f599dc70740 20 client.15978 populate_metadata read hostname 'smithi023' 2024-03-28T05:29:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.972+0000 7f599dc70740 1 --2- 172.21.15.23:0/2881174148 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55ecb160f5b0 0x55ecb16119a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.972+0000 7f599dc70740 4 client.15978 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:29:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.972+0000 7f599dc70740 1 -- 172.21.15.23:0/2881174148 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ecb1563050 con 0x55ecb160f5b0 2024-03-28T05:29:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.972+0000 7f5993fff640 1 --2- 172.21.15.23:0/2881174148 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55ecb160f5b0 0x55ecb16119a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.973+0000 7f5973fff640 10 client.15978 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:29:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.973+0000 7f5993fff640 1 --2- 172.21.15.23:0/2881174148 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55ecb160f5b0 0x55ecb16119a0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c17377df01bb1a62 server_cookie=aacb022240c8c035 in_seq=0 out_seq=0 2024-03-28T05:29:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.974+0000 7f5973fff640 1 -- 172.21.15.23:0/2881174148 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ecb1563050 con 0x55ecb160f5b0 2024-03-28T05:29:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.974+0000 7f5973fff640 10 client.15978 handle_command_reply: tid=0 2024-03-28T05:29:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.974+0000 7f599dc70740 1 client.15978 shutdown 2024-03-28T05:29:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.974+0000 7f599dc70740 1 -- 172.21.15.23:0/2881174148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15f1230 msgr2=0x55ecb15fc200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.974+0000 7f599dc70740 1 --2- 172.21.15.23:0/2881174148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15f1230 0x55ecb15fc200 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f598809b660 tx=0x7f598809c060 comp rx=0 tx=0).stop 2024-03-28T05:29:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.974+0000 7f599dc70740 1 -- 172.21.15.23:0/2881174148 shutdown_connections 2024-03-28T05:29:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.974+0000 7f599dc70740 1 --2- 172.21.15.23:0/2881174148 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55ecb160f5b0 0x55ecb16119a0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.974+0000 7f599dc70740 1 --2- 172.21.15.23:0/2881174148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecb15ff210 0x55ecb15fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.974+0000 7f599dc70740 1 --2- 172.21.15.23:0/2881174148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb15f1b50 0x55ecb15fc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.974+0000 7f5973fff640 0 client.15978 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:29:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.974+0000 7f599dc70740 1 --2- 172.21.15.23:0/2881174148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15f1230 0x55ecb15fc200 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.974+0000 7f599dc70740 1 -- 172.21.15.23:0/2881174148 >> 172.21.15.23:0/2881174148 conn(0x55ecb15ec030 msgr2=0x55ecb15edd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.975+0000 7f599dc70740 1 -- 172.21.15.23:0/2881174148 shutdown_connections 2024-03-28T05:29:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.975+0000 7f599dc70740 1 -- 172.21.15.23:0/2881174148 wait complete. 2024-03-28T05:29:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.975+0000 7f599dc70740 20 client.15978 trim_cache size 0 max 16384 2024-03-28T05:29:53.993 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.989+0000 7f599dc70740 1 Processor -- start 2024-03-28T05:29:53.993 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.989+0000 7f599dc70740 1 -- start start 2024-03-28T05:29:53.993 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.990+0000 7f599dc70740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecb15f1230 0x55ecb15fe930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:53.993 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.990+0000 7f599dc70740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb15f1b50 0x55ecb15edb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.990+0000 7f599dc70740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15ff210 0x55ecb160a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.990+0000 7f599dc70740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59941aa6f0 con 0x55ecb15ff210 2024-03-28T05:29:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.990+0000 7f599dc70740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59941accb0 con 0x55ecb15f1230 2024-03-28T05:29:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.990+0000 7f599dc70740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59941aacf0 con 0x55ecb15f1b50 2024-03-28T05:29:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.990+0000 7f5999627640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15ff210 0x55ecb160a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.990+0000 7f5999627640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15ff210 0x55ecb160a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53154/0 (socket says 172.21.15.23:53154) 2024-03-28T05:29:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.990+0000 7f5999627640 1 -- 172.21.15.23:0/2432507096 learned_addr learned my addr 172.21.15.23:0/2432507096 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:29:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.990+0000 7f5993fff640 1 --2- 172.21.15.23:0/2432507096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb15f1b50 0x55ecb15edb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.990+0000 7f5998e26640 1 --2- 172.21.15.23:0/2432507096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecb15f1230 0x55ecb15fe930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.990+0000 7f5999627640 1 -- 172.21.15.23:0/2432507096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb15f1b50 msgr2=0x55ecb15edb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.990+0000 7f5999627640 1 --2- 172.21.15.23:0/2432507096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb15f1b50 0x55ecb15edb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.990+0000 7f5999627640 1 -- 172.21.15.23:0/2432507096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecb15f1230 msgr2=0x55ecb15fe930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.990+0000 7f5999627640 1 --2- 172.21.15.23:0/2432507096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecb15f1230 0x55ecb15fe930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.990+0000 7f5999627640 1 -- 172.21.15.23:0/2432507096 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ecb15f6fe0 con 0x55ecb15ff210 2024-03-28T05:29:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.990+0000 7f5998e26640 1 --2- 172.21.15.23:0/2432507096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecb15f1230 0x55ecb15fe930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:29:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.990+0000 7f5993fff640 1 --2- 172.21.15.23:0/2432507096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb15f1b50 0x55ecb15edb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:29:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.990+0000 7f5999627640 1 --2- 172.21.15.23:0/2432507096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15ff210 0x55ecb160a6d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x55ecb15ed450 tx=0x7f598c008970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f58219448a8d2944 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.990+0000 7f597e7fc640 1 -- 172.21.15.23:0/2432507096 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598c004bc0 con 0x55ecb15ff210 2024-03-28T05:29:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.991+0000 7f597e7fc640 1 -- 172.21.15.23:0/2432507096 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f598c004d60 con 0x55ecb15ff210 2024-03-28T05:29:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.991+0000 7f597e7fc640 1 -- 172.21.15.23:0/2432507096 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598c0186e0 con 0x55ecb15ff210 2024-03-28T05:29:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.991+0000 7f599dc70740 1 -- 172.21.15.23:0/2432507096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15ff210 msgr2=0x55ecb160a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.991+0000 7f599dc70740 1 --2- 172.21.15.23:0/2432507096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15ff210 0x55ecb160a6d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x55ecb15ed450 tx=0x7f598c008970 comp rx=0 tx=0).stop 2024-03-28T05:29:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.991+0000 7f599dc70740 1 -- 172.21.15.23:0/2432507096 shutdown_connections 2024-03-28T05:29:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.991+0000 7f599dc70740 1 --2- 172.21.15.23:0/2432507096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15ff210 0x55ecb160a6d0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.991+0000 7f599dc70740 1 --2- 172.21.15.23:0/2432507096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb15f1b50 0x55ecb15edb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.991+0000 7f599dc70740 1 --2- 172.21.15.23:0/2432507096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecb15f1230 0x55ecb15fe930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.991+0000 7f599dc70740 1 -- 172.21.15.23:0/2432507096 >> 172.21.15.23:0/2432507096 conn(0x55ecb15ec030 msgr2=0x55ecb15ed480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.991+0000 7f599dc70740 1 -- 172.21.15.23:0/2432507096 shutdown_connections 2024-03-28T05:29:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.991+0000 7f599dc70740 1 -- 172.21.15.23:0/2432507096 wait complete. 2024-03-28T05:29:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.991+0000 7f599dc70740 1 -- start start 2024-03-28T05:29:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.992+0000 7f599dc70740 1 Processor -- start 2024-03-28T05:29:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.992+0000 7f599dc70740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15f1230 0x55ecb1602e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.992+0000 7f599dc70740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecb15f1b50 0x55ecb16033d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.992+0000 7f599dc70740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb15ff210 0x55ecb1663e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.992+0000 7f599dc70740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59941ab330 con 0x55ecb15f1230 2024-03-28T05:29:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.992+0000 7f599dc70740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59941ac650 con 0x55ecb15f1b50 2024-03-28T05:29:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.992+0000 7f599dc70740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59941ab990 con 0x55ecb15ff210 2024-03-28T05:29:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.992+0000 7f599dc70740 10 client.0 _ll_register_callbacks cb 0x7f598809b4c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:29:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.992+0000 7f5998e26640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15f1230 0x55ecb1602e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.992+0000 7f5998e26640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15f1230 0x55ecb1602e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53158/0 (socket says 172.21.15.23:53158) 2024-03-28T05:29:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.992+0000 7f5998e26640 1 -- 172.21.15.23:0/985118184 learned_addr learned my addr 172.21.15.23:0/985118184 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:29:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.992+0000 7f5999627640 1 --2- 172.21.15.23:0/985118184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb15ff210 0x55ecb1663e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.992+0000 7f5993fff640 1 --2- 172.21.15.23:0/985118184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecb15f1b50 0x55ecb16033d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.992+0000 7f5998e26640 1 -- 172.21.15.23:0/985118184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb15ff210 msgr2=0x55ecb1663e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.992+0000 7f5998e26640 1 --2- 172.21.15.23:0/985118184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb15ff210 0x55ecb1663e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.992+0000 7f5998e26640 1 -- 172.21.15.23:0/985118184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecb15f1b50 msgr2=0x55ecb16033d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:29:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.992+0000 7f5998e26640 1 --2- 172.21.15.23:0/985118184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecb15f1b50 0x55ecb16033d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.992+0000 7f5998e26640 1 -- 172.21.15.23:0/985118184 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ecb1664450 con 0x55ecb15f1230 2024-03-28T05:29:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.993+0000 7f5998e26640 1 --2- 172.21.15.23:0/985118184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15f1230 0x55ecb1602e90 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f598800c6a0 tx=0x7f59880cf450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6c453392b58f1ba server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:29:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.993+0000 7f5973fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:29:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.993+0000 7f5973fff640 1 -- 172.21.15.23:0/985118184 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59880b3040 con 0x55ecb15f1230 2024-03-28T05:29:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.993+0000 7f5973fff640 1 -- 172.21.15.23:0/985118184 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f59880860b0 con 0x55ecb15f1230 2024-03-28T05:29:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.993+0000 7f599dc70740 1 -- 172.21.15.23:0/985118184 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f598c004d60 con 0x55ecb15f1230 2024-03-28T05:29:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.993+0000 7f5973fff640 1 -- 172.21.15.23:0/985118184 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59880862b0 con 0x55ecb15f1230 2024-03-28T05:29:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.993+0000 7f599dc70740 10 client.15984 fetch_fsmap learned FSMap version 20 2024-03-28T05:29:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.993+0000 7f599dc70740 1 -- 172.21.15.23:0/985118184 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55ecb166b810 con 0x55ecb15f1230 2024-03-28T05:29:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.993+0000 7f5973fff640 1 -- 172.21.15.23:0/985118184 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f598c004d60 con 0x55ecb15f1230 2024-03-28T05:29:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.993+0000 7f5973fff640 1 -- 172.21.15.23:0/985118184 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f598809a820 con 0x55ecb15f1230 2024-03-28T05:29:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.993+0000 7f599dc70740 10 client.15984 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:29:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.993+0000 7f599dc70740 10 client.15984 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:29:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.993+0000 7f599dc70740 20 client.15984 populate_metadata read hostname 'smithi023' 2024-03-28T05:29:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.993+0000 7f599dc70740 1 --2- 172.21.15.23:0/985118184 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55ecb166bd40 0x55ecb166e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:29:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.993+0000 7f599dc70740 4 client.15984 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:29:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.993+0000 7f599dc70740 1 -- 172.21.15.23:0/985118184 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ecb155d3a0 con 0x55ecb166bd40 2024-03-28T05:29:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.993+0000 7f5993fff640 1 --2- 172.21.15.23:0/985118184 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55ecb166bd40 0x55ecb166e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:29:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.994+0000 7f5993fff640 1 --2- 172.21.15.23:0/985118184 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55ecb166bd40 0x55ecb166e130 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0017063fcf11f2b server_cookie=35bdb0b19a40a23b in_seq=0 out_seq=0 2024-03-28T05:29:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:53.994+0000 7f5973fff640 10 client.15984 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:29:54.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:53 smithi098 ceph-mon[36087]: pgmap v192: 129 pgs: 129 active+clean; 220 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 1.7 MiB/s wr, 391 op/s 2024-03-28T05:29:54.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:53 smithi113 ceph-mon[37164]: pgmap v192: 129 pgs: 129 active+clean; 220 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 1.7 MiB/s wr, 391 op/s 2024-03-28T05:29:54.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:53 smithi023 ceph-mon[31954]: pgmap v192: 129 pgs: 129 active+clean; 220 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 1.7 MiB/s wr, 391 op/s 2024-03-28T05:29:54.577 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.576+0000 7f5973fff640 1 -- 172.21.15.23:0/985118184 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ecb155d3a0 con 0x55ecb166bd40 2024-03-28T05:29:54.577 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.576+0000 7f5973fff640 10 client.15984 handle_command_reply: tid=0 2024-03-28T05:29:54.577 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.576+0000 7f599dc70740 1 client.15984 shutdown 2024-03-28T05:29:54.577 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.577+0000 7f599dc70740 1 -- 172.21.15.23:0/985118184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15f1230 msgr2=0x55ecb1602e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:54.577 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.577+0000 7f599dc70740 1 --2- 172.21.15.23:0/985118184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15f1230 0x55ecb1602e90 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f598800c6a0 tx=0x7f59880cf450 comp rx=0 tx=0).stop 2024-03-28T05:29:54.577 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.577+0000 7f599dc70740 1 -- 172.21.15.23:0/985118184 shutdown_connections 2024-03-28T05:29:54.577 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.577+0000 7f599dc70740 1 --2- 172.21.15.23:0/985118184 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb15ff210 0x55ecb1663e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:54.577 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.577+0000 7f599dc70740 1 --2- 172.21.15.23:0/985118184 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55ecb166bd40 0x55ecb166e130 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:54.577 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.577+0000 7f599dc70740 1 --2- 172.21.15.23:0/985118184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ecb15f1b50 0x55ecb16033d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.577+0000 7f5973fff640 0 client.15984 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:29:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.577+0000 7f599dc70740 1 --2- 172.21.15.23:0/985118184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ecb15f1230 0x55ecb1602e90 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.577+0000 7f599dc70740 1 -- 172.21.15.23:0/985118184 >> 172.21.15.23:0/985118184 conn(0x55ecb15ec030 msgr2=0x55ecb15edd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.577+0000 7f599dc70740 1 -- 172.21.15.23:0/985118184 shutdown_connections 2024-03-28T05:29:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.577+0000 7f599dc70740 1 -- 172.21.15.23:0/985118184 wait complete. 2024-03-28T05:29:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.577+0000 7f599dc70740 20 client.15984 trim_cache size 0 max 16384 2024-03-28T05:29:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:29:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:29:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:29:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:29:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.578+0000 7f599b08a640 1 -- 172.21.15.23:0/3847190076 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f59840a2a10 msgr2=0x7f59840a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.578+0000 7f599b08a640 1 --2- 172.21.15.23:0/3847190076 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f59840a2a10 0x7f59840a4ed0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5980009900 tx=0x7f598002ca00 comp rx=0 tx=0).stop 2024-03-28T05:29:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.578+0000 7f599b08a640 1 -- 172.21.15.23:0/3847190076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5994100880 msgr2=0x7f5994197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:29:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.578+0000 7f599b08a640 1 --2- 172.21.15.23:0/3847190076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5994100880 0x7f5994197e90 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f598800e9f0 tx=0x7f598800edf0 comp rx=0 tx=0).stop 2024-03-28T05:29:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.578+0000 7f599b08a640 1 -- 172.21.15.23:0/3847190076 shutdown_connections 2024-03-28T05:29:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.578+0000 7f599b08a640 1 --2- 172.21.15.23:0/3847190076 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f59840a2a10 0x7f59840a4ed0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.578+0000 7f599b08a640 1 --2- 172.21.15.23:0/3847190076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59941066e0 0x7f599419c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.578+0000 7f599b08a640 1 --2- 172.21.15.23:0/3847190076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5994101240 0x7f59941983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.578+0000 7f599b08a640 1 --2- 172.21.15.23:0/3847190076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5994100880 0x7f5994197e90 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:29:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.578+0000 7f599b08a640 1 -- 172.21.15.23:0/3847190076 >> 172.21.15.23:0/3847190076 conn(0x7f5994077380 msgr2=0x7f599406a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:29:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.578+0000 7f599b08a640 1 -- 172.21.15.23:0/3847190076 shutdown_connections 2024-03-28T05:29:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:29:54.578+0000 7f599b08a640 1 -- 172.21.15.23:0/3847190076 wait complete. 2024-03-28T05:29:56.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:55 smithi098 ceph-mon[36087]: pgmap v193: 129 pgs: 129 active+clean; 220 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 634 KiB/s wr, 312 op/s 2024-03-28T05:29:56.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:55 smithi113 ceph-mon[37164]: pgmap v193: 129 pgs: 129 active+clean; 220 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 634 KiB/s wr, 312 op/s 2024-03-28T05:29:56.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:55 smithi023 ceph-mon[31954]: pgmap v193: 129 pgs: 129 active+clean; 220 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 634 KiB/s wr, 312 op/s 2024-03-28T05:29:56.401 INFO:tasks.workunit.client.0.smithi023.stdout: 69 179648 104 121027 128 90955 241 2024-03-28T05:29:58.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:57 smithi098 ceph-mon[36087]: pgmap v194: 129 pgs: 129 active+clean; 214 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 879 KiB/s wr, 593 op/s 2024-03-28T05:29:58.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:57 smithi113 ceph-mon[37164]: pgmap v194: 129 pgs: 129 active+clean; 214 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 879 KiB/s wr, 593 op/s 2024-03-28T05:29:58.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:57 smithi023 ceph-mon[31954]: pgmap v194: 129 pgs: 129 active+clean; 214 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 879 KiB/s wr, 593 op/s 2024-03-28T05:30:00.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:29:59 smithi098 ceph-mon[36087]: pgmap v195: 129 pgs: 129 active+clean; 214 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 527 KiB/s wr, 330 op/s 2024-03-28T05:30:00.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:29:59 smithi113 ceph-mon[37164]: pgmap v195: 129 pgs: 129 active+clean; 214 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 527 KiB/s wr, 330 op/s 2024-03-28T05:30:00.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:29:59 smithi023 ceph-mon[31954]: pgmap v195: 129 pgs: 129 active+clean; 214 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 527 KiB/s wr, 330 op/s 2024-03-28T05:30:01.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:00 smithi098 ceph-mon[36087]: overall HEALTH_OK 2024-03-28T05:30:01.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:00 smithi113 ceph-mon[37164]: overall HEALTH_OK 2024-03-28T05:30:01.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:00 smithi023 ceph-mon[31954]: overall HEALTH_OK 2024-03-28T05:30:02.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:01 smithi098 ceph-mon[36087]: pgmap v196: 129 pgs: 129 active+clean; 222 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.3 MiB/s wr, 374 op/s 2024-03-28T05:30:02.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:01 smithi113 ceph-mon[37164]: pgmap v196: 129 pgs: 129 active+clean; 222 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.3 MiB/s wr, 374 op/s 2024-03-28T05:30:02.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:01 smithi023 ceph-mon[31954]: pgmap v196: 129 pgs: 129 active+clean; 222 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.3 MiB/s wr, 374 op/s 2024-03-28T05:30:04.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:03 smithi098 ceph-mon[36087]: pgmap v197: 129 pgs: 129 active+clean; 233 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 1.9 MiB/s wr, 418 op/s 2024-03-28T05:30:04.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:03 smithi113 ceph-mon[37164]: pgmap v197: 129 pgs: 129 active+clean; 233 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 1.9 MiB/s wr, 418 op/s 2024-03-28T05:30:04.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:03 smithi023 ceph-mon[31954]: pgmap v197: 129 pgs: 129 active+clean; 233 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 1.9 MiB/s wr, 418 op/s 2024-03-28T05:30:06.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:05 smithi098 ceph-mon[36087]: pgmap v198: 129 pgs: 129 active+clean; 233 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 1.9 MiB/s wr, 396 op/s 2024-03-28T05:30:06.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:05 smithi113 ceph-mon[37164]: pgmap v198: 129 pgs: 129 active+clean; 233 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 1.9 MiB/s wr, 396 op/s 2024-03-28T05:30:06.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:05 smithi023 ceph-mon[31954]: pgmap v198: 129 pgs: 129 active+clean; 233 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 1.9 MiB/s wr, 396 op/s 2024-03-28T05:30:06.401 INFO:tasks.workunit.client.0.smithi023.stdout: 70 193396 89 131495 20 101687 122 2024-03-28T05:30:07.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:06 smithi098 ceph-mon[36087]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T05:30:07.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:06 smithi113 ceph-mon[37164]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T05:30:07.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:06 smithi023 ceph-mon[31954]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T05:30:07.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.728+0000 7f5482f90640 1 -- 172.21.15.23:0/3194630028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f547c1007f0 msgr2=0x7f547c100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:07.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.729+0000 7f5482f90640 1 --2- 172.21.15.23:0/3194630028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f547c1007f0 0x7f547c100c70 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f547000d940 tx=0x7f547002dbd0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.730+0000 7f5482f90640 1 -- 172.21.15.23:0/3194630028 shutdown_connections 2024-03-28T05:30:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.730+0000 7f5482f90640 1 --2- 172.21.15.23:0/3194630028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f547c1011b0 0x7f547c10d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.730+0000 7f5482f90640 1 --2- 172.21.15.23:0/3194630028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f547c1007f0 0x7f547c100c70 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.730+0000 7f5482f90640 1 --2- 172.21.15.23:0/3194630028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f547c106730 0x7f547c106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.730+0000 7f5482f90640 1 -- 172.21.15.23:0/3194630028 >> 172.21.15.23:0/3194630028 conn(0x7f547c077360 msgr2=0x7f547c077770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:07.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.730+0000 7f5482f90640 1 -- 172.21.15.23:0/3194630028 shutdown_connections 2024-03-28T05:30:07.730 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.730+0000 7f5482f90640 1 -- 172.21.15.23:0/3194630028 wait complete. 2024-03-28T05:30:07.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.732+0000 7f5482f90640 1 Processor -- start 2024-03-28T05:30:07.731 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.732+0000 7f5482f90640 1 -- start start 2024-03-28T05:30:07.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.732+0000 7f5482f90640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f547c1007f0 0x7f547c072fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:07.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.732+0000 7f5482f90640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f547c1011b0 0x7f547c06d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:07.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.732+0000 7f5482f90640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f547c106730 0x7f547c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:07.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.733+0000 7f5482f90640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547c06bb10 con 0x7f547c1011b0 2024-03-28T05:30:07.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.733+0000 7f5482f90640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547c06b990 con 0x7f547c1007f0 2024-03-28T05:30:07.732 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.733+0000 7f5482f90640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547c06bc90 con 0x7f547c106730 2024-03-28T05:30:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.733+0000 7f5480d2c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f547c1007f0 0x7f547c072fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.733+0000 7f5480d2c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f547c1007f0 0x7f547c072fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.23:49790/0 (socket says 172.21.15.23:49790) 2024-03-28T05:30:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.733+0000 7f5480d2c640 1 -- 172.21.15.23:0/2816705630 learned_addr learned my addr 172.21.15.23:0/2816705630 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:30:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.733+0000 7f547bfff640 1 --2- 172.21.15.23:0/2816705630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f547c1011b0 0x7f547c06d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.733+0000 7f5480d2c640 1 -- 172.21.15.23:0/2816705630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f547c106730 msgr2=0x7f547c06d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.733+0000 7f5480d2c640 1 --2- 172.21.15.23:0/2816705630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f547c106730 0x7f547c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.733+0000 7f5480d2c640 1 -- 172.21.15.23:0/2816705630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f547c1011b0 msgr2=0x7f547c06d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.733+0000 7f5480d2c640 1 --2- 172.21.15.23:0/2816705630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f547c1011b0 0x7f547c06d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.734+0000 7f5480d2c640 1 -- 172.21.15.23:0/2816705630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547c06dcf0 con 0x7f547c1007f0 2024-03-28T05:30:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.734+0000 7f5480d2c640 1 --2- 172.21.15.23:0/2816705630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f547c1007f0 0x7f547c072fc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f547400ba50 tx=0x7f547400bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:07.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.735+0000 7f5479ffb640 1 -- 172.21.15.23:0/2816705630 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5474019070 con 0x7f547c1007f0 2024-03-28T05:30:07.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.735+0000 7f5479ffb640 1 -- 172.21.15.23:0/2816705630 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f547400cc10 con 0x7f547c1007f0 2024-03-28T05:30:07.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.735+0000 7f5479ffb640 1 -- 172.21.15.23:0/2816705630 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54740076a0 con 0x7f547c1007f0 2024-03-28T05:30:07.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.736+0000 7f5482f90640 1 -- 172.21.15.23:0/2816705630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f547c1aac50 con 0x7f547c1007f0 2024-03-28T05:30:07.735 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.736+0000 7f5482f90640 1 -- 172.21.15.23:0/2816705630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f547c1ab060 con 0x7f547c1007f0 2024-03-28T05:30:07.740 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.741+0000 7f5479ffb640 1 -- 172.21.15.23:0/2816705630 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f547401c070 con 0x7f547c1007f0 2024-03-28T05:30:07.741 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.742+0000 7f5485b7f740 1 Processor -- start 2024-03-28T05:30:07.746 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.746+0000 7f5485b7f740 1 -- start start 2024-03-28T05:30:07.747 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.747+0000 7f5485b7f740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557385cbf710 0x557385cbfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:07.747 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.747+0000 7f5485b7f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc0030 0x557385cc88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:07.747 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.747+0000 7f5485b7f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cc8e00 0x557385ccb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:07.747 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.747+0000 7f5485b7f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557385c23b10 con 0x557385cc8e00 2024-03-28T05:30:07.747 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.747+0000 7f5485b7f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x557385bcc900 con 0x557385cc0030 2024-03-28T05:30:07.747 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.747+0000 7f5485b7f740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x557385c1fbb0 con 0x557385cbf710 2024-03-28T05:30:07.747 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.747+0000 7f5479ffb640 1 --2- 172.21.15.23:0/2816705630 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f54640a2c40 0x7f54640a5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.748+0000 7f5479ffb640 1 -- 172.21.15.23:0/2816705630 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9613+0+0 (secure 0 0 0) 0x7f54740cf7f0 con 0x7f547c1007f0 2024-03-28T05:30:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.748+0000 7f547bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc0030 0x557385cc88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.748+0000 7f548152d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cc8e00 0x557385ccb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.748+0000 7f548152d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cc8e00 0x557385ccb1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50078/0 (socket says 172.21.15.23:50078) 2024-03-28T05:30:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.748+0000 7f548152d640 1 -- 172.21.15.23:0/1066977433 learned_addr learned my addr 172.21.15.23:0/1066977433 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:30:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.749+0000 7f548152d640 1 --2- 172.21.15.23:0/2816705630 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f54640a2c40 0x7f54640a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.749+0000 7f5480d2c640 1 --2- 172.21.15.23:0/1066977433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557385cbf710 0x557385cbfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:07.749 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.749+0000 7f548152d640 1 -- 172.21.15.23:0/1066977433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557385cbf710 msgr2=0x557385cbfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:07.749 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.749+0000 7f548152d640 1 --2- 172.21.15.23:0/1066977433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557385cbf710 0x557385cbfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.749 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.749+0000 7f548152d640 1 -- 172.21.15.23:0/1066977433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc0030 msgr2=0x557385cc88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:07.749 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.749+0000 7f548152d640 1 --2- 172.21.15.23:0/1066977433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc0030 0x557385cc88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.749 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.749+0000 7f548152d640 1 -- 172.21.15.23:0/1066977433 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557385938210 con 0x557385cc8e00 2024-03-28T05:30:07.749 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.749+0000 7f547bfff640 1 --2- 172.21.15.23:0/1066977433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc0030 0x557385cc88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:30:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.749+0000 7f548152d640 1 --2- 172.21.15.23:0/1066977433 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cc8e00 0x557385ccb1f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f546c013500 tx=0x7f546c0139c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5421ef69af8d40d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.750+0000 7f548152d640 1 --2- 172.21.15.23:0/2816705630 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f54640a2c40 0x7f54640a5100 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f546c022000 tx=0x7f546c01d000 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:07.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.751+0000 7f5462ffd640 1 -- 172.21.15.23:0/1066977433 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c028050 con 0x557385cc8e00 2024-03-28T05:30:07.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.751+0000 7f5462ffd640 1 -- 172.21.15.23:0/1066977433 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f546c0109f0 con 0x557385cc8e00 2024-03-28T05:30:07.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.751+0000 7f5462ffd640 1 -- 172.21.15.23:0/1066977433 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c022730 con 0x557385cc8e00 2024-03-28T05:30:07.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.753+0000 7f5485b7f740 1 -- 172.21.15.23:0/1066977433 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cc8e00 msgr2=0x557385ccb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:07.754 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.753+0000 7f5485b7f740 1 --2- 172.21.15.23:0/1066977433 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cc8e00 0x557385ccb1f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f546c013500 tx=0x7f546c0139c0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.755 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.755+0000 7f5485b7f740 1 -- 172.21.15.23:0/1066977433 shutdown_connections 2024-03-28T05:30:07.755 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.755+0000 7f5485b7f740 1 --2- 172.21.15.23:0/1066977433 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cc8e00 0x557385ccb1f0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.755 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.755+0000 7f5485b7f740 1 --2- 172.21.15.23:0/1066977433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc0030 0x557385cc88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.755 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.755+0000 7f5485b7f740 1 --2- 172.21.15.23:0/1066977433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557385cbf710 0x557385cbfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.755 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.755+0000 7f5485b7f740 1 -- 172.21.15.23:0/1066977433 >> 172.21.15.23:0/1066977433 conn(0x557385cba030 msgr2=0x557385cba410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:07.759 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.759+0000 7f5485b7f740 1 -- 172.21.15.23:0/1066977433 shutdown_connections 2024-03-28T05:30:07.761 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.761+0000 7f5485b7f740 1 -- 172.21.15.23:0/1066977433 wait complete. 2024-03-28T05:30:07.761 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.762+0000 7f5485b7f740 1 -- start start 2024-03-28T05:30:07.761 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.762+0000 7f5485b7f740 1 Processor -- start 2024-03-28T05:30:07.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.763+0000 7f5485b7f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cbf710 0x557385cc5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:07.763 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.763+0000 7f5485b7f740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557385cc0030 0x557385cc63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:07.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.763+0000 7f5485b7f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc8e00 0x557385cc68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:07.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.763+0000 7f5485b7f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557385ccba30 con 0x557385cbf710 2024-03-28T05:30:07.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.764+0000 7f5485b7f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x557385bdfd50 con 0x557385cc8e00 2024-03-28T05:30:07.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.764+0000 7f548152d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc8e00 0x557385cc68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:07.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.764+0000 7f548152d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc8e00 0x557385cc68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.23:49810/0 (socket says 172.21.15.23:49810) 2024-03-28T05:30:07.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.764+0000 7f548152d640 1 -- 172.21.15.23:0/1653516276 learned_addr learned my addr 172.21.15.23:0/1653516276 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:30:07.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.764+0000 7f5485b7f740 1 -- 172.21.15.23:0/1653516276 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x557385ccbbb0 con 0x557385cc0030 2024-03-28T05:30:07.764 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.764+0000 7f5485b7f740 10 client.0 _ll_register_callbacks cb 0x557385bdf720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:30:07.766 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.765+0000 7f5480d2c640 1 --2- 172.21.15.23:0/1653516276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cbf710 0x557385cc5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:07.766 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.766+0000 7f5480d2c640 1 -- 172.21.15.23:0/1653516276 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557385cc0030 msgr2=0x557385cc63a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T05:30:07.766 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.766+0000 7f5480d2c640 1 --2- 172.21.15.23:0/1653516276 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557385cc0030 0x557385cc63a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.766 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.766+0000 7f5480d2c640 1 -- 172.21.15.23:0/1653516276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc8e00 msgr2=0x557385cc68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:07.766 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.766+0000 7f5480d2c640 1 --2- 172.21.15.23:0/1653516276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc8e00 0x557385cc68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.766 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.766+0000 7f5480d2c640 1 -- 172.21.15.23:0/1653516276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557385a04620 con 0x557385cbf710 2024-03-28T05:30:07.766 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.766+0000 7f5480d2c640 1 --2- 172.21.15.23:0/1653516276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cbf710 0x557385cc5e60 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f54740ccbb0 tx=0x7f547409bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9976880083ddfa9a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:07.767 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.766+0000 7f545bfff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:30:07.767 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.766+0000 7f545bfff640 1 -- 172.21.15.23:0/1653516276 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54740b4060 con 0x557385cbf710 2024-03-28T05:30:07.767 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.767+0000 7f5485b7f740 1 -- 172.21.15.23:0/1653516276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f546c0109f0 con 0x557385cbf710 2024-03-28T05:30:07.768 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.767+0000 7f545bfff640 1 -- 172.21.15.23:0/1653516276 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9613+0+0 (secure 0 0 0) 0x7f54740ccd20 con 0x557385cbf710 2024-03-28T05:30:07.768 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.767+0000 7f545bfff640 1 -- 172.21.15.23:0/1653516276 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54740b9030 con 0x557385cbf710 2024-03-28T05:30:07.768 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.769+0000 7f5485b7f740 10 client.15996 fetch_fsmap learned FSMap version 20 2024-03-28T05:30:07.769 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.769+0000 7f5485b7f740 1 -- 172.21.15.23:0/1653516276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x557385cd7dc0 con 0x557385cbf710 2024-03-28T05:30:07.770 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.771+0000 7f545bfff640 1 -- 172.21.15.23:0/1653516276 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f546c0109f0 con 0x557385cbf710 2024-03-28T05:30:07.771 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.771+0000 7f545bfff640 1 -- 172.21.15.23:0/1653516276 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f54740b4b80 con 0x557385cbf710 2024-03-28T05:30:07.771 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.771+0000 7f5485b7f740 10 client.15996 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:30:07.771 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.771+0000 7f5485b7f740 10 client.15996 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:30:07.771 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.771+0000 7f5485b7f740 20 client.15996 populate_metadata read hostname 'smithi023' 2024-03-28T05:30:07.771 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.771+0000 7f5485b7f740 1 --2- 172.21.15.23:0/1653516276 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x557385cd96c0 0x557385cdbb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:07.771 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.771+0000 7f5485b7f740 4 client.15996 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:30:07.771 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.771+0000 7f5485b7f740 1 -- 172.21.15.23:0/1653516276 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557385c31050 con 0x557385cd96c0 2024-03-28T05:30:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.773+0000 7f547bfff640 1 --2- 172.21.15.23:0/1653516276 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x557385cd96c0 0x557385cdbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.779+0000 7f547bfff640 1 --2- 172.21.15.23:0/1653516276 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x557385cd96c0 0x557385cdbb20 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a568e1407b97d501 server_cookie=fc6f79a6037cf10e in_seq=0 out_seq=0 2024-03-28T05:30:07.779 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.780+0000 7f545bfff640 10 client.15996 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:30:07.789 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.789+0000 7f545bfff640 1 -- 172.21.15.23:0/1653516276 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557385c31050 con 0x557385cd96c0 2024-03-28T05:30:07.789 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.789+0000 7f545bfff640 10 client.15996 handle_command_reply: tid=0 2024-03-28T05:30:07.789 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.789+0000 7f5485b7f740 1 client.15996 shutdown 2024-03-28T05:30:07.791 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.792+0000 7f5485b7f740 1 -- 172.21.15.23:0/1653516276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cbf710 msgr2=0x557385cc5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:07.791 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.792+0000 7f5485b7f740 1 --2- 172.21.15.23:0/1653516276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cbf710 0x557385cc5e60 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f54740ccbb0 tx=0x7f547409bc00 comp rx=0 tx=0).stop 2024-03-28T05:30:07.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.797+0000 7f5485b7f740 1 -- 172.21.15.23:0/1653516276 shutdown_connections 2024-03-28T05:30:07.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.797+0000 7f5485b7f740 1 --2- 172.21.15.23:0/1653516276 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x557385cd96c0 0x557385cdbb20 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.797+0000 7f5485b7f740 1 --2- 172.21.15.23:0/1653516276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc8e00 0x557385cc68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.797+0000 7f5485b7f740 1 --2- 172.21.15.23:0/1653516276 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557385cc0030 0x557385cc63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.797+0000 7f5485b7f740 1 --2- 172.21.15.23:0/1653516276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cbf710 0x557385cc5e60 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.797+0000 7f5485b7f740 1 -- 172.21.15.23:0/1653516276 >> 172.21.15.23:0/1653516276 conn(0x557385cba030 msgr2=0x557385cbbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:07.797 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.798+0000 7f545bfff640 0 client.15996 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:30:07.798 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.799+0000 7f5485b7f740 1 -- 172.21.15.23:0/1653516276 shutdown_connections 2024-03-28T05:30:07.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.801+0000 7f5485b7f740 1 -- 172.21.15.23:0/1653516276 wait complete. 2024-03-28T05:30:07.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.801+0000 7f5485b7f740 20 client.15996 trim_cache size 0 max 16384 2024-03-28T05:30:07.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.832+0000 7f5485b7f740 1 Processor -- start 2024-03-28T05:30:07.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.832+0000 7f5485b7f740 1 -- start start 2024-03-28T05:30:07.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.833+0000 7f5485b7f740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557385cbf710 0x557385cc8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:07.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.833+0000 7f5485b7f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc0030 0x557385cc4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:07.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.833+0000 7f5485b7f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cc8e00 0x557385cc6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:07.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.833+0000 7f5480d2c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557385cbf710 0x557385cc8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:07.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.833+0000 7f5480d2c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557385cbf710 0x557385cc8590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:42046/0 (socket says 172.21.15.23:42046) 2024-03-28T05:30:07.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.833+0000 7f547bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc0030 0x557385cc4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:07.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.833+0000 7f548152d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cc8e00 0x557385cc6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:07.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.833+0000 7f548152d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cc8e00 0x557385cc6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50106/0 (socket says 172.21.15.23:50106) 2024-03-28T05:30:07.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.833+0000 7f547bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc0030 0x557385cc4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.23:49818/0 (socket says 172.21.15.23:49818) 2024-03-28T05:30:07.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.833+0000 7f5480d2c640 1 -- 172.21.15.23:0/1939686643 learned_addr learned my addr 172.21.15.23:0/1939686643 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:30:07.834 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.833+0000 7f5485b7f740 1 -- 172.21.15.23:0/1939686643 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546800e130 con 0x557385cc8e00 2024-03-28T05:30:07.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.833+0000 7f5485b7f740 1 -- 172.21.15.23:0/1939686643 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54680106d0 con 0x557385cc0030 2024-03-28T05:30:07.835 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.833+0000 7f5485b7f740 1 -- 172.21.15.23:0/1939686643 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546800e730 con 0x557385cbf710 2024-03-28T05:30:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.835+0000 7f548152d640 1 -- 172.21.15.23:0/1939686643 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557385cbf710 msgr2=0x557385cc8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.835+0000 7f548152d640 1 --2- 172.21.15.23:0/1939686643 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557385cbf710 0x557385cc8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.835+0000 7f548152d640 1 -- 172.21.15.23:0/1939686643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc0030 msgr2=0x557385cc4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.835+0000 7f548152d640 1 --2- 172.21.15.23:0/1939686643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc0030 0x557385cc4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.835+0000 7f548152d640 1 -- 172.21.15.23:0/1939686643 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557385cc4e10 con 0x557385cc8e00 2024-03-28T05:30:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.835+0000 7f548152d640 1 --2- 172.21.15.23:0/1939686643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cc8e00 0x557385cc6700 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f546c00cb10 tx=0x7f546c00c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26258a9abcc7ca96 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.836+0000 7f54627fc640 1 -- 172.21.15.23:0/1939686643 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c031070 con 0x557385cc8e00 2024-03-28T05:30:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.836+0000 7f54627fc640 1 -- 172.21.15.23:0/1939686643 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f546c017070 con 0x557385cc8e00 2024-03-28T05:30:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.836+0000 7f54627fc640 1 -- 172.21.15.23:0/1939686643 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c0226a0 con 0x557385cc8e00 2024-03-28T05:30:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.837+0000 7f5485b7f740 1 -- 172.21.15.23:0/1939686643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cc8e00 msgr2=0x557385cc6700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.837+0000 7f5485b7f740 1 --2- 172.21.15.23:0/1939686643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cc8e00 0x557385cc6700 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f546c00cb10 tx=0x7f546c00c350 comp rx=0 tx=0).stop 2024-03-28T05:30:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.837+0000 7f5485b7f740 1 -- 172.21.15.23:0/1939686643 shutdown_connections 2024-03-28T05:30:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.837+0000 7f5485b7f740 1 --2- 172.21.15.23:0/1939686643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cc8e00 0x557385cc6700 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.837+0000 7f5485b7f740 1 --2- 172.21.15.23:0/1939686643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc0030 0x557385cc4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.837+0000 7f5485b7f740 1 --2- 172.21.15.23:0/1939686643 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557385cbf710 0x557385cc8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.837+0000 7f5485b7f740 1 -- 172.21.15.23:0/1939686643 >> 172.21.15.23:0/1939686643 conn(0x557385cba030 msgr2=0x557385cbbce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:07.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.839+0000 7f5485b7f740 1 -- 172.21.15.23:0/1939686643 shutdown_connections 2024-03-28T05:30:07.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.841+0000 7f5485b7f740 1 -- 172.21.15.23:0/1939686643 wait complete. 2024-03-28T05:30:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.841+0000 7f5485b7f740 1 -- start start 2024-03-28T05:30:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.841+0000 7f5485b7f740 1 Processor -- start 2024-03-28T05:30:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.842+0000 7f5485b7f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cc0030 0x557385d2e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.843+0000 7f5485b7f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc8e00 0x557385d2e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.843+0000 7f5485b7f740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557385ccc8e0 0x557385cccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.843+0000 7f5485b7f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546800ed50 con 0x557385cc0030 2024-03-28T05:30:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.843+0000 7f5485b7f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5468010070 con 0x557385cc8e00 2024-03-28T05:30:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.843+0000 7f5480d2c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cc0030 0x557385d2e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.843+0000 7f5480d2c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cc0030 0x557385d2e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50112/0 (socket says 172.21.15.23:50112) 2024-03-28T05:30:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.843+0000 7f5480d2c640 1 -- 172.21.15.23:0/85450743 learned_addr learned my addr 172.21.15.23:0/85450743 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:30:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.843+0000 7f547bfff640 1 --2- 172.21.15.23:0/85450743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc8e00 0x557385d2e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.843+0000 7f5485b7f740 1 -- 172.21.15.23:0/85450743 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546800f3b0 con 0x557385ccc8e0 2024-03-28T05:30:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.843+0000 7f548152d640 1 --2- 172.21.15.23:0/85450743 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557385ccc8e0 0x557385cccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.843+0000 7f5485b7f740 10 client.0 _ll_register_callbacks cb 0x7f547409c250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:30:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.844+0000 7f548152d640 1 -- 172.21.15.23:0/85450743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc8e00 msgr2=0x557385d2e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.844+0000 7f548152d640 1 --2- 172.21.15.23:0/85450743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc8e00 0x557385d2e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.844+0000 7f548152d640 1 -- 172.21.15.23:0/85450743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cc0030 msgr2=0x557385d2e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.844+0000 7f548152d640 1 --2- 172.21.15.23:0/85450743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cc0030 0x557385d2e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.844+0000 7f548152d640 1 -- 172.21.15.23:0/85450743 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557385cbfe60 con 0x557385ccc8e0 2024-03-28T05:30:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.844+0000 7f548152d640 1 --2- 172.21.15.23:0/85450743 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557385ccc8e0 0x557385cccd60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f546c00c2f0 tx=0x7f546c013f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f5bf5bf7f586673 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.844+0000 7f545bfff640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-28T05:30:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.846+0000 7f545bfff640 1 -- 172.21.15.23:0/85450743 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c031070 con 0x557385ccc8e0 2024-03-28T05:30:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.846+0000 7f545bfff640 1 -- 172.21.15.23:0/85450743 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9613+0+0 (secure 0 0 0) 0x7f546c013040 con 0x557385ccc8e0 2024-03-28T05:30:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.847+0000 7f5485b7f740 1 -- 172.21.15.23:0/85450743 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f546c017070 con 0x557385ccc8e0 2024-03-28T05:30:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.848+0000 7f545bfff640 1 -- 172.21.15.23:0/85450743 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f546c03b050 con 0x557385ccc8e0 2024-03-28T05:30:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.849+0000 7f5485b7f740 10 client.25828 fetch_fsmap learned FSMap version 20 2024-03-28T05:30:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.849+0000 7f5485b7f740 1 -- 172.21.15.23:0/85450743 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x557385d35ca0 con 0x557385ccc8e0 2024-03-28T05:30:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.850+0000 7f545bfff640 1 -- 172.21.15.23:0/85450743 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f546c013260 con 0x557385ccc8e0 2024-03-28T05:30:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.850+0000 7f545bfff640 1 -- 172.21.15.23:0/85450743 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f546c00f040 con 0x557385ccc8e0 2024-03-28T05:30:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.850+0000 7f5485b7f740 10 client.25828 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:30:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.850+0000 7f5485b7f740 10 client.25828 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:30:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.850+0000 7f5485b7f740 20 client.25828 populate_metadata read hostname 'smithi023' 2024-03-28T05:30:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.850+0000 7f5485b7f740 1 --2- 172.21.15.23:0/85450743 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x557385d360e0 0x557385d38500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.850+0000 7f5485b7f740 4 client.25828 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:30:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.850+0000 7f5485b7f740 1 -- 172.21.15.23:0/85450743 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x7f546c017070 con 0x557385d360e0 2024-03-28T05:30:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.851+0000 7f5480d2c640 1 --2- 172.21.15.23:0/85450743 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x557385d360e0 0x557385d38500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.855+0000 7f5480d2c640 1 --2- 172.21.15.23:0/85450743 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x557385d360e0 0x557385d38500 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=561fd0ba2d03245a server_cookie=ec852f8485d7626 in_seq=0 out_seq=0 2024-03-28T05:30:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:07.855+0000 7f545bfff640 10 client.25828 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:30:08.118 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:07 smithi098 ceph-mon[36087]: pgmap v200: 129 pgs: 129 active+clean; 235 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 2.3 MiB/s wr, 179 op/s 2024-03-28T05:30:08.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:07 smithi113 ceph-mon[37164]: pgmap v200: 129 pgs: 129 active+clean; 235 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 2.3 MiB/s wr, 179 op/s 2024-03-28T05:30:08.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:07 smithi023 ceph-mon[31954]: pgmap v200: 129 pgs: 129 active+clean; 235 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 2.3 MiB/s wr, 179 op/s 2024-03-28T05:30:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.538+0000 7f545bfff640 1 -- 172.21.15.23:0/85450743 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x7f546c017070 con 0x557385d360e0 2024-03-28T05:30:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.538+0000 7f545bfff640 10 client.25828 handle_command_reply: tid=0 2024-03-28T05:30:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.538+0000 7f5485b7f740 1 client.25828 shutdown 2024-03-28T05:30:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.538+0000 7f5485b7f740 1 -- 172.21.15.23:0/85450743 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557385ccc8e0 msgr2=0x557385cccd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.538+0000 7f5485b7f740 1 --2- 172.21.15.23:0/85450743 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557385ccc8e0 0x557385cccd60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f546c00c2f0 tx=0x7f546c013f70 comp rx=0 tx=0).stop 2024-03-28T05:30:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.538+0000 7f5485b7f740 1 -- 172.21.15.23:0/85450743 shutdown_connections 2024-03-28T05:30:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.538+0000 7f5485b7f740 1 --2- 172.21.15.23:0/85450743 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557385ccc8e0 0x557385cccd60 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.538+0000 7f5485b7f740 1 --2- 172.21.15.23:0/85450743 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x557385d360e0 0x557385d38500 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.539+0000 7f545bfff640 0 client.25828 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:30:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.539+0000 7f5485b7f740 1 --2- 172.21.15.23:0/85450743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557385cc8e00 0x557385d2e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.539+0000 7f5485b7f740 1 --2- 172.21.15.23:0/85450743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557385cc0030 0x557385d2e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.539+0000 7f5485b7f740 1 -- 172.21.15.23:0/85450743 >> 172.21.15.23:0/85450743 conn(0x557385cba030 msgr2=0x557385cb8790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.539+0000 7f5485b7f740 1 -- 172.21.15.23:0/85450743 shutdown_connections 2024-03-28T05:30:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.539+0000 7f5485b7f740 1 -- 172.21.15.23:0/85450743 wait complete. 2024-03-28T05:30:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.539+0000 7f5485b7f740 20 client.25828 trim_cache size 0 max 16384 2024-03-28T05:30:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:30:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:30:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:30:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:30:08.542 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.541+0000 7f5482f90640 1 -- 172.21.15.23:0/2816705630 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f54640a2c40 msgr2=0x7f54640a5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:08.542 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.541+0000 7f5482f90640 1 --2- 172.21.15.23:0/2816705630 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f54640a2c40 0x7f54640a5100 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f546c022000 tx=0x7f546c01d000 comp rx=0 tx=0).stop 2024-03-28T05:30:08.542 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.541+0000 7f5482f90640 1 -- 172.21.15.23:0/2816705630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f547c1007f0 msgr2=0x7f547c072fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:08.542 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.541+0000 7f5482f90640 1 --2- 172.21.15.23:0/2816705630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f547c1007f0 0x7f547c072fc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f547400ba50 tx=0x7f547400bf10 comp rx=0 tx=0).stop 2024-03-28T05:30:08.542 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.541+0000 7f5482f90640 1 -- 172.21.15.23:0/2816705630 shutdown_connections 2024-03-28T05:30:08.542 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.541+0000 7f5482f90640 1 --2- 172.21.15.23:0/2816705630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f547c106730 0x7f547c06d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:08.542 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.541+0000 7f5482f90640 1 --2- 172.21.15.23:0/2816705630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f547c1011b0 0x7f547c06d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:08.542 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.541+0000 7f5482f90640 1 --2- 172.21.15.23:0/2816705630 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f54640a2c40 0x7f54640a5100 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:08.542 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.541+0000 7f5482f90640 1 --2- 172.21.15.23:0/2816705630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f547c1007f0 0x7f547c072fc0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:08.542 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.542+0000 7f5482f90640 1 -- 172.21.15.23:0/2816705630 >> 172.21.15.23:0/2816705630 conn(0x7f547c077360 msgr2=0x7f547c0ff2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:08.542 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.542+0000 7f5482f90640 1 -- 172.21.15.23:0/2816705630 shutdown_connections 2024-03-28T05:30:08.542 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:08.542+0000 7f5482f90640 1 -- 172.21.15.23:0/2816705630 wait complete. 2024-03-28T05:30:10.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:09 smithi098 ceph-mon[36087]: pgmap v201: 129 pgs: 129 active+clean; 235 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 2.3 MiB/s wr, 179 op/s 2024-03-28T05:30:10.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:09 smithi113 ceph-mon[37164]: pgmap v201: 129 pgs: 129 active+clean; 235 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 2.3 MiB/s wr, 179 op/s 2024-03-28T05:30:10.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:09 smithi023 ceph-mon[31954]: pgmap v201: 129 pgs: 129 active+clean; 235 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 2.3 MiB/s wr, 179 op/s 2024-03-28T05:30:12.222 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-28T05:30:12.222 INFO:tasks.workunit.client.0.smithi023.stdout:Final score for writes: 70 2024-03-28T05:30:12.222 INFO:tasks.workunit.client.0.smithi023.stdout:Final score for reads : 123242 2024-03-28T05:30:12.222 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-28T05:30:12.223 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T05:30:12.223 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T05:30:12.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:11 smithi113 ceph-mon[37164]: pgmap v202: 129 pgs: 6 active+clean+snaptrim, 111 active+clean, 12 active+clean+snaptrim_wait; 228 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 1.5 MiB/s wr, 229 op/s 2024-03-28T05:30:12.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:11 smithi023 ceph-mon[31954]: pgmap v202: 129 pgs: 6 active+clean+snaptrim, 111 active+clean, 12 active+clean+snaptrim_wait; 228 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 1.5 MiB/s wr, 229 op/s 2024-03-28T05:30:12.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:11 smithi098 ceph-mon[36087]: pgmap v202: 129 pgs: 6 active+clean+snaptrim, 111 active+clean, 12 active+clean+snaptrim_wait; 228 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 1.5 MiB/s wr, 229 op/s 2024-03-28T05:30:14.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:13 smithi113 ceph-mon[37164]: pgmap v203: 129 pgs: 16 active+clean+snaptrim, 71 active+clean, 42 active+clean+snaptrim_wait; 230 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 727 KiB/s wr, 281 op/s 2024-03-28T05:30:14.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:13 smithi023 ceph-mon[31954]: pgmap v203: 129 pgs: 16 active+clean+snaptrim, 71 active+clean, 42 active+clean+snaptrim_wait; 230 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 727 KiB/s wr, 281 op/s 2024-03-28T05:30:14.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:13 smithi098 ceph-mon[36087]: pgmap v203: 129 pgs: 16 active+clean+snaptrim, 71 active+clean, 42 active+clean+snaptrim_wait; 230 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 727 KiB/s wr, 281 op/s 2024-03-28T05:30:16.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:15 smithi113 ceph-mon[37164]: pgmap v204: 129 pgs: 16 active+clean+snaptrim, 71 active+clean, 42 active+clean+snaptrim_wait; 230 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 727 KiB/s wr, 281 op/s 2024-03-28T05:30:16.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:15 smithi023 ceph-mon[31954]: pgmap v204: 129 pgs: 16 active+clean+snaptrim, 71 active+clean, 42 active+clean+snaptrim_wait; 230 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 727 KiB/s wr, 281 op/s 2024-03-28T05:30:16.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:15 smithi098 ceph-mon[36087]: pgmap v204: 129 pgs: 16 active+clean+snaptrim, 71 active+clean, 42 active+clean+snaptrim_wait; 230 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 727 KiB/s wr, 281 op/s 2024-03-28T05:30:18.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:17 smithi113 ceph-mon[37164]: pgmap v205: 129 pgs: 129 active+clean; 235 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 968 KiB/s wr, 373 op/s 2024-03-28T05:30:18.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:17 smithi023 ceph-mon[31954]: pgmap v205: 129 pgs: 129 active+clean; 235 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 968 KiB/s wr, 373 op/s 2024-03-28T05:30:18.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:17 smithi098 ceph-mon[36087]: pgmap v205: 129 pgs: 129 active+clean; 235 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 968 KiB/s wr, 373 op/s 2024-03-28T05:30:20.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:19 smithi023 ceph-mon[31954]: pgmap v206: 129 pgs: 129 active+clean; 235 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 825 KiB/s wr, 318 op/s 2024-03-28T05:30:20.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:19 smithi023 ceph-mon[31954]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T05:30:20.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:19 smithi098 ceph-mon[36087]: pgmap v206: 129 pgs: 129 active+clean; 235 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 825 KiB/s wr, 318 op/s 2024-03-28T05:30:20.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:19 smithi098 ceph-mon[36087]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T05:30:20.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:19 smithi113 ceph-mon[37164]: pgmap v206: 129 pgs: 129 active+clean; 235 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 825 KiB/s wr, 318 op/s 2024-03-28T05:30:20.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:19 smithi113 ceph-mon[37164]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T05:30:21.642 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.639+0000 7f3e5eeb2640 1 -- 172.21.15.23:0/3180835743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e581007f0 msgr2=0x7f3e58100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.639+0000 7f3e5eeb2640 1 --2- 172.21.15.23:0/3180835743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e581007f0 0x7f3e58100c70 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f3e44009900 tx=0x7f3e4402bc30 comp rx=0 tx=0).stop 2024-03-28T05:30:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.639+0000 7f3e5eeb2640 1 -- 172.21.15.23:0/3180835743 shutdown_connections 2024-03-28T05:30:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.639+0000 7f3e5eeb2640 1 --2- 172.21.15.23:0/3180835743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e581011b0 0x7f3e5810d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.639+0000 7f3e5eeb2640 1 --2- 172.21.15.23:0/3180835743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e581007f0 0x7f3e58100c70 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.639+0000 7f3e5eeb2640 1 --2- 172.21.15.23:0/3180835743 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e58105540 0x7f3e58105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.639+0000 7f3e5eeb2640 1 -- 172.21.15.23:0/3180835743 >> 172.21.15.23:0/3180835743 conn(0x7f3e580773f0 msgr2=0x7f3e58077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.639+0000 7f3e5eeb2640 1 -- 172.21.15.23:0/3180835743 shutdown_connections 2024-03-28T05:30:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.639+0000 7f3e5eeb2640 1 -- 172.21.15.23:0/3180835743 wait complete. 2024-03-28T05:30:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.640+0000 7f3e5eeb2640 1 Processor -- start 2024-03-28T05:30:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.640+0000 7f3e5eeb2640 1 -- start start 2024-03-28T05:30:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.640+0000 7f3e5eeb2640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e581007f0 0x7f3e58102210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.640+0000 7f3e5eeb2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e581011b0 0x7f3e58102750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.640+0000 7f3e5eeb2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e58105540 0x7f3e58102c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.640+0000 7f3e5eeb2640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e58110ab0 con 0x7f3e58105540 2024-03-28T05:30:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.640+0000 7f3e5eeb2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e58110930 con 0x7f3e581011b0 2024-03-28T05:30:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.640+0000 7f3e5eeb2640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e58110c30 con 0x7f3e581007f0 2024-03-28T05:30:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.640+0000 7f3e5d44f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e58105540 0x7f3e58102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.640+0000 7f3e5d44f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e58105540 0x7f3e58102c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45736/0 (socket says 172.21.15.23:45736) 2024-03-28T05:30:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.640+0000 7f3e5d44f640 1 -- 172.21.15.23:0/2897616337 learned_addr learned my addr 172.21.15.23:0/2897616337 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:30:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.640+0000 7f3e5cc4e640 1 --2- 172.21.15.23:0/2897616337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e581007f0 0x7f3e58102210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.640+0000 7f3e57fff640 1 --2- 172.21.15.23:0/2897616337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e581011b0 0x7f3e58102750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.640+0000 7f3e5d44f640 1 -- 172.21.15.23:0/2897616337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e581007f0 msgr2=0x7f3e58102210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.640+0000 7f3e5d44f640 1 --2- 172.21.15.23:0/2897616337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e581007f0 0x7f3e58102210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.640+0000 7f3e5d44f640 1 -- 172.21.15.23:0/2897616337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e581011b0 msgr2=0x7f3e58102750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.640+0000 7f3e5d44f640 1 --2- 172.21.15.23:0/2897616337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e581011b0 0x7f3e58102750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.640+0000 7f3e5d44f640 1 -- 172.21.15.23:0/2897616337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e581a6450 con 0x7f3e58105540 2024-03-28T05:30:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.640+0000 7f3e5d44f640 1 --2- 172.21.15.23:0/2897616337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e58105540 0x7f3e58102c90 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3e5000da40 tx=0x7f3e5000df00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.641+0000 7f3e55ffb640 1 -- 172.21.15.23:0/2897616337 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e50017070 con 0x7f3e58105540 2024-03-28T05:30:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.641+0000 7f3e55ffb640 1 -- 172.21.15.23:0/2897616337 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e50003e50 con 0x7f3e58105540 2024-03-28T05:30:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.641+0000 7f3e5eeb2640 1 -- 172.21.15.23:0/2897616337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e581a6740 con 0x7f3e58105540 2024-03-28T05:30:21.644 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.641+0000 7f3e55ffb640 1 -- 172.21.15.23:0/2897616337 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e50010460 con 0x7f3e58105540 2024-03-28T05:30:21.645 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.641+0000 7f3e5eeb2640 1 -- 172.21.15.23:0/2897616337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e581a6c60 con 0x7f3e58105540 2024-03-28T05:30:21.650 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.643+0000 7f3e55ffb640 1 -- 172.21.15.23:0/2897616337 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f3e50003970 con 0x7f3e58105540 2024-03-28T05:30:21.650 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.643+0000 7f3e61b7d740 1 Processor -- start 2024-03-28T05:30:21.650 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.643+0000 7f3e61b7d740 1 -- start start 2024-03-28T05:30:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.644+0000 7f3e55ffb640 1 --2- 172.21.15.23:0/2897616337 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f3e480a2ac0 0x7f3e480a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.644+0000 7f3e61b7d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5638f07f1710 0x5638f07f1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.644+0000 7f3e61b7d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07f2030 0x5638f07fa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.644+0000 7f3e61b7d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638f07fae00 0x5638f07fd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.644+0000 7f3e61b7d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638f0755b10 con 0x5638f07f2030 2024-03-28T05:30:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.644+0000 7f3e61b7d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638f06fe900 con 0x5638f07fae00 2024-03-28T05:30:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.644+0000 7f3e61b7d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638f0751bb0 con 0x5638f07f1710 2024-03-28T05:30:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.644+0000 7f3e57fff640 1 --2- 172.21.15.23:0/2897616337 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f3e480a2ac0 0x7f3e480a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.644+0000 7f3e55ffb640 1 -- 172.21.15.23:0/2897616337 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f3e500ced80 con 0x7f3e58105540 2024-03-28T05:30:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.644+0000 7f3e57fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07f2030 0x5638f07fa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.644+0000 7f3e5cc4e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5638f07f1710 0x5638f07f1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.644+0000 7f3e57fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07f2030 0x5638f07fa8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45746/0 (socket says 172.21.15.23:45746) 2024-03-28T05:30:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.644+0000 7f3e57fff640 1 -- 172.21.15.23:0/4177218042 learned_addr learned my addr 172.21.15.23:0/4177218042 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:30:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.644+0000 7f3e5d44f640 1 --2- 172.21.15.23:0/4177218042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638f07fae00 0x5638f07fd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.644+0000 7f3e5cc4e640 1 -- 172.21.15.23:0/4177218042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638f07fae00 msgr2=0x5638f07fd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.644+0000 7f3e5cc4e640 1 --2- 172.21.15.23:0/4177218042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638f07fae00 0x5638f07fd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.644+0000 7f3e5cc4e640 1 -- 172.21.15.23:0/4177218042 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07f2030 msgr2=0x5638f07fa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.644+0000 7f3e5cc4e640 1 --2- 172.21.15.23:0/4177218042 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07f2030 0x5638f07fa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.644+0000 7f3e5cc4e640 1 -- 172.21.15.23:0/4177218042 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5638f046a210 con 0x5638f07f1710 2024-03-28T05:30:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.644+0000 7f3e5d44f640 1 --2- 172.21.15.23:0/4177218042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638f07fae00 0x5638f07fd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:30:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.644+0000 7f3e57fff640 1 --2- 172.21.15.23:0/4177218042 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07f2030 0x5638f07fa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:30:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.644+0000 7f3e57fff640 1 --2- 172.21.15.23:0/2897616337 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f3e480a2ac0 0x7f3e480a4f80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3e4402c850 tx=0x7f3e44004120 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.645+0000 7f3e5cc4e640 1 --2- 172.21.15.23:0/4177218042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5638f07f1710 0x5638f07f1af0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009c00 tx=0x7f3e4c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=904d5f094e01b4cf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.645+0000 7f3e42ffd640 1 -- 172.21.15.23:0/4177218042 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c004050 con 0x5638f07f1710 2024-03-28T05:30:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.645+0000 7f3e42ffd640 1 -- 172.21.15.23:0/4177218042 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e4c005500 con 0x5638f07f1710 2024-03-28T05:30:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.645+0000 7f3e42ffd640 1 -- 172.21.15.23:0/4177218042 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c004cd0 con 0x5638f07f1710 2024-03-28T05:30:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.645+0000 7f3e61b7d740 1 -- 172.21.15.23:0/4177218042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5638f07f1710 msgr2=0x5638f07f1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.645+0000 7f3e61b7d740 1 --2- 172.21.15.23:0/4177218042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5638f07f1710 0x5638f07f1af0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009c00 tx=0x7f3e4c011810 comp rx=0 tx=0).stop 2024-03-28T05:30:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.645+0000 7f3e61b7d740 1 -- 172.21.15.23:0/4177218042 shutdown_connections 2024-03-28T05:30:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.645+0000 7f3e61b7d740 1 --2- 172.21.15.23:0/4177218042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638f07fae00 0x5638f07fd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.645+0000 7f3e61b7d740 1 --2- 172.21.15.23:0/4177218042 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07f2030 0x5638f07fa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.645+0000 7f3e61b7d740 1 --2- 172.21.15.23:0/4177218042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5638f07f1710 0x5638f07f1af0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.645+0000 7f3e61b7d740 1 -- 172.21.15.23:0/4177218042 >> 172.21.15.23:0/4177218042 conn(0x5638f07ec030 msgr2=0x5638f07ec410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.645+0000 7f3e61b7d740 1 -- 172.21.15.23:0/4177218042 shutdown_connections 2024-03-28T05:30:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.645+0000 7f3e61b7d740 1 -- 172.21.15.23:0/4177218042 wait complete. 2024-03-28T05:30:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.646+0000 7f3e61b7d740 1 -- start start 2024-03-28T05:30:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.646+0000 7f3e61b7d740 1 Processor -- start 2024-03-28T05:30:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.646+0000 7f3e61b7d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07f1710 0x5638f07f7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.646+0000 7f3e61b7d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5638f07f2030 0x5638f07f8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.646+0000 7f3e61b7d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638f07fae00 0x5638f07f8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.646+0000 7f3e61b7d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638f07fda30 con 0x5638f07f1710 2024-03-28T05:30:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.646+0000 7f3e61b7d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638f0711d50 con 0x5638f07fae00 2024-03-28T05:30:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.646+0000 7f3e61b7d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638f07fdbb0 con 0x5638f07f2030 2024-03-28T05:30:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.646+0000 7f3e61b7d740 10 client.0 _ll_register_callbacks cb 0x5638f0711720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:30:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.646+0000 7f3e5cc4e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07f1710 0x5638f07f7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.646+0000 7f3e5cc4e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07f1710 0x5638f07f7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45748/0 (socket says 172.21.15.23:45748) 2024-03-28T05:30:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.646+0000 7f3e5cc4e640 1 -- 172.21.15.23:0/2657510751 learned_addr learned my addr 172.21.15.23:0/2657510751 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:30:21.653 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.646+0000 7f3e57fff640 1 --2- 172.21.15.23:0/2657510751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5638f07f2030 0x5638f07f8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:21.654 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.646+0000 7f3e5d44f640 1 --2- 172.21.15.23:0/2657510751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638f07fae00 0x5638f07f8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:21.654 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.647+0000 7f3e5cc4e640 1 -- 172.21.15.23:0/2657510751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5638f07f2030 msgr2=0x5638f07f8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:21.654 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.647+0000 7f3e5cc4e640 1 --2- 172.21.15.23:0/2657510751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5638f07f2030 0x5638f07f8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.654 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.647+0000 7f3e5cc4e640 1 -- 172.21.15.23:0/2657510751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638f07fae00 msgr2=0x5638f07f8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:21.654 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.647+0000 7f3e5cc4e640 1 --2- 172.21.15.23:0/2657510751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638f07fae00 0x5638f07f8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.654 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.647+0000 7f3e5cc4e640 1 -- 172.21.15.23:0/2657510751 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5638f0536620 con 0x5638f07f1710 2024-03-28T05:30:21.654 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.647+0000 7f3e5cc4e640 1 --2- 172.21.15.23:0/2657510751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07f1710 0x5638f07f7e00 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009bd0 tx=0x7f3e4c0056b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f6a13cafcff9df server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:21.654 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.647+0000 7f3e37fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:30:21.654 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.647+0000 7f3e37fff640 1 -- 172.21.15.23:0/2657510751 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c029070 con 0x5638f07f1710 2024-03-28T05:30:21.654 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.647+0000 7f3e37fff640 1 -- 172.21.15.23:0/2657510751 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f3e4c003a00 con 0x5638f07f1710 2024-03-28T05:30:21.654 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.647+0000 7f3e61b7d740 1 -- 172.21.15.23:0/2657510751 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3e4c005500 con 0x5638f07f1710 2024-03-28T05:30:21.654 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.647+0000 7f3e37fff640 1 -- 172.21.15.23:0/2657510751 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e4c035050 con 0x5638f07f1710 2024-03-28T05:30:21.654 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.647+0000 7f3e61b7d740 10 client.16014 fetch_fsmap learned FSMap version 20 2024-03-28T05:30:21.654 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.647+0000 7f3e61b7d740 1 -- 172.21.15.23:0/2657510751 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5638f0809c50 con 0x5638f07f1710 2024-03-28T05:30:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.647+0000 7f3e37fff640 1 -- 172.21.15.23:0/2657510751 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e4c005500 con 0x5638f07f1710 2024-03-28T05:30:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.648+0000 7f3e37fff640 1 -- 172.21.15.23:0/2657510751 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f3e4c0051a0 con 0x5638f07f1710 2024-03-28T05:30:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.648+0000 7f3e61b7d740 10 client.16014 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:30:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.648+0000 7f3e61b7d740 10 client.16014 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:30:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.648+0000 7f3e61b7d740 20 client.16014 populate_metadata read hostname 'smithi023' 2024-03-28T05:30:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.648+0000 7f3e61b7d740 1 --2- 172.21.15.23:0/2657510751 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5638f080b000 0x5638f080d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.648+0000 7f3e61b7d740 4 client.16014 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:30:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.648+0000 7f3e61b7d740 1 -- 172.21.15.23:0/2657510751 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5638f0763050 con 0x5638f080b000 2024-03-28T05:30:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.648+0000 7f3e57fff640 1 --2- 172.21.15.23:0/2657510751 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5638f080b000 0x5638f080d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.648+0000 7f3e57fff640 1 --2- 172.21.15.23:0/2657510751 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5638f080b000 0x5638f080d3f0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b77384de24650d2f server_cookie=c2934b611e29c51d in_seq=0 out_seq=0 2024-03-28T05:30:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.648+0000 7f3e37fff640 10 client.16014 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:30:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.649+0000 7f3e37fff640 1 -- 172.21.15.23:0/2657510751 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5638f0763050 con 0x5638f080b000 2024-03-28T05:30:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.649+0000 7f3e37fff640 10 client.16014 handle_command_reply: tid=0 2024-03-28T05:30:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.649+0000 7f3e61b7d740 1 client.16014 shutdown 2024-03-28T05:30:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.649+0000 7f3e61b7d740 1 -- 172.21.15.23:0/2657510751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07f1710 msgr2=0x5638f07f7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.649+0000 7f3e61b7d740 1 --2- 172.21.15.23:0/2657510751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07f1710 0x5638f07f7e00 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009bd0 tx=0x7f3e4c0056b0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.649+0000 7f3e61b7d740 1 -- 172.21.15.23:0/2657510751 shutdown_connections 2024-03-28T05:30:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.649+0000 7f3e61b7d740 1 --2- 172.21.15.23:0/2657510751 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5638f080b000 0x5638f080d3f0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.656 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.649+0000 7f3e61b7d740 1 --2- 172.21.15.23:0/2657510751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638f07fae00 0x5638f07f8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.656 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.649+0000 7f3e61b7d740 1 --2- 172.21.15.23:0/2657510751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5638f07f2030 0x5638f07f8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.656 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.649+0000 7f3e61b7d740 1 --2- 172.21.15.23:0/2657510751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07f1710 0x5638f07f7e00 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.656 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.649+0000 7f3e37fff640 0 client.16014 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:30:21.656 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.649+0000 7f3e61b7d740 1 -- 172.21.15.23:0/2657510751 >> 172.21.15.23:0/2657510751 conn(0x5638f07ec030 msgr2=0x5638f07edd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:21.656 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.649+0000 7f3e61b7d740 1 -- 172.21.15.23:0/2657510751 shutdown_connections 2024-03-28T05:30:21.656 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.649+0000 7f3e61b7d740 1 -- 172.21.15.23:0/2657510751 wait complete. 2024-03-28T05:30:21.656 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.649+0000 7f3e61b7d740 20 client.16014 trim_cache size 0 max 16384 2024-03-28T05:30:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.665+0000 7f3e61b7d740 1 Processor -- start 2024-03-28T05:30:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.665+0000 7f3e61b7d740 1 -- start start 2024-03-28T05:30:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.665+0000 7f3e61b7d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5638f07f1710 0x5638f07fa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.665+0000 7f3e61b7d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07f2030 0x5638f07eaea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.665+0000 7f3e61b7d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638f07fae00 0x5638f08062c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:21.665 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.665+0000 7f3e61b7d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e581aed20 con 0x5638f07f2030 2024-03-28T05:30:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.665+0000 7f3e61b7d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e581b12c0 con 0x5638f07fae00 2024-03-28T05:30:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.665+0000 7f3e61b7d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e581af320 con 0x5638f07f1710 2024-03-28T05:30:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.665+0000 7f3e57fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07f2030 0x5638f07eaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.665+0000 7f3e57fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07f2030 0x5638f07eaea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45764/0 (socket says 172.21.15.23:45764) 2024-03-28T05:30:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.665+0000 7f3e57fff640 1 -- 172.21.15.23:0/2006740105 learned_addr learned my addr 172.21.15.23:0/2006740105 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:30:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.665+0000 7f3e5cc4e640 1 --2- 172.21.15.23:0/2006740105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5638f07f1710 0x5638f07fa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.665+0000 7f3e5d44f640 1 --2- 172.21.15.23:0/2006740105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638f07fae00 0x5638f08062c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.665+0000 7f3e57fff640 1 -- 172.21.15.23:0/2006740105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5638f07f1710 msgr2=0x5638f07fa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.665+0000 7f3e57fff640 1 --2- 172.21.15.23:0/2006740105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5638f07f1710 0x5638f07fa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.665+0000 7f3e57fff640 1 -- 172.21.15.23:0/2006740105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638f07fae00 msgr2=0x5638f08062c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.665+0000 7f3e57fff640 1 --2- 172.21.15.23:0/2006740105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638f07fae00 0x5638f08062c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.665+0000 7f3e57fff640 1 -- 172.21.15.23:0/2006740105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5638f07ebe60 con 0x5638f07f2030 2024-03-28T05:30:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.665+0000 7f3e57fff640 1 --2- 172.21.15.23:0/2006740105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07f2030 0x5638f07eaea0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f3e4402c170 tx=0x7f3e4402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46702f7fc4bd4834 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.666+0000 7f3e427fc640 1 -- 172.21.15.23:0/2006740105 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4405b070 con 0x5638f07f2030 2024-03-28T05:30:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.666+0000 7f3e427fc640 1 -- 172.21.15.23:0/2006740105 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e44004ba0 con 0x5638f07f2030 2024-03-28T05:30:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.666+0000 7f3e427fc640 1 -- 172.21.15.23:0/2006740105 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e44055520 con 0x5638f07f2030 2024-03-28T05:30:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.666+0000 7f3e61b7d740 1 -- 172.21.15.23:0/2006740105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07f2030 msgr2=0x5638f07eaea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.666+0000 7f3e61b7d740 1 --2- 172.21.15.23:0/2006740105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07f2030 0x5638f07eaea0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f3e4402c170 tx=0x7f3e4402bc30 comp rx=0 tx=0).stop 2024-03-28T05:30:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.666+0000 7f3e61b7d740 1 -- 172.21.15.23:0/2006740105 shutdown_connections 2024-03-28T05:30:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.666+0000 7f3e61b7d740 1 --2- 172.21.15.23:0/2006740105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638f07fae00 0x5638f08062c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.666+0000 7f3e61b7d740 1 --2- 172.21.15.23:0/2006740105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07f2030 0x5638f07eaea0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.666+0000 7f3e61b7d740 1 --2- 172.21.15.23:0/2006740105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5638f07f1710 0x5638f07fa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.666+0000 7f3e61b7d740 1 -- 172.21.15.23:0/2006740105 >> 172.21.15.23:0/2006740105 conn(0x5638f07ec030 msgr2=0x5638f07ea790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.666+0000 7f3e61b7d740 1 -- 172.21.15.23:0/2006740105 shutdown_connections 2024-03-28T05:30:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.666+0000 7f3e61b7d740 1 -- 172.21.15.23:0/2006740105 wait complete. 2024-03-28T05:30:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.667+0000 7f3e61b7d740 1 -- start start 2024-03-28T05:30:21.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.667+0000 7f3e61b7d740 1 Processor -- start 2024-03-28T05:30:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.667+0000 7f3e61b7d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5638f07f1710 0x5638f07f5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.667+0000 7f3e61b7d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638f07f2030 0x5638f07f40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.667+0000 7f3e61b7d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07fae00 0x5638f07f4610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.667+0000 7f3e61b7d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e581af940 con 0x5638f07fae00 2024-03-28T05:30:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.667+0000 7f3e61b7d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e581b0c60 con 0x5638f07f2030 2024-03-28T05:30:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.667+0000 7f3e61b7d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e581affa0 con 0x5638f07f1710 2024-03-28T05:30:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.667+0000 7f3e61b7d740 10 client.0 _ll_register_callbacks cb 0x7f3e4c009a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:30:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.667+0000 7f3e5d44f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07fae00 0x5638f07f4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.667+0000 7f3e5d44f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07fae00 0x5638f07f4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45778/0 (socket says 172.21.15.23:45778) 2024-03-28T05:30:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.667+0000 7f3e5cc4e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5638f07f1710 0x5638f07f5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.667+0000 7f3e5d44f640 1 -- 172.21.15.23:0/2767164247 learned_addr learned my addr 172.21.15.23:0/2767164247 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:30:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.667+0000 7f3e57fff640 1 --2- 172.21.15.23:0/2767164247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638f07f2030 0x5638f07f40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.667+0000 7f3e5d44f640 1 -- 172.21.15.23:0/2767164247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5638f07f1710 msgr2=0x5638f07f5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.667+0000 7f3e5d44f640 1 --2- 172.21.15.23:0/2767164247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5638f07f1710 0x5638f07f5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.667+0000 7f3e5d44f640 1 -- 172.21.15.23:0/2767164247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638f07f2030 msgr2=0x5638f07f40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.667+0000 7f3e5d44f640 1 --2- 172.21.15.23:0/2767164247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638f07f2030 0x5638f07f40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.668+0000 7f3e5d44f640 1 -- 172.21.15.23:0/2767164247 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5638f07fd1f0 con 0x5638f07fae00 2024-03-28T05:30:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.668+0000 7f3e5d44f640 1 --2- 172.21.15.23:0/2767164247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07fae00 0x5638f07f4610 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3e5000b6f0 tx=0x7f3e5009b180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5407860e2403cbe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.668+0000 7f3e37fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:30:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.668+0000 7f3e37fff640 1 -- 172.21.15.23:0/2767164247 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e50020040 con 0x5638f07fae00 2024-03-28T05:30:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.668+0000 7f3e37fff640 1 -- 172.21.15.23:0/2767164247 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f3e500cc300 con 0x5638f07fae00 2024-03-28T05:30:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.668+0000 7f3e61b7d740 1 -- 172.21.15.23:0/2767164247 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3e44004ba0 con 0x5638f07fae00 2024-03-28T05:30:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.668+0000 7f3e37fff640 1 -- 172.21.15.23:0/2767164247 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e500cc500 con 0x5638f07fae00 2024-03-28T05:30:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.668+0000 7f3e61b7d740 10 client.16020 fetch_fsmap learned FSMap version 20 2024-03-28T05:30:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.668+0000 7f3e61b7d740 1 -- 172.21.15.23:0/2767164247 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5638f0866410 con 0x5638f07fae00 2024-03-28T05:30:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.668+0000 7f3e37fff640 1 -- 172.21.15.23:0/2767164247 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e44004ba0 con 0x5638f07fae00 2024-03-28T05:30:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.668+0000 7f3e37fff640 1 -- 172.21.15.23:0/2767164247 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f3e50096d90 con 0x5638f07fae00 2024-03-28T05:30:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.668+0000 7f3e61b7d740 10 client.16020 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:30:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.668+0000 7f3e61b7d740 10 client.16020 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:30:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.668+0000 7f3e61b7d740 20 client.16020 populate_metadata read hostname 'smithi023' 2024-03-28T05:30:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.668+0000 7f3e61b7d740 1 --2- 172.21.15.23:0/2767164247 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5638f0866b30 0x5638f0868f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.668+0000 7f3e61b7d740 4 client.16020 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:30:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.668+0000 7f3e61b7d740 1 -- 172.21.15.23:0/2767164247 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5638f075d3a0 con 0x5638f0866b30 2024-03-28T05:30:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.669+0000 7f3e5cc4e640 1 --2- 172.21.15.23:0/2767164247 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5638f0866b30 0x5638f0868f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.669+0000 7f3e5cc4e640 1 --2- 172.21.15.23:0/2767164247 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5638f0866b30 0x5638f0868f20 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68374a7f6b1c7a80 server_cookie=1e731f583651b9f8 in_seq=0 out_seq=0 2024-03-28T05:30:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.669+0000 7f3e37fff640 10 client.16020 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:30:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.987+0000 7f3e37fff640 1 -- 172.21.15.23:0/2767164247 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5638f075d3a0 con 0x5638f0866b30 2024-03-28T05:30:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.987+0000 7f3e37fff640 10 client.16020 handle_command_reply: tid=0 2024-03-28T05:30:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.987+0000 7f3e61b7d740 1 client.16020 shutdown 2024-03-28T05:30:21.987 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.987+0000 7f3e61b7d740 1 -- 172.21.15.23:0/2767164247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07fae00 msgr2=0x5638f07f4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.987+0000 7f3e61b7d740 1 --2- 172.21.15.23:0/2767164247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07fae00 0x5638f07f4610 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3e5000b6f0 tx=0x7f3e5009b180 comp rx=0 tx=0).stop 2024-03-28T05:30:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.987+0000 7f3e61b7d740 1 -- 172.21.15.23:0/2767164247 shutdown_connections 2024-03-28T05:30:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.987+0000 7f3e61b7d740 1 --2- 172.21.15.23:0/2767164247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5638f07fae00 0x5638f07f4610 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.987+0000 7f3e61b7d740 1 --2- 172.21.15.23:0/2767164247 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5638f0866b30 0x5638f0868f20 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.987+0000 7f3e61b7d740 1 --2- 172.21.15.23:0/2767164247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638f07f2030 0x5638f07f40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.987+0000 7f3e61b7d740 1 --2- 172.21.15.23:0/2767164247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5638f07f1710 0x5638f07f5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.987+0000 7f3e61b7d740 1 -- 172.21.15.23:0/2767164247 >> 172.21.15.23:0/2767164247 conn(0x5638f07ec030 msgr2=0x5638f07f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.987+0000 7f3e37fff640 0 client.16020 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:30:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.987+0000 7f3e61b7d740 1 -- 172.21.15.23:0/2767164247 shutdown_connections 2024-03-28T05:30:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.987+0000 7f3e61b7d740 1 -- 172.21.15.23:0/2767164247 wait complete. 2024-03-28T05:30:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.987+0000 7f3e61b7d740 20 client.16020 trim_cache size 0 max 16384 2024-03-28T05:30:21.988 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:30:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:30:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:30:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:30:21.989 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.988+0000 7f3e5eeb2640 1 -- 172.21.15.23:0/2897616337 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f3e480a2ac0 msgr2=0x7f3e480a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.988+0000 7f3e5eeb2640 1 --2- 172.21.15.23:0/2897616337 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f3e480a2ac0 0x7f3e480a4f80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3e4402c850 tx=0x7f3e44004120 comp rx=0 tx=0).stop 2024-03-28T05:30:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.988+0000 7f3e5eeb2640 1 -- 172.21.15.23:0/2897616337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e58105540 msgr2=0x7f3e58102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.988+0000 7f3e5eeb2640 1 --2- 172.21.15.23:0/2897616337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e58105540 0x7f3e58102c90 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3e5000da40 tx=0x7f3e5000df00 comp rx=0 tx=0).stop 2024-03-28T05:30:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.988+0000 7f3e5eeb2640 1 -- 172.21.15.23:0/2897616337 shutdown_connections 2024-03-28T05:30:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.988+0000 7f3e5eeb2640 1 --2- 172.21.15.23:0/2897616337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e58105540 0x7f3e58102c90 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.988+0000 7f3e5eeb2640 1 --2- 172.21.15.23:0/2897616337 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f3e480a2ac0 0x7f3e480a4f80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.988+0000 7f3e5eeb2640 1 --2- 172.21.15.23:0/2897616337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e581011b0 0x7f3e58102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.988+0000 7f3e5eeb2640 1 --2- 172.21.15.23:0/2897616337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e581007f0 0x7f3e58102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.988+0000 7f3e5eeb2640 1 -- 172.21.15.23:0/2897616337 >> 172.21.15.23:0/2897616337 conn(0x7f3e580773f0 msgr2=0x7f3e58104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.988+0000 7f3e5eeb2640 1 -- 172.21.15.23:0/2897616337 shutdown_connections 2024-03-28T05:30:21.990 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:21.988+0000 7f3e5eeb2640 1 -- 172.21.15.23:0/2897616337 wait complete. 2024-03-28T05:30:22.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:21 smithi023 ceph-mon[31954]: pgmap v208: 129 pgs: 129 active+clean; 239 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 1.3 MiB/s wr, 391 op/s 2024-03-28T05:30:22.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:21 smithi098 ceph-mon[36087]: pgmap v208: 129 pgs: 129 active+clean; 239 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 1.3 MiB/s wr, 391 op/s 2024-03-28T05:30:22.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:21 smithi113 ceph-mon[37164]: pgmap v208: 129 pgs: 129 active+clean; 239 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 1.3 MiB/s wr, 391 op/s 2024-03-28T05:30:24.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:23 smithi023 ceph-mon[31954]: pgmap v209: 129 pgs: 129 active+clean; 247 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 414 op/s 2024-03-28T05:30:24.045 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:23 smithi113 ceph-mon[37164]: pgmap v209: 129 pgs: 129 active+clean; 247 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 414 op/s 2024-03-28T05:30:24.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:23 smithi098 ceph-mon[36087]: pgmap v209: 129 pgs: 129 active+clean; 247 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 414 op/s 2024-03-28T05:30:26.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:25 smithi023 ceph-mon[31954]: pgmap v210: 129 pgs: 129 active+clean; 247 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 414 op/s 2024-03-28T05:30:26.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:25 smithi098 ceph-mon[36087]: pgmap v210: 129 pgs: 129 active+clean; 247 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 414 op/s 2024-03-28T05:30:26.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:25 smithi113 ceph-mon[37164]: pgmap v210: 129 pgs: 129 active+clean; 247 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 414 op/s 2024-03-28T05:30:28.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:27 smithi023 ceph-mon[31954]: pgmap v211: 129 pgs: 129 active+clean; 240 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 644 op/s 2024-03-28T05:30:28.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:27 smithi098 ceph-mon[36087]: pgmap v211: 129 pgs: 129 active+clean; 240 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 644 op/s 2024-03-28T05:30:28.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:27 smithi113 ceph-mon[37164]: pgmap v211: 129 pgs: 129 active+clean; 240 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 644 op/s 2024-03-28T05:30:30.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:29 smithi023 ceph-mon[31954]: pgmap v212: 129 pgs: 129 active+clean; 240 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 644 op/s 2024-03-28T05:30:30.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:29 smithi098 ceph-mon[36087]: pgmap v212: 129 pgs: 129 active+clean; 240 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 644 op/s 2024-03-28T05:30:30.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:29 smithi113 ceph-mon[37164]: pgmap v212: 129 pgs: 129 active+clean; 240 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 644 op/s 2024-03-28T05:30:32.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:31 smithi023 ceph-mon[31954]: pgmap v213: 129 pgs: 129 active+clean; 241 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 658 op/s 2024-03-28T05:30:32.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:31 smithi098 ceph-mon[36087]: pgmap v213: 129 pgs: 129 active+clean; 241 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 658 op/s 2024-03-28T05:30:32.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:31 smithi113 ceph-mon[37164]: pgmap v213: 129 pgs: 129 active+clean; 241 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 658 op/s 2024-03-28T05:30:34.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:33 smithi023 ceph-mon[31954]: pgmap v214: 129 pgs: 129 active+clean; 245 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 693 op/s 2024-03-28T05:30:34.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:33 smithi098 ceph-mon[36087]: pgmap v214: 129 pgs: 129 active+clean; 245 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 693 op/s 2024-03-28T05:30:34.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:33 smithi113 ceph-mon[37164]: pgmap v214: 129 pgs: 129 active+clean; 245 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 693 op/s 2024-03-28T05:30:35.079 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.077+0000 7fc80776f640 1 -- 172.21.15.23:0/46696452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8001066d0 msgr2=0x7fc800106ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:35.079 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.077+0000 7fc80776f640 1 --2- 172.21.15.23:0/46696452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8001066d0 0x7fc800106ab0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc009930 tx=0x7fc7fc02bc30 comp rx=0 tx=0).stop 2024-03-28T05:30:35.079 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.077+0000 7fc80776f640 1 -- 172.21.15.23:0/46696452 shutdown_connections 2024-03-28T05:30:35.079 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.077+0000 7fc80776f640 1 --2- 172.21.15.23:0/46696452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8001011b0 0x7fc80010d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.077+0000 7fc80776f640 1 --2- 172.21.15.23:0/46696452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8001007f0 0x7fc800100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.077+0000 7fc80776f640 1 --2- 172.21.15.23:0/46696452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8001066d0 0x7fc800106ab0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.077+0000 7fc80776f640 1 -- 172.21.15.23:0/46696452 >> 172.21.15.23:0/46696452 conn(0x7fc800077400 msgr2=0x7fc800077810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.077+0000 7fc80776f640 1 -- 172.21.15.23:0/46696452 shutdown_connections 2024-03-28T05:30:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.077+0000 7fc80776f640 1 -- 172.21.15.23:0/46696452 wait complete. 2024-03-28T05:30:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.078+0000 7fc80776f640 1 Processor -- start 2024-03-28T05:30:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.078+0000 7fc80776f640 1 -- start start 2024-03-28T05:30:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.078+0000 7fc80776f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8001007f0 0x7fc800102200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.078+0000 7fc80776f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8001011b0 0x7fc800102740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.078+0000 7fc80776f640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8001066d0 0x7fc800102c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.078+0000 7fc80776f640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc800110a50 con 0x7fc8001007f0 2024-03-28T05:30:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.078+0000 7fc80776f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8001108d0 con 0x7fc8001011b0 2024-03-28T05:30:35.080 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.078+0000 7fc80776f640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc800110bd0 con 0x7fc8001066d0 2024-03-28T05:30:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.078+0000 7fc80550b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8001007f0 0x7fc800102200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.078+0000 7fc80550b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8001007f0 0x7fc800102200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34548/0 (socket says 172.21.15.23:34548) 2024-03-28T05:30:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.078+0000 7fc80550b640 1 -- 172.21.15.23:0/3942684878 learned_addr learned my addr 172.21.15.23:0/3942684878 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:30:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.078+0000 7fc804d0a640 1 --2- 172.21.15.23:0/3942684878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8001011b0 0x7fc800102740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.078+0000 7fc805d0c640 1 --2- 172.21.15.23:0/3942684878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8001066d0 0x7fc800102c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.078+0000 7fc80550b640 1 -- 172.21.15.23:0/3942684878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8001066d0 msgr2=0x7fc800102c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.078+0000 7fc80550b640 1 --2- 172.21.15.23:0/3942684878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8001066d0 0x7fc800102c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.078+0000 7fc80550b640 1 -- 172.21.15.23:0/3942684878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8001011b0 msgr2=0x7fc800102740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.078+0000 7fc80550b640 1 --2- 172.21.15.23:0/3942684878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8001011b0 0x7fc800102740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.078+0000 7fc80550b640 1 -- 172.21.15.23:0/3942684878 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8001a6420 con 0x7fc8001007f0 2024-03-28T05:30:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.078+0000 7fc804d0a640 1 --2- 172.21.15.23:0/3942684878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8001011b0 0x7fc800102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:30:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.079+0000 7fc80550b640 1 --2- 172.21.15.23:0/3942684878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8001007f0 0x7fc800102200 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc009230 tx=0x7fc7fc02cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.079+0000 7fc7f67fc640 1 -- 172.21.15.23:0/3942684878 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc041070 con 0x7fc8001007f0 2024-03-28T05:30:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.079+0000 7fc7f67fc640 1 -- 172.21.15.23:0/3942684878 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7fc004450 con 0x7fc8001007f0 2024-03-28T05:30:35.081 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.079+0000 7fc80776f640 1 -- 172.21.15.23:0/3942684878 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8001a66b0 con 0x7fc8001007f0 2024-03-28T05:30:35.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.079+0000 7fc7f67fc640 1 -- 172.21.15.23:0/3942684878 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc003e50 con 0x7fc8001007f0 2024-03-28T05:30:35.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.079+0000 7fc80776f640 1 -- 172.21.15.23:0/3942684878 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8001a6bf0 con 0x7fc8001007f0 2024-03-28T05:30:35.088 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.081+0000 7fc7f67fc640 1 -- 172.21.15.23:0/3942684878 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fc7fc044070 con 0x7fc8001007f0 2024-03-28T05:30:35.088 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.081+0000 7fc80a430740 1 Processor -- start 2024-03-28T05:30:35.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.081+0000 7fc80a430740 1 -- start start 2024-03-28T05:30:35.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.082+0000 7fc7f67fc640 1 --2- 172.21.15.23:0/3942684878 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fc7ec0a2c30 0x7fc7ec0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:35.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.082+0000 7fc80a430740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938ad710 0x55d2938adaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:35.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.082+0000 7fc80a430740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2938ae030 0x55d2938b68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:35.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.082+0000 7fc80a430740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2938b6e00 0x55d2938b91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:35.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.082+0000 7fc80a430740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d293811b10 con 0x55d2938ad710 2024-03-28T05:30:35.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.082+0000 7fc80a430740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2937ba900 con 0x55d2938ae030 2024-03-28T05:30:35.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.082+0000 7fc80a430740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d29380dbb0 con 0x55d2938b6e00 2024-03-28T05:30:35.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.082+0000 7fc7f67fc640 1 -- 172.21.15.23:0/3942684878 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fc7fc0f09b0 con 0x7fc8001007f0 2024-03-28T05:30:35.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.082+0000 7fc805d0c640 1 --2- 172.21.15.23:0/3942684878 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fc7ec0a2c30 0x7fc7ec0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:35.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.082+0000 7fc80550b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938ad710 0x55d2938adaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:35.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.082+0000 7fc80550b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938ad710 0x55d2938adaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34564/0 (socket says 172.21.15.23:34564) 2024-03-28T05:30:35.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.082+0000 7fc80550b640 1 -- 172.21.15.23:0/1943821653 learned_addr learned my addr 172.21.15.23:0/1943821653 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:30:35.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.082+0000 7fc805d0c640 1 --2- 172.21.15.23:0/1943821653 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2938b6e00 0x55d2938b91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:35.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.082+0000 7fc804d0a640 1 --2- 172.21.15.23:0/1943821653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2938ae030 0x55d2938b68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:35.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.082+0000 7fc80550b640 1 -- 172.21.15.23:0/1943821653 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2938b6e00 msgr2=0x55d2938b91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:35.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.082+0000 7fc80550b640 1 --2- 172.21.15.23:0/1943821653 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2938b6e00 0x55d2938b91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.082+0000 7fc80550b640 1 -- 172.21.15.23:0/1943821653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2938ae030 msgr2=0x55d2938b68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:35.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.082+0000 7fc80550b640 1 --2- 172.21.15.23:0/1943821653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2938ae030 0x55d2938b68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.082+0000 7fc80550b640 1 -- 172.21.15.23:0/1943821653 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d293526210 con 0x55d2938ad710 2024-03-28T05:30:35.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.082+0000 7fc804d0a640 1 --2- 172.21.15.23:0/1943821653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2938ae030 0x55d2938b68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:30:35.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.082+0000 7fc805d0c640 1 --2- 172.21.15.23:0/3942684878 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fc7ec0a2c30 0x7fc7ec0a50f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8005fd0 tx=0x7fc7f8009d70 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:35.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.082+0000 7fc80550b640 1 --2- 172.21.15.23:0/1943821653 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938ad710 0x55d2938adaf0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc006010 tx=0x7fc7fc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5ba35947141d5bc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:35.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.083+0000 7fc7eb7fe640 1 -- 172.21.15.23:0/1943821653 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc04b040 con 0x55d2938ad710 2024-03-28T05:30:35.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.083+0000 7fc7eb7fe640 1 -- 172.21.15.23:0/1943821653 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7fc004c80 con 0x55d2938ad710 2024-03-28T05:30:35.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.083+0000 7fc7eb7fe640 1 -- 172.21.15.23:0/1943821653 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc0c0e80 con 0x55d2938ad710 2024-03-28T05:30:35.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.083+0000 7fc80a430740 1 -- 172.21.15.23:0/1943821653 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938ad710 msgr2=0x55d2938adaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:35.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.083+0000 7fc80a430740 1 --2- 172.21.15.23:0/1943821653 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938ad710 0x55d2938adaf0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc006010 tx=0x7fc7fc000bc0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.083+0000 7fc80a430740 1 -- 172.21.15.23:0/1943821653 shutdown_connections 2024-03-28T05:30:35.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.083+0000 7fc80a430740 1 --2- 172.21.15.23:0/1943821653 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2938b6e00 0x55d2938b91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.083+0000 7fc80a430740 1 --2- 172.21.15.23:0/1943821653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2938ae030 0x55d2938b68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.083+0000 7fc80a430740 1 --2- 172.21.15.23:0/1943821653 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938ad710 0x55d2938adaf0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.083+0000 7fc80a430740 1 -- 172.21.15.23:0/1943821653 >> 172.21.15.23:0/1943821653 conn(0x55d2938a8030 msgr2=0x55d2938a8410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:35.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.083+0000 7fc80a430740 1 -- 172.21.15.23:0/1943821653 shutdown_connections 2024-03-28T05:30:35.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.083+0000 7fc80a430740 1 -- 172.21.15.23:0/1943821653 wait complete. 2024-03-28T05:30:35.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.084+0000 7fc80a430740 1 -- start start 2024-03-28T05:30:35.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.084+0000 7fc80a430740 1 Processor -- start 2024-03-28T05:30:35.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.084+0000 7fc80a430740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2938ad710 0x55d2938b3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:35.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.084+0000 7fc80a430740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938ae030 0x55d2938b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:35.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.084+0000 7fc80a430740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2938b6e00 0x55d2938b4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:35.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.084+0000 7fc80a430740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2938b97f0 con 0x55d2938ae030 2024-03-28T05:30:35.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.084+0000 7fc80a430740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2937cdd50 con 0x55d2938b6e00 2024-03-28T05:30:35.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.084+0000 7fc80a430740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2938b9970 con 0x55d2938ad710 2024-03-28T05:30:35.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.084+0000 7fc80a430740 10 client.0 _ll_register_callbacks cb 0x55d2937cd720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:30:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.084+0000 7fc804d0a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938ae030 0x55d2938b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.084+0000 7fc804d0a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938ae030 0x55d2938b4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34576/0 (socket says 172.21.15.23:34576) 2024-03-28T05:30:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.084+0000 7fc804d0a640 1 -- 172.21.15.23:0/4018189174 learned_addr learned my addr 172.21.15.23:0/4018189174 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:30:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.084+0000 7fc80550b640 1 --2- 172.21.15.23:0/4018189174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2938ad710 0x55d2938b3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.084+0000 7fc805d0c640 1 --2- 172.21.15.23:0/4018189174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2938b6e00 0x55d2938b4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.085+0000 7fc804d0a640 1 -- 172.21.15.23:0/4018189174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2938ad710 msgr2=0x55d2938b3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.085+0000 7fc804d0a640 1 --2- 172.21.15.23:0/4018189174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2938ad710 0x55d2938b3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.085+0000 7fc804d0a640 1 -- 172.21.15.23:0/4018189174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2938b6e00 msgr2=0x55d2938b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.085+0000 7fc804d0a640 1 --2- 172.21.15.23:0/4018189174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2938b6e00 0x55d2938b4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.085+0000 7fc804d0a640 1 -- 172.21.15.23:0/4018189174 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d2935f2620 con 0x55d2938ae030 2024-03-28T05:30:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.085+0000 7fc804d0a640 1 --2- 172.21.15.23:0/4018189174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938ae030 0x55d2938b4340 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0012d00 tx=0x7fc7f000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2af2c197233cea00 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.085+0000 7fc7e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:30:35.092 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.085+0000 7fc7e8ff9640 1 -- 172.21.15.23:0/4018189174 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0009d90 con 0x55d2938ae030 2024-03-28T05:30:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.085+0000 7fc7e8ff9640 1 -- 172.21.15.23:0/4018189174 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fc7f00120d0 con 0x55d2938ae030 2024-03-28T05:30:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.085+0000 7fc80a430740 1 -- 172.21.15.23:0/4018189174 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc7fc004c80 con 0x55d2938ae030 2024-03-28T05:30:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.085+0000 7fc7e8ff9640 1 -- 172.21.15.23:0/4018189174 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc7f0010430 con 0x55d2938ae030 2024-03-28T05:30:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.085+0000 7fc80a430740 10 client.16032 fetch_fsmap learned FSMap version 20 2024-03-28T05:30:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.085+0000 7fc80a430740 1 -- 172.21.15.23:0/4018189174 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55d2938c59e0 con 0x55d2938ae030 2024-03-28T05:30:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.085+0000 7fc7e8ff9640 1 -- 172.21.15.23:0/4018189174 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7fc004c80 con 0x55d2938ae030 2024-03-28T05:30:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.085+0000 7fc7e8ff9640 1 -- 172.21.15.23:0/4018189174 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fc7f001c5c0 con 0x55d2938ae030 2024-03-28T05:30:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.085+0000 7fc80a430740 10 client.16032 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:30:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.085+0000 7fc80a430740 10 client.16032 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:30:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.085+0000 7fc80a430740 20 client.16032 populate_metadata read hostname 'smithi023' 2024-03-28T05:30:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.085+0000 7fc80a430740 1 --2- 172.21.15.23:0/4018189174 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d2938c6ef0 0x55d2938c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.085+0000 7fc80a430740 4 client.16032 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:30:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.085+0000 7fc80a430740 1 -- 172.21.15.23:0/4018189174 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d29381f050 con 0x55d2938c6ef0 2024-03-28T05:30:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.086+0000 7fc80550b640 1 --2- 172.21.15.23:0/4018189174 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d2938c6ef0 0x55d2938c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.086+0000 7fc80550b640 1 --2- 172.21.15.23:0/4018189174 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d2938c6ef0 0x55d2938c92e0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=442dfe623e78fadc server_cookie=84667a86b657a59 in_seq=0 out_seq=0 2024-03-28T05:30:35.093 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.086+0000 7fc7e8ff9640 10 client.16032 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:30:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.088+0000 7fc7e8ff9640 1 -- 172.21.15.23:0/4018189174 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d29381f050 con 0x55d2938c6ef0 2024-03-28T05:30:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.088+0000 7fc7e8ff9640 10 client.16032 handle_command_reply: tid=0 2024-03-28T05:30:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.088+0000 7fc80a430740 1 client.16032 shutdown 2024-03-28T05:30:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.088+0000 7fc80a430740 1 -- 172.21.15.23:0/4018189174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938ae030 msgr2=0x55d2938b4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.088+0000 7fc80a430740 1 --2- 172.21.15.23:0/4018189174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938ae030 0x55d2938b4340 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0012d00 tx=0x7fc7f000f790 comp rx=0 tx=0).stop 2024-03-28T05:30:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.089+0000 7fc80a430740 1 -- 172.21.15.23:0/4018189174 shutdown_connections 2024-03-28T05:30:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.089+0000 7fc80a430740 1 --2- 172.21.15.23:0/4018189174 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d2938c6ef0 0x55d2938c92e0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.089+0000 7fc80a430740 1 --2- 172.21.15.23:0/4018189174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2938b6e00 0x55d2938b4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.089+0000 7fc7e8ff9640 0 client.16032 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:30:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.089+0000 7fc80a430740 1 --2- 172.21.15.23:0/4018189174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938ae030 0x55d2938b4340 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.089+0000 7fc80a430740 1 --2- 172.21.15.23:0/4018189174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2938ad710 0x55d2938b3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.089+0000 7fc80a430740 1 -- 172.21.15.23:0/4018189174 >> 172.21.15.23:0/4018189174 conn(0x55d2938a8030 msgr2=0x55d2938a9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.089+0000 7fc80a430740 1 -- 172.21.15.23:0/4018189174 shutdown_connections 2024-03-28T05:30:35.094 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.089+0000 7fc80a430740 1 -- 172.21.15.23:0/4018189174 wait complete. 2024-03-28T05:30:35.095 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.089+0000 7fc80a430740 20 client.16032 trim_cache size 0 max 16384 2024-03-28T05:30:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.103+0000 7fc80a430740 1 Processor -- start 2024-03-28T05:30:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.103+0000 7fc80a430740 1 -- start start 2024-03-28T05:30:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.103+0000 7fc80a430740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2938ad710 0x55d2938b6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.103+0000 7fc80a430740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2938ae030 0x55d2938c1f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.103+0000 7fc80a430740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938b6e00 0x55d2938c2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.103+0000 7fc80a430740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8001aedc0 con 0x55d2938b6e00 2024-03-28T05:30:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.103+0000 7fc80a430740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8001b1360 con 0x55d2938ad710 2024-03-28T05:30:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.103+0000 7fc80a430740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8001af3c0 con 0x55d2938ae030 2024-03-28T05:30:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.103+0000 7fc805d0c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938b6e00 0x55d2938c2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.103+0000 7fc805d0c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938b6e00 0x55d2938c2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34582/0 (socket says 172.21.15.23:34582) 2024-03-28T05:30:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.103+0000 7fc805d0c640 1 -- 172.21.15.23:0/71220076 learned_addr learned my addr 172.21.15.23:0/71220076 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:30:35.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.103+0000 7fc804d0a640 1 --2- 172.21.15.23:0/71220076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2938ae030 0x55d2938c1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.104+0000 7fc80550b640 1 --2- 172.21.15.23:0/71220076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2938ad710 0x55d2938b6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.104+0000 7fc805d0c640 1 -- 172.21.15.23:0/71220076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2938ae030 msgr2=0x55d2938c1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.104+0000 7fc805d0c640 1 --2- 172.21.15.23:0/71220076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2938ae030 0x55d2938c1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.104+0000 7fc805d0c640 1 -- 172.21.15.23:0/71220076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2938ad710 msgr2=0x55d2938b6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.104+0000 7fc805d0c640 1 --2- 172.21.15.23:0/71220076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2938ad710 0x55d2938b6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.104+0000 7fc805d0c640 1 -- 172.21.15.23:0/71220076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d2938ade60 con 0x55d2938b6e00 2024-03-28T05:30:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.104+0000 7fc805d0c640 1 --2- 172.21.15.23:0/71220076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938b6e00 0x55d2938c2450 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80098e0 tx=0x7fc7f8009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e6ac09cab6cc7b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.104+0000 7fc7eaffd640 1 -- 172.21.15.23:0/71220076 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8004b20 con 0x55d2938b6e00 2024-03-28T05:30:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.104+0000 7fc7eaffd640 1 -- 172.21.15.23:0/71220076 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7f8004cc0 con 0x55d2938b6e00 2024-03-28T05:30:35.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.104+0000 7fc7eaffd640 1 -- 172.21.15.23:0/71220076 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f80053d0 con 0x55d2938b6e00 2024-03-28T05:30:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.105+0000 7fc80a430740 1 -- 172.21.15.23:0/71220076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938b6e00 msgr2=0x55d2938c2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.105+0000 7fc80a430740 1 --2- 172.21.15.23:0/71220076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938b6e00 0x55d2938c2450 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80098e0 tx=0x7fc7f8009500 comp rx=0 tx=0).stop 2024-03-28T05:30:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.105+0000 7fc80a430740 1 -- 172.21.15.23:0/71220076 shutdown_connections 2024-03-28T05:30:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.105+0000 7fc80a430740 1 --2- 172.21.15.23:0/71220076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938b6e00 0x55d2938c2450 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.105+0000 7fc80a430740 1 --2- 172.21.15.23:0/71220076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2938ae030 0x55d2938c1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.105+0000 7fc80a430740 1 --2- 172.21.15.23:0/71220076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2938ad710 0x55d2938b6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.105+0000 7fc80a430740 1 -- 172.21.15.23:0/71220076 >> 172.21.15.23:0/71220076 conn(0x55d2938a8030 msgr2=0x55d2938a9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.105+0000 7fc80a430740 1 -- 172.21.15.23:0/71220076 shutdown_connections 2024-03-28T05:30:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.105+0000 7fc80a430740 1 -- 172.21.15.23:0/71220076 wait complete. 2024-03-28T05:30:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.105+0000 7fc80a430740 1 -- start start 2024-03-28T05:30:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.105+0000 7fc80a430740 1 Processor -- start 2024-03-28T05:30:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc80a430740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2938ad710 0x55d2938ba560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc80a430740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938ae030 0x55d2938baaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:35.109 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc80a430740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2938b6e00 0x55d2939217b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc80a430740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8001af9e0 con 0x55d2938ae030 2024-03-28T05:30:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc80a430740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8001b0d00 con 0x55d2938b6e00 2024-03-28T05:30:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc80a430740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8001b0040 con 0x55d2938ad710 2024-03-28T05:30:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc80a430740 10 client.0 _ll_register_callbacks cb 0x7fc7f00127f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:30:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc804d0a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938ae030 0x55d2938baaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc804d0a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938ae030 0x55d2938baaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34592/0 (socket says 172.21.15.23:34592) 2024-03-28T05:30:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc804d0a640 1 -- 172.21.15.23:0/1920409257 learned_addr learned my addr 172.21.15.23:0/1920409257 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:30:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc80550b640 1 --2- 172.21.15.23:0/1920409257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2938ad710 0x55d2938ba560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc805d0c640 1 --2- 172.21.15.23:0/1920409257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2938b6e00 0x55d2939217b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc804d0a640 1 -- 172.21.15.23:0/1920409257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2938ad710 msgr2=0x55d2938ba560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc804d0a640 1 --2- 172.21.15.23:0/1920409257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2938ad710 0x55d2938ba560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc804d0a640 1 -- 172.21.15.23:0/1920409257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2938b6e00 msgr2=0x55d2939217b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc804d0a640 1 --2- 172.21.15.23:0/1920409257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2938b6e00 0x55d2939217b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc804d0a640 1 -- 172.21.15.23:0/1920409257 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d293921cf0 con 0x55d2938ae030 2024-03-28T05:30:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc804d0a640 1 --2- 172.21.15.23:0/1920409257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938ae030 0x55d2938baaa0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x55d2938b53d0 tx=0x7fc7f0012150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e9db547ad5ab048 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:35.110 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc7e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:30:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc7e8ff9640 1 -- 172.21.15.23:0/1920409257 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f001c630 con 0x55d2938ae030 2024-03-28T05:30:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc7e8ff9640 1 -- 172.21.15.23:0/1920409257 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fc7f00050d0 con 0x55d2938ae030 2024-03-28T05:30:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc80a430740 1 -- 172.21.15.23:0/1920409257 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc7f8004cc0 con 0x55d2938ae030 2024-03-28T05:30:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc7e8ff9640 1 -- 172.21.15.23:0/1920409257 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc7f00052d0 con 0x55d2938ae030 2024-03-28T05:30:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc80a430740 10 client.16038 fetch_fsmap learned FSMap version 20 2024-03-28T05:30:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.106+0000 7fc80a430740 1 -- 172.21.15.23:0/1920409257 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55d293923030 con 0x55d2938ae030 2024-03-28T05:30:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.107+0000 7fc7e8ff9640 1 -- 172.21.15.23:0/1920409257 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7f8004cc0 con 0x55d2938ae030 2024-03-28T05:30:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.107+0000 7fc7e8ff9640 1 -- 172.21.15.23:0/1920409257 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fc7f00105d0 con 0x55d2938ae030 2024-03-28T05:30:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.107+0000 7fc80a430740 10 client.16038 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:30:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.107+0000 7fc80a430740 10 client.16038 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:30:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.107+0000 7fc80a430740 20 client.16038 populate_metadata read hostname 'smithi023' 2024-03-28T05:30:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.107+0000 7fc80a430740 1 --2- 172.21.15.23:0/1920409257 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d293923770 0x55d293925b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.107+0000 7fc80a430740 4 client.16038 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:30:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.107+0000 7fc80a430740 1 -- 172.21.15.23:0/1920409257 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d2938193a0 con 0x55d293923770 2024-03-28T05:30:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.107+0000 7fc80550b640 1 --2- 172.21.15.23:0/1920409257 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d293923770 0x55d293925b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.107+0000 7fc80550b640 1 --2- 172.21.15.23:0/1920409257 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d293923770 0x55d293925b90 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3d59c71aad0028d server_cookie=79c10b40fa9397d in_seq=0 out_seq=0 2024-03-28T05:30:35.111 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.107+0000 7fc7e8ff9640 10 client.16038 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:30:35.898 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.898+0000 7fc7e8ff9640 1 -- 172.21.15.23:0/1920409257 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d2938193a0 con 0x55d293923770 2024-03-28T05:30:35.898 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.898+0000 7fc7e8ff9640 10 client.16038 handle_command_reply: tid=0 2024-03-28T05:30:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.898+0000 7fc80a430740 1 client.16038 shutdown 2024-03-28T05:30:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.898+0000 7fc80a430740 1 -- 172.21.15.23:0/1920409257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938ae030 msgr2=0x55d2938baaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.898+0000 7fc80a430740 1 --2- 172.21.15.23:0/1920409257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938ae030 0x55d2938baaa0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x55d2938b53d0 tx=0x7fc7f0012150 comp rx=0 tx=0).stop 2024-03-28T05:30:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.898+0000 7fc80a430740 1 -- 172.21.15.23:0/1920409257 shutdown_connections 2024-03-28T05:30:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.898+0000 7fc80a430740 1 --2- 172.21.15.23:0/1920409257 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55d293923770 0x55d293925b90 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.898+0000 7fc7e8ff9640 0 client.16038 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:30:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.898+0000 7fc80a430740 1 --2- 172.21.15.23:0/1920409257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d2938b6e00 0x55d2939217b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.898+0000 7fc80a430740 1 --2- 172.21.15.23:0/1920409257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d2938ae030 0x55d2938baaa0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.898+0000 7fc80a430740 1 --2- 172.21.15.23:0/1920409257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2938ad710 0x55d2938ba560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.899 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.898+0000 7fc80a430740 1 -- 172.21.15.23:0/1920409257 >> 172.21.15.23:0/1920409257 conn(0x55d2938a8030 msgr2=0x55d2938a6790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.899+0000 7fc80a430740 1 -- 172.21.15.23:0/1920409257 shutdown_connections 2024-03-28T05:30:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.899+0000 7fc80a430740 1 -- 172.21.15.23:0/1920409257 wait complete. 2024-03-28T05:30:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.899+0000 7fc80a430740 20 client.16038 trim_cache size 0 max 16384 2024-03-28T05:30:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:30:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:30:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:30:35.900 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:30:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.900+0000 7fc80776f640 1 -- 172.21.15.23:0/3942684878 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fc7ec0a2c30 msgr2=0x7fc7ec0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.900+0000 7fc80776f640 1 --2- 172.21.15.23:0/3942684878 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fc7ec0a2c30 0x7fc7ec0a50f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8005fd0 tx=0x7fc7f8009d70 comp rx=0 tx=0).stop 2024-03-28T05:30:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.900+0000 7fc80776f640 1 -- 172.21.15.23:0/3942684878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8001007f0 msgr2=0x7fc800102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.900+0000 7fc80776f640 1 --2- 172.21.15.23:0/3942684878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8001007f0 0x7fc800102200 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc009230 tx=0x7fc7fc02cb30 comp rx=0 tx=0).stop 2024-03-28T05:30:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.900+0000 7fc80776f640 1 -- 172.21.15.23:0/3942684878 shutdown_connections 2024-03-28T05:30:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.900+0000 7fc80776f640 1 --2- 172.21.15.23:0/3942684878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8001066d0 0x7fc800102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.900+0000 7fc80776f640 1 --2- 172.21.15.23:0/3942684878 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fc7ec0a2c30 0x7fc7ec0a50f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.900+0000 7fc80776f640 1 --2- 172.21.15.23:0/3942684878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8001011b0 0x7fc800102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.900+0000 7fc80776f640 1 --2- 172.21.15.23:0/3942684878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8001007f0 0x7fc800102200 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.900+0000 7fc80776f640 1 -- 172.21.15.23:0/3942684878 >> 172.21.15.23:0/3942684878 conn(0x7fc800077400 msgr2=0x7fc80006a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.900+0000 7fc80776f640 1 -- 172.21.15.23:0/3942684878 shutdown_connections 2024-03-28T05:30:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:35.900+0000 7fc80776f640 1 -- 172.21.15.23:0/3942684878 wait complete. 2024-03-28T05:30:36.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:35 smithi098 ceph-mon[36087]: pgmap v215: 129 pgs: 129 active+clean; 245 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 558 op/s 2024-03-28T05:30:36.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:35 smithi113 ceph-mon[37164]: pgmap v215: 129 pgs: 129 active+clean; 245 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 558 op/s 2024-03-28T05:30:36.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:35 smithi023 ceph-mon[31954]: pgmap v215: 129 pgs: 129 active+clean; 245 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 558 op/s 2024-03-28T05:30:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:37 smithi098 ceph-mon[36087]: pgmap v216: 129 pgs: 129 active+clean; 250 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 788 op/s 2024-03-28T05:30:38.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:37 smithi113 ceph-mon[37164]: pgmap v216: 129 pgs: 129 active+clean; 250 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 788 op/s 2024-03-28T05:30:38.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:37 smithi023 ceph-mon[31954]: pgmap v216: 129 pgs: 129 active+clean; 250 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 788 op/s 2024-03-28T05:30:40.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:39 smithi098 ceph-mon[36087]: pgmap v217: 129 pgs: 129 active+clean; 250 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 1.1 MiB/s wr, 478 op/s 2024-03-28T05:30:40.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:39 smithi113 ceph-mon[37164]: pgmap v217: 129 pgs: 129 active+clean; 250 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 1.1 MiB/s wr, 478 op/s 2024-03-28T05:30:40.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:39 smithi023 ceph-mon[31954]: pgmap v217: 129 pgs: 129 active+clean; 250 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 1.1 MiB/s wr, 478 op/s 2024-03-28T05:30:42.083 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:41 smithi023 ceph-mon[31954]: pgmap v218: 129 pgs: 129 active+clean; 243 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 571 op/s 2024-03-28T05:30:42.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:41 smithi098 ceph-mon[36087]: pgmap v218: 129 pgs: 129 active+clean; 243 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 571 op/s 2024-03-28T05:30:42.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:41 smithi113 ceph-mon[37164]: pgmap v218: 129 pgs: 129 active+clean; 243 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 571 op/s 2024-03-28T05:30:44.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:43 smithi098 ceph-mon[36087]: pgmap v219: 129 pgs: 129 active+clean; 245 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 658 op/s 2024-03-28T05:30:44.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:43 smithi113 ceph-mon[37164]: pgmap v219: 129 pgs: 129 active+clean; 245 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 658 op/s 2024-03-28T05:30:44.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:43 smithi023 ceph-mon[31954]: pgmap v219: 129 pgs: 129 active+clean; 245 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 658 op/s 2024-03-28T05:30:46.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:45 smithi098 ceph-mon[36087]: pgmap v220: 129 pgs: 129 active+clean; 245 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 492 op/s 2024-03-28T05:30:46.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:45 smithi113 ceph-mon[37164]: pgmap v220: 129 pgs: 129 active+clean; 245 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 492 op/s 2024-03-28T05:30:46.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:45 smithi023 ceph-mon[31954]: pgmap v220: 129 pgs: 129 active+clean; 245 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 492 op/s 2024-03-28T05:30:47.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:46 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:30:47.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:46 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:30:47.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:46 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:30:48.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:47 smithi098 ceph-mon[36087]: pgmap v221: 129 pgs: 129 active+clean; 261 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 630 op/s 2024-03-28T05:30:48.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:47 smithi113 ceph-mon[37164]: pgmap v221: 129 pgs: 129 active+clean; 261 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 630 op/s 2024-03-28T05:30:48.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:47 smithi023 ceph-mon[31954]: pgmap v221: 129 pgs: 129 active+clean; 261 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 630 op/s 2024-03-28T05:30:49.001 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.997+0000 7fa3e064b640 1 -- 172.21.15.23:0/3928498471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d8068d20 msgr2=0x7fa3d8069180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:49.002 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.997+0000 7fa3e064b640 1 --2- 172.21.15.23:0/3928498471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d8068d20 0x7fa3d8069180 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009930 tx=0x7fa3c802bc30 comp rx=0 tx=0).stop 2024-03-28T05:30:49.002 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.997+0000 7fa3e064b640 1 -- 172.21.15.23:0/3928498471 shutdown_connections 2024-03-28T05:30:49.002 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.997+0000 7fa3e064b640 1 --2- 172.21.15.23:0/3928498471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3d80696c0 0x7fa3d810e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.002 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.997+0000 7fa3e064b640 1 --2- 172.21.15.23:0/3928498471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d8068d20 0x7fa3d8069180 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.002 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.997+0000 7fa3e064b640 1 --2- 172.21.15.23:0/3928498471 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3d8107770 0x7fa3d8107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.002 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.997+0000 7fa3e064b640 1 -- 172.21.15.23:0/3928498471 >> 172.21.15.23:0/3928498471 conn(0x7fa3d80773f0 msgr2=0x7fa3d8077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:49.003 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.997+0000 7fa3e064b640 1 -- 172.21.15.23:0/3928498471 shutdown_connections 2024-03-28T05:30:49.003 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.997+0000 7fa3e064b640 1 -- 172.21.15.23:0/3928498471 wait complete. 2024-03-28T05:30:49.003 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.997+0000 7fa3e064b640 1 Processor -- start 2024-03-28T05:30:49.003 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.997+0000 7fa3e064b640 1 -- start start 2024-03-28T05:30:49.003 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.998+0000 7fa3e064b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3d8068d20 0x7fa3d819b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:49.003 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.998+0000 7fa3e064b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d80696c0 0x7fa3d819be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:49.003 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.998+0000 7fa3e064b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3d8107770 0x7fa3d8195af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:49.003 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.998+0000 7fa3e064b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d810f870 con 0x7fa3d80696c0 2024-03-28T05:30:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.998+0000 7fa3e064b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d810f6f0 con 0x7fa3d8107770 2024-03-28T05:30:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.998+0000 7fa3e064b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d810f9f0 con 0x7fa3d8068d20 2024-03-28T05:30:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.998+0000 7fa3ddbe6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d80696c0 0x7fa3d819be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.998+0000 7fa3ddbe6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d80696c0 0x7fa3d819be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47938/0 (socket says 172.21.15.23:47938) 2024-03-28T05:30:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.998+0000 7fa3ddbe6640 1 -- 172.21.15.23:0/3936262886 learned_addr learned my addr 172.21.15.23:0/3936262886 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:30:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.998+0000 7fa3de3e7640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3d8068d20 0x7fa3d819b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.998+0000 7fa3debe8640 1 --2- 172.21.15.23:0/3936262886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3d8107770 0x7fa3d8195af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.998+0000 7fa3ddbe6640 1 -- 172.21.15.23:0/3936262886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3d8068d20 msgr2=0x7fa3d819b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.998+0000 7fa3ddbe6640 1 --2- 172.21.15.23:0/3936262886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3d8068d20 0x7fa3d819b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.998+0000 7fa3ddbe6640 1 -- 172.21.15.23:0/3936262886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3d8107770 msgr2=0x7fa3d8195af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.998+0000 7fa3ddbe6640 1 --2- 172.21.15.23:0/3936262886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3d8107770 0x7fa3d8195af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.998+0000 7fa3ddbe6640 1 -- 172.21.15.23:0/3936262886 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d8196270 con 0x7fa3d80696c0 2024-03-28T05:30:49.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.998+0000 7fa3ddbe6640 1 --2- 172.21.15.23:0/3936262886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d80696c0 0x7fa3d819be30 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009230 tx=0x7fa3c8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:49.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.998+0000 7fa3cf7fe640 1 -- 172.21.15.23:0/3936262886 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8041070 con 0x7fa3d80696c0 2024-03-28T05:30:49.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.998+0000 7fa3cf7fe640 1 -- 172.21.15.23:0/3936262886 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3c80054e0 con 0x7fa3d80696c0 2024-03-28T05:30:49.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.998+0000 7fa3e064b640 1 -- 172.21.15.23:0/3936262886 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d8196500 con 0x7fa3d80696c0 2024-03-28T05:30:49.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.998+0000 7fa3cf7fe640 1 -- 172.21.15.23:0/3936262886 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8003c90 con 0x7fa3d80696c0 2024-03-28T05:30:49.005 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:48.999+0000 7fa3e064b640 1 -- 172.21.15.23:0/3936262886 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d81a27b0 con 0x7fa3d80696c0 2024-03-28T05:30:49.014 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.001+0000 7fa3cf7fe640 1 -- 172.21.15.23:0/3936262886 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fa3c8044070 con 0x7fa3d80696c0 2024-03-28T05:30:49.014 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.002+0000 7fa3e3234740 1 Processor -- start 2024-03-28T05:30:49.014 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.002+0000 7fa3e3234740 1 -- start start 2024-03-28T05:30:49.014 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.002+0000 7fa3e3234740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00028710 0x55fa00028af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:49.014 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.002+0000 7fa3e3234740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa00029030 0x55fa000318c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:49.015 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.002+0000 7fa3e3234740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00031e00 0x55fa000341f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:49.015 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.002+0000 7fa3e3234740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9fff8cb10 con 0x55fa00031e00 2024-03-28T05:30:49.015 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.002+0000 7fa3e3234740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9fff35900 con 0x55fa00029030 2024-03-28T05:30:49.015 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.002+0000 7fa3e3234740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9fff88bb0 con 0x55fa00028710 2024-03-28T05:30:49.015 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.002+0000 7fa3cf7fe640 1 --2- 172.21.15.23:0/3936262886 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fa3c40a2a90 0x7fa3c40a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:49.015 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.002+0000 7fa3ddbe6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00031e00 0x55fa000341f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:49.015 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.002+0000 7fa3debe8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00028710 0x55fa00028af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:49.015 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.002+0000 7fa3debe8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00028710 0x55fa00028af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:34018/0 (socket says 172.21.15.23:34018) 2024-03-28T05:30:49.015 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.002+0000 7fa3debe8640 1 -- 172.21.15.23:0/1711960721 learned_addr learned my addr 172.21.15.23:0/1711960721 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:30:49.015 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.003+0000 7fa3debe8640 1 --2- 172.21.15.23:0/3936262886 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fa3c40a2a90 0x7fa3c40a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:49.015 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.003+0000 7fa3de3e7640 1 --2- 172.21.15.23:0/1711960721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa00029030 0x55fa000318c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:49.016 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.003+0000 7fa3cf7fe640 1 -- 172.21.15.23:0/3936262886 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fa3c80f0c70 con 0x7fa3d80696c0 2024-03-28T05:30:49.016 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.003+0000 7fa3debe8640 1 -- 172.21.15.23:0/1711960721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa00029030 msgr2=0x55fa000318c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:49.016 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.003+0000 7fa3debe8640 1 --2- 172.21.15.23:0/1711960721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa00029030 0x55fa000318c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.016 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.003+0000 7fa3debe8640 1 -- 172.21.15.23:0/1711960721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00031e00 msgr2=0x55fa000341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:49.016 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.003+0000 7fa3debe8640 1 --2- 172.21.15.23:0/1711960721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00031e00 0x55fa000341f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.016 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.003+0000 7fa3debe8640 1 -- 172.21.15.23:0/1711960721 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9ffca1210 con 0x55fa00028710 2024-03-28T05:30:49.016 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.003+0000 7fa3debe8640 1 --2- 172.21.15.23:0/3936262886 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fa3c40a2a90 0x7fa3c40a4f50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa3d000fc90 tx=0x7fa3d001e000 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:49.016 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.003+0000 7fa3debe8640 1 --2- 172.21.15.23:0/1711960721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00028710 0x55fa00028af0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0011e00 tx=0x7fa3d000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98728572ab66d429 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:49.016 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.003+0000 7fa3ccbfe640 1 -- 172.21.15.23:0/1711960721 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d001dcb0 con 0x55fa00028710 2024-03-28T05:30:49.016 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.003+0000 7fa3ccbfe640 1 -- 172.21.15.23:0/1711960721 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3d001de50 con 0x55fa00028710 2024-03-28T05:30:49.016 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.003+0000 7fa3ccbfe640 1 -- 172.21.15.23:0/1711960721 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d0004280 con 0x55fa00028710 2024-03-28T05:30:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.004+0000 7fa3e3234740 1 -- 172.21.15.23:0/1711960721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00028710 msgr2=0x55fa00028af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.004+0000 7fa3e3234740 1 --2- 172.21.15.23:0/1711960721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00028710 0x55fa00028af0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0011e00 tx=0x7fa3d000f790 comp rx=0 tx=0).stop 2024-03-28T05:30:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.004+0000 7fa3e3234740 1 -- 172.21.15.23:0/1711960721 shutdown_connections 2024-03-28T05:30:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.004+0000 7fa3e3234740 1 --2- 172.21.15.23:0/1711960721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00031e00 0x55fa000341f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.004+0000 7fa3e3234740 1 --2- 172.21.15.23:0/1711960721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa00029030 0x55fa000318c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.004+0000 7fa3e3234740 1 --2- 172.21.15.23:0/1711960721 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00028710 0x55fa00028af0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.004+0000 7fa3e3234740 1 -- 172.21.15.23:0/1711960721 >> 172.21.15.23:0/1711960721 conn(0x55fa00023030 msgr2=0x55fa00023410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.004+0000 7fa3e3234740 1 -- 172.21.15.23:0/1711960721 shutdown_connections 2024-03-28T05:30:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.004+0000 7fa3e3234740 1 -- 172.21.15.23:0/1711960721 wait complete. 2024-03-28T05:30:49.017 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.005+0000 7fa3e3234740 1 -- start start 2024-03-28T05:30:49.018 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.005+0000 7fa3e3234740 1 Processor -- start 2024-03-28T05:30:49.018 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.005+0000 7fa3e3234740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00028710 0x55fa0002ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:49.018 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.005+0000 7fa3e3234740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00029030 0x55fa0002f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:49.018 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.005+0000 7fa3e3234740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa00031e00 0x55fa0002f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:49.018 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.005+0000 7fa3e3234740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa00034970 con 0x55fa00028710 2024-03-28T05:30:49.018 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.005+0000 7fa3e3234740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9fff48d50 con 0x55fa00031e00 2024-03-28T05:30:49.018 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.005+0000 7fa3e3234740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa00034af0 con 0x55fa00029030 2024-03-28T05:30:49.018 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.005+0000 7fa3e3234740 10 client.0 _ll_register_callbacks cb 0x55f9fff48720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:30:49.019 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.005+0000 7fa3de3e7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00028710 0x55fa0002ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:49.019 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.005+0000 7fa3de3e7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00028710 0x55fa0002ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47956/0 (socket says 172.21.15.23:47956) 2024-03-28T05:30:49.019 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.005+0000 7fa3de3e7640 1 -- 172.21.15.23:0/1478997696 learned_addr learned my addr 172.21.15.23:0/1478997696 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:30:49.019 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.005+0000 7fa3ddbe6640 1 --2- 172.21.15.23:0/1478997696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00029030 0x55fa0002f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:49.019 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.005+0000 7fa3debe8640 1 --2- 172.21.15.23:0/1478997696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa00031e00 0x55fa0002f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:49.019 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.005+0000 7fa3de3e7640 1 -- 172.21.15.23:0/1478997696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00029030 msgr2=0x55fa0002f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:49.019 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.005+0000 7fa3de3e7640 1 --2- 172.21.15.23:0/1478997696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00029030 0x55fa0002f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.019 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.005+0000 7fa3de3e7640 1 -- 172.21.15.23:0/1478997696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa00031e00 msgr2=0x55fa0002f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:49.019 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.005+0000 7fa3de3e7640 1 --2- 172.21.15.23:0/1478997696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa00031e00 0x55fa0002f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.019 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.005+0000 7fa3de3e7640 1 -- 172.21.15.23:0/1478997696 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f9ffd6d620 con 0x55fa00028710 2024-03-28T05:30:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.005+0000 7fa3ddbe6640 1 --2- 172.21.15.23:0/1478997696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00029030 0x55fa0002f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:30:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.006+0000 7fa3de3e7640 1 --2- 172.21.15.23:0/1478997696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00028710 0x55fa0002ee20 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4004550 tx=0x7fa3d40039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acfbf898710ea436 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.006+0000 7fa3c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:30:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.006+0000 7fa3c1ffb640 1 -- 172.21.15.23:0/1478997696 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d400f220 con 0x55fa00028710 2024-03-28T05:30:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.006+0000 7fa3c1ffb640 1 -- 172.21.15.23:0/1478997696 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fa3d400f3c0 con 0x55fa00028710 2024-03-28T05:30:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.006+0000 7fa3e3234740 1 -- 172.21.15.23:0/1478997696 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa3d001de50 con 0x55fa00028710 2024-03-28T05:30:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.006+0000 7fa3c1ffb640 1 -- 172.21.15.23:0/1478997696 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3d402b050 con 0x55fa00028710 2024-03-28T05:30:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.006+0000 7fa3e3234740 10 client.16050 fetch_fsmap learned FSMap version 20 2024-03-28T05:30:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.006+0000 7fa3e3234740 1 -- 172.21.15.23:0/1478997696 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55fa00040c60 con 0x55fa00028710 2024-03-28T05:30:49.020 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.006+0000 7fa3c1ffb640 1 -- 172.21.15.23:0/1478997696 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3d001de50 con 0x55fa00028710 2024-03-28T05:30:49.021 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.006+0000 7fa3c1ffb640 1 -- 172.21.15.23:0/1478997696 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fa3d400ebc0 con 0x55fa00028710 2024-03-28T05:30:49.021 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.006+0000 7fa3e3234740 10 client.16050 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:30:49.021 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.006+0000 7fa3e3234740 10 client.16050 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:30:49.021 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.006+0000 7fa3e3234740 20 client.16050 populate_metadata read hostname 'smithi023' 2024-03-28T05:30:49.021 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.006+0000 7fa3e3234740 1 --2- 172.21.15.23:0/1478997696 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55fa000420d0 0x55fa000444c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:49.021 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.006+0000 7fa3e3234740 4 client.16050 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:30:49.021 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.006+0000 7fa3e3234740 1 -- 172.21.15.23:0/1478997696 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f9fff9a050 con 0x55fa000420d0 2024-03-28T05:30:49.021 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.007+0000 7fa3ddbe6640 1 --2- 172.21.15.23:0/1478997696 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55fa000420d0 0x55fa000444c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:49.021 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.007+0000 7fa3ddbe6640 1 --2- 172.21.15.23:0/1478997696 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55fa000420d0 0x55fa000444c0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32c778b2c06d3421 server_cookie=354ae6ecbbd2c179 in_seq=0 out_seq=0 2024-03-28T05:30:49.021 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.007+0000 7fa3c1ffb640 10 client.16050 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:30:49.021 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.009+0000 7fa3c1ffb640 1 -- 172.21.15.23:0/1478997696 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f9fff9a050 con 0x55fa000420d0 2024-03-28T05:30:49.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.009+0000 7fa3c1ffb640 10 client.16050 handle_command_reply: tid=0 2024-03-28T05:30:49.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.009+0000 7fa3e3234740 1 client.16050 shutdown 2024-03-28T05:30:49.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.010+0000 7fa3e3234740 1 -- 172.21.15.23:0/1478997696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00028710 msgr2=0x55fa0002ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:49.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.010+0000 7fa3e3234740 1 --2- 172.21.15.23:0/1478997696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00028710 0x55fa0002ee20 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4004550 tx=0x7fa3d40039c0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.010+0000 7fa3e3234740 1 -- 172.21.15.23:0/1478997696 shutdown_connections 2024-03-28T05:30:49.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.010+0000 7fa3e3234740 1 --2- 172.21.15.23:0/1478997696 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55fa000420d0 0x55fa000444c0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.010+0000 7fa3e3234740 1 --2- 172.21.15.23:0/1478997696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa00031e00 0x55fa0002f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.010+0000 7fa3e3234740 1 --2- 172.21.15.23:0/1478997696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00029030 0x55fa0002f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.010+0000 7fa3c1ffb640 0 client.16050 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:30:49.022 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.010+0000 7fa3e3234740 1 --2- 172.21.15.23:0/1478997696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00028710 0x55fa0002ee20 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.010+0000 7fa3e3234740 1 -- 172.21.15.23:0/1478997696 >> 172.21.15.23:0/1478997696 conn(0x55fa00023030 msgr2=0x55fa00024d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:49.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.010+0000 7fa3e3234740 1 -- 172.21.15.23:0/1478997696 shutdown_connections 2024-03-28T05:30:49.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.010+0000 7fa3e3234740 1 -- 172.21.15.23:0/1478997696 wait complete. 2024-03-28T05:30:49.023 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.010+0000 7fa3e3234740 20 client.16050 trim_cache size 0 max 16384 2024-03-28T05:30:49.032 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.025+0000 7fa3e3234740 1 Processor -- start 2024-03-28T05:30:49.032 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.025+0000 7fa3e3234740 1 -- start start 2024-03-28T05:30:49.032 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.025+0000 7fa3e3234740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00028710 0x55fa00031550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:49.032 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.025+0000 7fa3e3234740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa00029030 0x55fa0003d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:49.032 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.025+0000 7fa3e3234740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00031e00 0x55fa0003d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:49.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.025+0000 7fa3e3234740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d81aa8b0 con 0x55fa00028710 2024-03-28T05:30:49.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.025+0000 7fa3e3234740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d81ace50 con 0x55fa00029030 2024-03-28T05:30:49.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.025+0000 7fa3e3234740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d81aaeb0 con 0x55fa00031e00 2024-03-28T05:30:49.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.025+0000 7fa3de3e7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00028710 0x55fa00031550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:49.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.025+0000 7fa3de3e7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00028710 0x55fa00031550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47968/0 (socket says 172.21.15.23:47968) 2024-03-28T05:30:49.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.025+0000 7fa3de3e7640 1 -- 172.21.15.23:0/167847053 learned_addr learned my addr 172.21.15.23:0/167847053 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:30:49.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.026+0000 7fa3debe8640 1 --2- 172.21.15.23:0/167847053 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00031e00 0x55fa0003d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:49.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.026+0000 7fa3ddbe6640 1 --2- 172.21.15.23:0/167847053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa00029030 0x55fa0003d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:49.033 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.026+0000 7fa3de3e7640 1 -- 172.21.15.23:0/167847053 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00031e00 msgr2=0x55fa0003d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:49.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.026+0000 7fa3de3e7640 1 --2- 172.21.15.23:0/167847053 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00031e00 0x55fa0003d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.026+0000 7fa3de3e7640 1 -- 172.21.15.23:0/167847053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa00029030 msgr2=0x55fa0003d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:49.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.026+0000 7fa3de3e7640 1 --2- 172.21.15.23:0/167847053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa00029030 0x55fa0003d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.026+0000 7fa3de3e7640 1 -- 172.21.15.23:0/167847053 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa00028e60 con 0x55fa00028710 2024-03-28T05:30:49.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.026+0000 7fa3ddbe6640 1 --2- 172.21.15.23:0/167847053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa00029030 0x55fa0003d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:30:49.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.026+0000 7fa3de3e7640 1 --2- 172.21.15.23:0/167847053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00028710 0x55fa00031550 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4000c00 tx=0x7fa3d4005870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a2b7f1b729e7f01 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:49.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.026+0000 7fa3c3fff640 1 -- 172.21.15.23:0/167847053 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d400efe0 con 0x55fa00028710 2024-03-28T05:30:49.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.026+0000 7fa3c3fff640 1 -- 172.21.15.23:0/167847053 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3d401b910 con 0x55fa00028710 2024-03-28T05:30:49.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.026+0000 7fa3c3fff640 1 -- 172.21.15.23:0/167847053 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d400eb60 con 0x55fa00028710 2024-03-28T05:30:49.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.027+0000 7fa3e3234740 1 -- 172.21.15.23:0/167847053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00028710 msgr2=0x55fa00031550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:49.034 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.027+0000 7fa3e3234740 1 --2- 172.21.15.23:0/167847053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00028710 0x55fa00031550 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4000c00 tx=0x7fa3d4005870 comp rx=0 tx=0).stop 2024-03-28T05:30:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.027+0000 7fa3e3234740 1 -- 172.21.15.23:0/167847053 shutdown_connections 2024-03-28T05:30:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.027+0000 7fa3e3234740 1 --2- 172.21.15.23:0/167847053 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00031e00 0x55fa0003d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.027+0000 7fa3e3234740 1 --2- 172.21.15.23:0/167847053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa00029030 0x55fa0003d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.027+0000 7fa3e3234740 1 --2- 172.21.15.23:0/167847053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00028710 0x55fa00031550 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.027+0000 7fa3e3234740 1 -- 172.21.15.23:0/167847053 >> 172.21.15.23:0/167847053 conn(0x55fa00023030 msgr2=0x55fa00023410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.027+0000 7fa3e3234740 1 -- 172.21.15.23:0/167847053 shutdown_connections 2024-03-28T05:30:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.027+0000 7fa3e3234740 1 -- 172.21.15.23:0/167847053 wait complete. 2024-03-28T05:30:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.027+0000 7fa3e3234740 1 -- start start 2024-03-28T05:30:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.027+0000 7fa3e3234740 1 Processor -- start 2024-03-28T05:30:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.027+0000 7fa3e3234740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00028710 0x55fa0002b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.027+0000 7fa3e3234740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00029030 0x55fa0002bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.027+0000 7fa3e3234740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa00031e00 0x55fa0002c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.027+0000 7fa3e3234740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d81ab4d0 con 0x55fa00028710 2024-03-28T05:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.027+0000 7fa3e3234740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d81ac7f0 con 0x55fa00031e00 2024-03-28T05:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.027+0000 7fa3e3234740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d81abb30 con 0x55fa00029030 2024-03-28T05:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.027+0000 7fa3e3234740 10 client.0 _ll_register_callbacks cb 0x7fa3d400fca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.028+0000 7fa3de3e7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00028710 0x55fa0002b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.028+0000 7fa3de3e7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00028710 0x55fa0002b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47976/0 (socket says 172.21.15.23:47976) 2024-03-28T05:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.028+0000 7fa3de3e7640 1 -- 172.21.15.23:0/3830091233 learned_addr learned my addr 172.21.15.23:0/3830091233 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.028+0000 7fa3ddbe6640 1 --2- 172.21.15.23:0/3830091233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00029030 0x55fa0002bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.028+0000 7fa3debe8640 1 --2- 172.21.15.23:0/3830091233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa00031e00 0x55fa0002c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.028+0000 7fa3de3e7640 1 -- 172.21.15.23:0/3830091233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00029030 msgr2=0x55fa0002bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.028+0000 7fa3de3e7640 1 --2- 172.21.15.23:0/3830091233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00029030 0x55fa0002bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.028+0000 7fa3de3e7640 1 -- 172.21.15.23:0/3830091233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa00031e00 msgr2=0x55fa0002c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.028+0000 7fa3de3e7640 1 --2- 172.21.15.23:0/3830091233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa00031e00 0x55fa0002c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.028+0000 7fa3de3e7640 1 -- 172.21.15.23:0/3830091233 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fa0009c980 con 0x55fa00028710 2024-03-28T05:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.028+0000 7fa3debe8640 1 --2- 172.21.15.23:0/3830091233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa00031e00 0x55fa0002c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.028+0000 7fa3de3e7640 1 --2- 172.21.15.23:0/3830091233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00028710 0x55fa0002b650 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fa3d401b440 tx=0x7fa3d4005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82bf4d462fd979e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.028+0000 7fa3c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.028+0000 7fa3c1ffb640 1 -- 172.21.15.23:0/3830091233 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d401bcd0 con 0x55fa00028710 2024-03-28T05:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.028+0000 7fa3c1ffb640 1 -- 172.21.15.23:0/3830091233 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fa3d400e8a0 con 0x55fa00028710 2024-03-28T05:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.028+0000 7fa3e3234740 1 -- 172.21.15.23:0/3830091233 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa3d401b910 con 0x55fa00028710 2024-03-28T05:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.028+0000 7fa3c1ffb640 1 -- 172.21.15.23:0/3830091233 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3d4047050 con 0x55fa00028710 2024-03-28T05:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.028+0000 7fa3e3234740 10 client.16056 fetch_fsmap learned FSMap version 20 2024-03-28T05:30:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.028+0000 7fa3e3234740 1 -- 172.21.15.23:0/3830091233 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55fa0009dba0 con 0x55fa00028710 2024-03-28T05:30:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.029+0000 7fa3c1ffb640 1 -- 172.21.15.23:0/3830091233 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3d401b910 con 0x55fa00028710 2024-03-28T05:30:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.029+0000 7fa3c1ffb640 1 -- 172.21.15.23:0/3830091233 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fa3d40098c0 con 0x55fa00028710 2024-03-28T05:30:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.029+0000 7fa3e3234740 10 client.16056 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:30:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.029+0000 7fa3e3234740 10 client.16056 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:30:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.029+0000 7fa3e3234740 20 client.16056 populate_metadata read hostname 'smithi023' 2024-03-28T05:30:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.029+0000 7fa3e3234740 1 --2- 172.21.15.23:0/3830091233 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55fa0009e290 0x55fa000a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:30:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.029+0000 7fa3e3234740 4 client.16056 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:30:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.029+0000 7fa3e3234740 1 -- 172.21.15.23:0/3830091233 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f9fff943a0 con 0x55fa0009e290 2024-03-28T05:30:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.029+0000 7fa3ddbe6640 1 --2- 172.21.15.23:0/3830091233 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55fa0009e290 0x55fa000a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:30:49.039 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.029+0000 7fa3ddbe6640 1 --2- 172.21.15.23:0/3830091233 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55fa0009e290 0x55fa000a06b0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97f5ef27467abf56 server_cookie=7c0c4dfa27778166 in_seq=0 out_seq=0 2024-03-28T05:30:49.039 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.029+0000 7fa3c1ffb640 10 client.16056 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:30:49.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:48 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:30:49.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:48 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:30:49.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:48 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:30:49.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:48 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:30:49.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:48 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:30:49.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:48 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:30:49.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:48 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:30:49.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:48 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:30:49.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:48 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:30:49.493 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.492+0000 7fa3c1ffb640 1 -- 172.21.15.23:0/3830091233 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f9fff943a0 con 0x55fa0009e290 2024-03-28T05:30:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.492+0000 7fa3c1ffb640 10 client.16056 handle_command_reply: tid=0 2024-03-28T05:30:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.492+0000 7fa3e3234740 1 client.16056 shutdown 2024-03-28T05:30:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.492+0000 7fa3e3234740 1 -- 172.21.15.23:0/3830091233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00028710 msgr2=0x55fa0002b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.492+0000 7fa3e3234740 1 --2- 172.21.15.23:0/3830091233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00028710 0x55fa0002b650 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fa3d401b440 tx=0x7fa3d4005970 comp rx=0 tx=0).stop 2024-03-28T05:30:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.492+0000 7fa3e3234740 1 -- 172.21.15.23:0/3830091233 shutdown_connections 2024-03-28T05:30:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.492+0000 7fa3e3234740 1 --2- 172.21.15.23:0/3830091233 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55fa0009e290 0x55fa000a06b0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.492+0000 7fa3e3234740 1 --2- 172.21.15.23:0/3830091233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa00031e00 0x55fa0002c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.492+0000 7fa3e3234740 1 --2- 172.21.15.23:0/3830091233 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fa00029030 0x55fa0002bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.492+0000 7fa3e3234740 1 --2- 172.21.15.23:0/3830091233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fa00028710 0x55fa0002b650 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.492+0000 7fa3e3234740 1 -- 172.21.15.23:0/3830091233 >> 172.21.15.23:0/3830091233 conn(0x55fa00023030 msgr2=0x55fa0002ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.492+0000 7fa3c1ffb640 0 client.16056 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:30:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.492+0000 7fa3e3234740 1 -- 172.21.15.23:0/3830091233 shutdown_connections 2024-03-28T05:30:49.495 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.492+0000 7fa3e3234740 1 -- 172.21.15.23:0/3830091233 wait complete. 2024-03-28T05:30:49.495 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.493+0000 7fa3e3234740 20 client.16056 trim_cache size 0 max 16384 2024-03-28T05:30:49.495 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:30:49.495 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:30:49.495 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:30:49.495 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:30:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.493+0000 7fa3e064b640 1 -- 172.21.15.23:0/3936262886 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fa3c40a2a90 msgr2=0x7fa3c40a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.493+0000 7fa3e064b640 1 --2- 172.21.15.23:0/3936262886 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fa3c40a2a90 0x7fa3c40a4f50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa3d000fc90 tx=0x7fa3d001e000 comp rx=0 tx=0).stop 2024-03-28T05:30:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.493+0000 7fa3e064b640 1 -- 172.21.15.23:0/3936262886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d80696c0 msgr2=0x7fa3d819be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:30:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.493+0000 7fa3e064b640 1 --2- 172.21.15.23:0/3936262886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d80696c0 0x7fa3d819be30 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009230 tx=0x7fa3c8004270 comp rx=0 tx=0).stop 2024-03-28T05:30:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.493+0000 7fa3e064b640 1 -- 172.21.15.23:0/3936262886 shutdown_connections 2024-03-28T05:30:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.493+0000 7fa3e064b640 1 --2- 172.21.15.23:0/3936262886 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fa3c40a2a90 0x7fa3c40a4f50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.493+0000 7fa3e064b640 1 --2- 172.21.15.23:0/3936262886 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3d8107770 0x7fa3d8195af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.493+0000 7fa3e064b640 1 --2- 172.21.15.23:0/3936262886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d80696c0 0x7fa3d819be30 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.493+0000 7fa3e064b640 1 --2- 172.21.15.23:0/3936262886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa3d8068d20 0x7fa3d819b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:30:49.496 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.493+0000 7fa3e064b640 1 -- 172.21.15.23:0/3936262886 >> 172.21.15.23:0/3936262886 conn(0x7fa3d80773f0 msgr2=0x7fa3d80fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:30:49.497 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.493+0000 7fa3e064b640 1 -- 172.21.15.23:0/3936262886 shutdown_connections 2024-03-28T05:30:49.497 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:30:49.494+0000 7fa3e064b640 1 -- 172.21.15.23:0/3936262886 wait complete. 2024-03-28T05:30:50.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:49 smithi113 ceph-mon[37164]: pgmap v222: 129 pgs: 129 active+clean; 261 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 401 op/s 2024-03-28T05:30:50.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:49 smithi023 ceph-mon[31954]: pgmap v222: 129 pgs: 129 active+clean; 261 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 401 op/s 2024-03-28T05:30:50.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:49 smithi098 ceph-mon[36087]: pgmap v222: 129 pgs: 129 active+clean; 261 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 401 op/s 2024-03-28T05:30:52.183 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:51 smithi098 ceph-mon[36087]: pgmap v223: 129 pgs: 129 active+clean; 250 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 591 op/s 2024-03-28T05:30:52.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:51 smithi113 ceph-mon[37164]: pgmap v223: 129 pgs: 129 active+clean; 250 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 591 op/s 2024-03-28T05:30:52.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:51 smithi023 ceph-mon[31954]: pgmap v223: 129 pgs: 129 active+clean; 250 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 591 op/s 2024-03-28T05:30:54.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:53 smithi113 ceph-mon[37164]: pgmap v224: 129 pgs: 129 active+clean; 234 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 651 op/s 2024-03-28T05:30:54.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:53 smithi023 ceph-mon[31954]: pgmap v224: 129 pgs: 129 active+clean; 234 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 651 op/s 2024-03-28T05:30:54.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:53 smithi098 ceph-mon[36087]: pgmap v224: 129 pgs: 129 active+clean; 234 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 651 op/s 2024-03-28T05:30:56.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:55 smithi113 ceph-mon[37164]: pgmap v225: 129 pgs: 129 active+clean; 234 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 481 op/s 2024-03-28T05:30:56.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:55 smithi023 ceph-mon[31954]: pgmap v225: 129 pgs: 129 active+clean; 234 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 481 op/s 2024-03-28T05:30:56.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:55 smithi098 ceph-mon[36087]: pgmap v225: 129 pgs: 129 active+clean; 234 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.2 MiB/s wr, 481 op/s 2024-03-28T05:30:58.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:57 smithi113 ceph-mon[37164]: pgmap v226: 129 pgs: 129 active+clean; 240 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 696 op/s 2024-03-28T05:30:58.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:57 smithi023 ceph-mon[31954]: pgmap v226: 129 pgs: 129 active+clean; 240 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 696 op/s 2024-03-28T05:30:58.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:57 smithi098 ceph-mon[36087]: pgmap v226: 129 pgs: 129 active+clean; 240 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 696 op/s 2024-03-28T05:31:00.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:30:59 smithi113 ceph-mon[37164]: pgmap v227: 129 pgs: 129 active+clean; 240 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 1.4 MiB/s wr, 558 op/s 2024-03-28T05:31:00.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:30:59 smithi023 ceph-mon[31954]: pgmap v227: 129 pgs: 129 active+clean; 240 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 1.4 MiB/s wr, 558 op/s 2024-03-28T05:31:00.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:30:59 smithi098 ceph-mon[36087]: pgmap v227: 129 pgs: 129 active+clean; 240 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 1.4 MiB/s wr, 558 op/s 2024-03-28T05:31:02.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:01 smithi113 ceph-mon[37164]: pgmap v228: 129 pgs: 129 active+clean; 243 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 676 op/s 2024-03-28T05:31:02.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:01 smithi023 ceph-mon[31954]: pgmap v228: 129 pgs: 129 active+clean; 243 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 676 op/s 2024-03-28T05:31:02.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:01 smithi098 ceph-mon[36087]: pgmap v228: 129 pgs: 129 active+clean; 243 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 676 op/s 2024-03-28T05:31:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.592+0000 7fb5b764d640 1 -- 172.21.15.23:0/42222996 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5b0100400 msgr2=0x7fb5b01007e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:31:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.592+0000 7fb5b764d640 1 --2- 172.21.15.23:0/42222996 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5b0100400 0x7fb5b01007e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0009900 tx=0x7fb5a002bc30 comp rx=0 tx=0).stop 2024-03-28T05:31:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.592+0000 7fb5b764d640 1 -- 172.21.15.23:0/42222996 shutdown_connections 2024-03-28T05:31:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.592+0000 7fb5b764d640 1 --2- 172.21.15.23:0/42222996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b010dfb0 0x7fb5b01103a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.592+0000 7fb5b764d640 1 --2- 172.21.15.23:0/42222996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5b0100db0 0x7fb5b010da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.592+0000 7fb5b764d640 1 --2- 172.21.15.23:0/42222996 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5b0100400 0x7fb5b01007e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.592+0000 7fb5b764d640 1 -- 172.21.15.23:0/42222996 >> 172.21.15.23:0/42222996 conn(0x7fb5b00776d0 msgr2=0x7fb5b0077ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:31:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.592+0000 7fb5b764d640 1 -- 172.21.15.23:0/42222996 shutdown_connections 2024-03-28T05:31:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.592+0000 7fb5b764d640 1 -- 172.21.15.23:0/42222996 wait complete. 2024-03-28T05:31:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.592+0000 7fb5b764d640 1 Processor -- start 2024-03-28T05:31:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.592+0000 7fb5b764d640 1 -- start start 2024-03-28T05:31:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.593+0000 7fb5b764d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5b0100400 0x7fb5b019a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.593+0000 7fb5b764d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b0100db0 0x7fb5b019a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.593+0000 7fb5b764d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5b010dfb0 0x7fb5b019e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.593+0000 7fb5b764d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b0113c80 con 0x7fb5b0100400 2024-03-28T05:31:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.593+0000 7fb5b764d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b0113b00 con 0x7fb5b0100db0 2024-03-28T05:31:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.593+0000 7fb5b764d640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b0113e00 con 0x7fb5b010dfb0 2024-03-28T05:31:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.593+0000 7fb5b53e9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5b0100400 0x7fb5b019a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.593+0000 7fb5b53e9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5b0100400 0x7fb5b019a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37778/0 (socket says 172.21.15.23:37778) 2024-03-28T05:31:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.593+0000 7fb5b53e9640 1 -- 172.21.15.23:0/4119185225 learned_addr learned my addr 172.21.15.23:0/4119185225 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:31:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.593+0000 7fb5b5bea640 1 --2- 172.21.15.23:0/4119185225 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5b010dfb0 0x7fb5b019e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.593+0000 7fb5b53e9640 1 -- 172.21.15.23:0/4119185225 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5b010dfb0 msgr2=0x7fb5b019e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:31:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.593+0000 7fb5b53e9640 1 --2- 172.21.15.23:0/4119185225 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5b010dfb0 0x7fb5b019e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.593+0000 7fb5b53e9640 1 -- 172.21.15.23:0/4119185225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b0100db0 msgr2=0x7fb5b019a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:31:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.593+0000 7fb5b53e9640 1 --2- 172.21.15.23:0/4119185225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b0100db0 0x7fb5b019a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.593+0000 7fb5b53e9640 1 -- 172.21.15.23:0/4119185225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b019f040 con 0x7fb5b0100400 2024-03-28T05:31:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.593+0000 7fb5b53e9640 1 --2- 172.21.15.23:0/4119185225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5b0100400 0x7fb5b019a000 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0005940 tx=0x7fb5a002cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:31:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.593+0000 7fb5a67fc640 1 -- 172.21.15.23:0/4119185225 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0041070 con 0x7fb5b0100400 2024-03-28T05:31:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.593+0000 7fb5a67fc640 1 -- 172.21.15.23:0/4119185225 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5a0004450 con 0x7fb5b0100400 2024-03-28T05:31:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.593+0000 7fb5b764d640 1 -- 172.21.15.23:0/4119185225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5b019f2d0 con 0x7fb5b0100400 2024-03-28T05:31:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.593+0000 7fb5a67fc640 1 -- 172.21.15.23:0/4119185225 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0003df0 con 0x7fb5b0100400 2024-03-28T05:31:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.593+0000 7fb5b764d640 1 -- 172.21.15.23:0/4119185225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b01a6b70 con 0x7fb5b0100400 2024-03-28T05:31:02.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.596+0000 7fb5a67fc640 1 -- 172.21.15.23:0/4119185225 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fb5a0044070 con 0x7fb5b0100400 2024-03-28T05:31:02.605 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.596+0000 7fb5ba231740 1 Processor -- start 2024-03-28T05:31:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.596+0000 7fb5ba231740 1 -- start start 2024-03-28T05:31:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.597+0000 7fb5a67fc640 1 --2- 172.21.15.23:0/4119185225 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fb59c0a2a10 0x7fb59c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.597+0000 7fb5ba231740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559798292710 0x559798292af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.597+0000 7fb5ba231740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559798293030 0x55979829b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.597+0000 7fb5ba231740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55979829be00 0x55979829e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.597+0000 7fb5ba231740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597981f6b10 con 0x55979829be00 2024-03-28T05:31:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.597+0000 7fb5ba231740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55979819f900 con 0x559798292710 2024-03-28T05:31:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.597+0000 7fb5ba231740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597981f2bb0 con 0x559798293030 2024-03-28T05:31:02.606 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.597+0000 7fb5b5bea640 1 --2- 172.21.15.23:0/4119185225 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fb59c0a2a10 0x7fb59c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.597+0000 7fb5a67fc640 1 -- 172.21.15.23:0/4119185225 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fb5a00f0c50 con 0x7fb5b0100400 2024-03-28T05:31:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.597+0000 7fb5b4be8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559798293030 0x55979829b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.597+0000 7fb5b4be8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559798293030 0x55979829b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:58258/0 (socket says 172.21.15.23:58258) 2024-03-28T05:31:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.597+0000 7fb5b4be8640 1 -- 172.21.15.23:0/4172716702 learned_addr learned my addr 172.21.15.23:0/4172716702 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:31:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.597+0000 7fb5b5bea640 1 --2- 172.21.15.23:0/4172716702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55979829be00 0x55979829e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.597+0000 7fb5b53e9640 1 --2- 172.21.15.23:0/4172716702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559798292710 0x559798292af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.597+0000 7fb5b4be8640 1 -- 172.21.15.23:0/4172716702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559798292710 msgr2=0x559798292af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:31:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.597+0000 7fb5b4be8640 1 --2- 172.21.15.23:0/4172716702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559798292710 0x559798292af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.597+0000 7fb5b4be8640 1 -- 172.21.15.23:0/4172716702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55979829be00 msgr2=0x55979829e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:31:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.597+0000 7fb5b4be8640 1 --2- 172.21.15.23:0/4172716702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55979829be00 0x55979829e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.597+0000 7fb5b4be8640 1 -- 172.21.15.23:0/4172716702 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559797f0b210 con 0x559798293030 2024-03-28T05:31:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.597+0000 7fb5b5bea640 1 --2- 172.21.15.23:0/4119185225 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fb59c0a2a10 0x7fb59c0a4ed0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0069ce0 tx=0x7fb5a8009e90 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:31:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.597+0000 7fb5b4be8640 1 --2- 172.21.15.23:0/4172716702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559798293030 0x55979829b8c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac009b20 tx=0x7fb5ac00e350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4073609f2bcc93ab server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:31:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.598+0000 7fb59b7fe640 1 -- 172.21.15.23:0/4172716702 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac00ed80 con 0x559798293030 2024-03-28T05:31:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.598+0000 7fb59b7fe640 1 -- 172.21.15.23:0/4172716702 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5ac010910 con 0x559798293030 2024-03-28T05:31:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.598+0000 7fb59b7fe640 1 -- 172.21.15.23:0/4172716702 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac0179b0 con 0x559798293030 2024-03-28T05:31:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.598+0000 7fb5ba231740 1 -- 172.21.15.23:0/4172716702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559798293030 msgr2=0x55979829b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:31:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.598+0000 7fb5ba231740 1 --2- 172.21.15.23:0/4172716702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559798293030 0x55979829b8c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac009b20 tx=0x7fb5ac00e350 comp rx=0 tx=0).stop 2024-03-28T05:31:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.598+0000 7fb5ba231740 1 -- 172.21.15.23:0/4172716702 shutdown_connections 2024-03-28T05:31:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.598+0000 7fb5ba231740 1 --2- 172.21.15.23:0/4172716702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55979829be00 0x55979829e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.598+0000 7fb5ba231740 1 --2- 172.21.15.23:0/4172716702 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559798293030 0x55979829b8c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.598+0000 7fb5ba231740 1 --2- 172.21.15.23:0/4172716702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559798292710 0x559798292af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.598+0000 7fb5ba231740 1 -- 172.21.15.23:0/4172716702 >> 172.21.15.23:0/4172716702 conn(0x55979828d030 msgr2=0x55979828d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:31:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.598+0000 7fb5ba231740 1 -- 172.21.15.23:0/4172716702 shutdown_connections 2024-03-28T05:31:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.598+0000 7fb5ba231740 1 -- 172.21.15.23:0/4172716702 wait complete. 2024-03-28T05:31:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.599+0000 7fb5ba231740 1 -- start start 2024-03-28T05:31:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.599+0000 7fb5ba231740 1 Processor -- start 2024-03-28T05:31:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.599+0000 7fb5ba231740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559798292710 0x559798298e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.599+0000 7fb5ba231740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559798293030 0x5597982993d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.599+0000 7fb5ba231740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55979829be00 0x559798299910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.599+0000 7fb5ba231740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55979829ea30 con 0x559798292710 2024-03-28T05:31:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.599+0000 7fb5ba231740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597981b2d50 con 0x55979829be00 2024-03-28T05:31:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.599+0000 7fb5ba231740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55979829ebb0 con 0x559798293030 2024-03-28T05:31:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.599+0000 7fb5ba231740 10 client.0 _ll_register_callbacks cb 0x5597981b2720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:31:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.599+0000 7fb5b53e9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559798292710 0x559798298e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.599+0000 7fb5b53e9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559798292710 0x559798298e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37806/0 (socket says 172.21.15.23:37806) 2024-03-28T05:31:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.599+0000 7fb5b4be8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559798293030 0x5597982993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.599+0000 7fb5b53e9640 1 -- 172.21.15.23:0/2192334193 learned_addr learned my addr 172.21.15.23:0/2192334193 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:31:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.600+0000 7fb5b5bea640 1 --2- 172.21.15.23:0/2192334193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55979829be00 0x559798299910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.600+0000 7fb5b4be8640 1 -- 172.21.15.23:0/2192334193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55979829be00 msgr2=0x559798299910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:31:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.600+0000 7fb5b4be8640 1 --2- 172.21.15.23:0/2192334193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55979829be00 0x559798299910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.600+0000 7fb5b4be8640 1 -- 172.21.15.23:0/2192334193 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559798292710 msgr2=0x559798298e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:31:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.600+0000 7fb5b4be8640 1 --2- 172.21.15.23:0/2192334193 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559798292710 0x559798298e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.600+0000 7fb5b4be8640 1 -- 172.21.15.23:0/2192334193 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559797fd7620 con 0x559798293030 2024-03-28T05:31:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.600+0000 7fb5b53e9640 1 --2- 172.21.15.23:0/2192334193 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559798292710 0x559798298e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:31:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.600+0000 7fb5b4be8640 1 --2- 172.21.15.23:0/2192334193 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559798293030 0x5597982993d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac000c00 tx=0x7fb5ac00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=884f48787902f3b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:31:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.600+0000 7fb598ff9640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-28T05:31:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.600+0000 7fb598ff9640 1 -- 172.21.15.23:0/2192334193 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac026030 con 0x559798293030 2024-03-28T05:31:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.600+0000 7fb5ba231740 1 -- 172.21.15.23:0/2192334193 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb5ac010910 con 0x559798293030 2024-03-28T05:31:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.600+0000 7fb598ff9640 1 -- 172.21.15.23:0/2192334193 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fb5ac01d070 con 0x559798293030 2024-03-28T05:31:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.600+0000 7fb598ff9640 1 -- 172.21.15.23:0/2192334193 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5ac030050 con 0x559798293030 2024-03-28T05:31:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.600+0000 7fb5ba231740 10 client.25894 fetch_fsmap learned FSMap version 20 2024-03-28T05:31:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.600+0000 7fb5ba231740 1 -- 172.21.15.23:0/2192334193 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5597982aacf0 con 0x559798293030 2024-03-28T05:31:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.601+0000 7fb598ff9640 1 -- 172.21.15.23:0/2192334193 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5ac010910 con 0x559798293030 2024-03-28T05:31:02.611 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.601+0000 7fb598ff9640 1 -- 172.21.15.23:0/2192334193 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fb5ac022460 con 0x559798293030 2024-03-28T05:31:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.601+0000 7fb5ba231740 10 client.25894 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:31:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.601+0000 7fb5ba231740 10 client.25894 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:31:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.601+0000 7fb5ba231740 20 client.25894 populate_metadata read hostname 'smithi023' 2024-03-28T05:31:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.601+0000 7fb5ba231740 1 --2- 172.21.15.23:0/2192334193 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5597982ac100 0x5597982ae4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.601+0000 7fb5ba231740 4 client.25894 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:31:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.601+0000 7fb5ba231740 1 -- 172.21.15.23:0/2192334193 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559798204050 con 0x5597982ac100 2024-03-28T05:31:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.601+0000 7fb5b53e9640 1 --2- 172.21.15.23:0/2192334193 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5597982ac100 0x5597982ae4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.601+0000 7fb5b53e9640 1 --2- 172.21.15.23:0/2192334193 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5597982ac100 0x5597982ae4f0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1501fbd81672aba3 server_cookie=a86a5308973c13e6 in_seq=0 out_seq=0 2024-03-28T05:31:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.601+0000 7fb598ff9640 10 client.25894 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:31:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.602+0000 7fb598ff9640 1 -- 172.21.15.23:0/2192334193 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559798204050 con 0x5597982ac100 2024-03-28T05:31:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.602+0000 7fb598ff9640 10 client.25894 handle_command_reply: tid=0 2024-03-28T05:31:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.602+0000 7fb5ba231740 1 client.25894 shutdown 2024-03-28T05:31:02.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.602+0000 7fb5ba231740 1 -- 172.21.15.23:0/2192334193 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559798293030 msgr2=0x5597982993d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:31:02.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.602+0000 7fb5ba231740 1 --2- 172.21.15.23:0/2192334193 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559798293030 0x5597982993d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac000c00 tx=0x7fb5ac00ef10 comp rx=0 tx=0).stop 2024-03-28T05:31:02.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.602+0000 7fb5ba231740 1 -- 172.21.15.23:0/2192334193 shutdown_connections 2024-03-28T05:31:02.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.602+0000 7fb5ba231740 1 --2- 172.21.15.23:0/2192334193 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5597982ac100 0x5597982ae4f0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.602+0000 7fb5ba231740 1 --2- 172.21.15.23:0/2192334193 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55979829be00 0x559798299910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.602+0000 7fb5ba231740 1 --2- 172.21.15.23:0/2192334193 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559798293030 0x5597982993d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.602+0000 7fb598ff9640 0 client.25894 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:31:02.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.602+0000 7fb5ba231740 1 --2- 172.21.15.23:0/2192334193 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559798292710 0x559798298e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.613 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.603+0000 7fb5ba231740 1 -- 172.21.15.23:0/2192334193 >> 172.21.15.23:0/2192334193 conn(0x55979828d030 msgr2=0x55979828ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:31:02.614 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.603+0000 7fb5ba231740 1 -- 172.21.15.23:0/2192334193 shutdown_connections 2024-03-28T05:31:02.614 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.603+0000 7fb5ba231740 1 -- 172.21.15.23:0/2192334193 wait complete. 2024-03-28T05:31:02.614 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.603+0000 7fb5ba231740 20 client.25894 trim_cache size 0 max 16384 2024-03-28T05:31:02.622 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.617+0000 7fb5ba231740 1 Processor -- start 2024-03-28T05:31:02.622 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.617+0000 7fb5ba231740 1 -- start start 2024-03-28T05:31:02.622 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.617+0000 7fb5ba231740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559798292710 0x55979829b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:02.622 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.617+0000 7fb5ba231740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559798293030 0x55979828eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:02.622 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.617+0000 7fb5ba231740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55979829be00 0x5597982960f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:02.622 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.617+0000 7fb5ba231740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b01aedb0 con 0x559798293030 2024-03-28T05:31:02.622 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.617+0000 7fb5ba231740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b01b1350 con 0x55979829be00 2024-03-28T05:31:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.617+0000 7fb5ba231740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b01af3b0 con 0x559798292710 2024-03-28T05:31:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.617+0000 7fb5b4be8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559798293030 0x55979828eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.617+0000 7fb5b4be8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559798293030 0x55979828eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37808/0 (socket says 172.21.15.23:37808) 2024-03-28T05:31:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.617+0000 7fb5b4be8640 1 -- 172.21.15.23:0/846683758 learned_addr learned my addr 172.21.15.23:0/846683758 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:31:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.617+0000 7fb5b53e9640 1 --2- 172.21.15.23:0/846683758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559798292710 0x55979829b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.617+0000 7fb5b5bea640 1 --2- 172.21.15.23:0/846683758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55979829be00 0x5597982960f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.617+0000 7fb5b4be8640 1 -- 172.21.15.23:0/846683758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559798292710 msgr2=0x55979829b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:31:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.617+0000 7fb5b4be8640 1 --2- 172.21.15.23:0/846683758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559798292710 0x55979829b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.617+0000 7fb5b4be8640 1 -- 172.21.15.23:0/846683758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55979829be00 msgr2=0x5597982960f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:31:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.617+0000 7fb5b4be8640 1 --2- 172.21.15.23:0/846683758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55979829be00 0x5597982960f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.617+0000 7fb5b4be8640 1 -- 172.21.15.23:0/846683758 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559798292e60 con 0x559798293030 2024-03-28T05:31:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.617+0000 7fb5b5bea640 1 --2- 172.21.15.23:0/846683758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55979829be00 0x5597982960f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:31:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.617+0000 7fb5b4be8640 1 --2- 172.21.15.23:0/846683758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559798293030 0x55979828eb90 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0228b0 tx=0x7fb5ac0098c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63ab74348c082328 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:31:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.618+0000 7fb59affd640 1 -- 172.21.15.23:0/846683758 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac039070 con 0x559798293030 2024-03-28T05:31:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.618+0000 7fb59affd640 1 -- 172.21.15.23:0/846683758 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5ac022e50 con 0x559798293030 2024-03-28T05:31:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.618+0000 7fb59affd640 1 -- 172.21.15.23:0/846683758 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac026070 con 0x559798293030 2024-03-28T05:31:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.618+0000 7fb5ba231740 1 -- 172.21.15.23:0/846683758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559798293030 msgr2=0x55979828eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:31:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.618+0000 7fb5ba231740 1 --2- 172.21.15.23:0/846683758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559798293030 0x55979828eb90 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0228b0 tx=0x7fb5ac0098c0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.618+0000 7fb5ba231740 1 -- 172.21.15.23:0/846683758 shutdown_connections 2024-03-28T05:31:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.618+0000 7fb5ba231740 1 --2- 172.21.15.23:0/846683758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55979829be00 0x5597982960f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.618+0000 7fb5ba231740 1 --2- 172.21.15.23:0/846683758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559798293030 0x55979828eb90 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.618+0000 7fb5ba231740 1 --2- 172.21.15.23:0/846683758 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559798292710 0x55979829b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.618+0000 7fb5ba231740 1 -- 172.21.15.23:0/846683758 >> 172.21.15.23:0/846683758 conn(0x55979828d030 msgr2=0x55979828e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:31:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.618+0000 7fb5ba231740 1 -- 172.21.15.23:0/846683758 shutdown_connections 2024-03-28T05:31:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.618+0000 7fb5ba231740 1 -- 172.21.15.23:0/846683758 wait complete. 2024-03-28T05:31:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.618+0000 7fb5ba231740 1 -- start start 2024-03-28T05:31:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.619+0000 7fb5ba231740 1 Processor -- start 2024-03-28T05:31:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.619+0000 7fb5ba231740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559798292710 0x5597982aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.619+0000 7fb5ba231740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559798293030 0x5597982af530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.619+0000 7fb5ba231740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55979829be00 0x559798306630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.619+0000 7fb5ba231740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b01af9d0 con 0x559798293030 2024-03-28T05:31:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.619+0000 7fb5ba231740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b01b0cf0 con 0x559798292710 2024-03-28T05:31:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.619+0000 7fb5ba231740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b01b0030 con 0x55979829be00 2024-03-28T05:31:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.619+0000 7fb5ba231740 10 client.0 _ll_register_callbacks cb 0x7fb5ac009a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:31:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.619+0000 7fb5b4be8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559798293030 0x5597982af530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.619+0000 7fb5b4be8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559798293030 0x5597982af530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37824/0 (socket says 172.21.15.23:37824) 2024-03-28T05:31:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.619+0000 7fb5b4be8640 1 -- 172.21.15.23:0/2539221245 learned_addr learned my addr 172.21.15.23:0/2539221245 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:31:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.619+0000 7fb5b53e9640 1 --2- 172.21.15.23:0/2539221245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559798292710 0x5597982aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.619+0000 7fb5b5bea640 1 --2- 172.21.15.23:0/2539221245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55979829be00 0x559798306630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.619+0000 7fb5b4be8640 1 -- 172.21.15.23:0/2539221245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55979829be00 msgr2=0x559798306630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:31:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.619+0000 7fb5b4be8640 1 --2- 172.21.15.23:0/2539221245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55979829be00 0x559798306630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.619+0000 7fb5b4be8640 1 -- 172.21.15.23:0/2539221245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559798292710 msgr2=0x5597982aeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:31:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.619+0000 7fb5b4be8640 1 --2- 172.21.15.23:0/2539221245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559798292710 0x5597982aeff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.619+0000 7fb5b4be8640 1 -- 172.21.15.23:0/2539221245 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559798306b70 con 0x559798293030 2024-03-28T05:31:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.620+0000 7fb5b4be8640 1 --2- 172.21.15.23:0/2539221245 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559798293030 0x5597982af530 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac037040 tx=0x7fb5ac022d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e0fc82c43ff02da server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:31:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.620+0000 7fb598ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T05:31:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.620+0000 7fb598ff9640 1 -- 172.21.15.23:0/2539221245 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac039070 con 0x559798293030 2024-03-28T05:31:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.620+0000 7fb598ff9640 1 -- 172.21.15.23:0/2539221245 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fb5ac010840 con 0x559798293030 2024-03-28T05:31:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.620+0000 7fb5ba231740 1 -- 172.21.15.23:0/2539221245 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb5ac022e50 con 0x559798293030 2024-03-28T05:31:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.620+0000 7fb598ff9640 1 -- 172.21.15.23:0/2539221245 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5ac046050 con 0x559798293030 2024-03-28T05:31:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.620+0000 7fb5ba231740 10 client.16074 fetch_fsmap learned FSMap version 20 2024-03-28T05:31:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.620+0000 7fb5ba231740 1 -- 172.21.15.23:0/2539221245 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x559798307d90 con 0x559798293030 2024-03-28T05:31:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.620+0000 7fb598ff9640 1 -- 172.21.15.23:0/2539221245 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5ac022e50 con 0x559798293030 2024-03-28T05:31:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.620+0000 7fb598ff9640 1 -- 172.21.15.23:0/2539221245 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fb5ac017a90 con 0x559798293030 2024-03-28T05:31:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.620+0000 7fb5ba231740 10 client.16074 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:31:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.620+0000 7fb5ba231740 10 client.16074 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:31:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.620+0000 7fb5ba231740 20 client.16074 populate_metadata read hostname 'smithi023' 2024-03-28T05:31:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.620+0000 7fb5ba231740 1 --2- 172.21.15.23:0/2539221245 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5597983083f0 0x55979830a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.620+0000 7fb5ba231740 4 client.16074 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:31:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.620+0000 7fb5ba231740 1 -- 172.21.15.23:0/2539221245 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5597981fe3a0 con 0x5597983083f0 2024-03-28T05:31:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.621+0000 7fb5b53e9640 1 --2- 172.21.15.23:0/2539221245 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5597983083f0 0x55979830a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.621+0000 7fb5b53e9640 1 --2- 172.21.15.23:0/2539221245 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5597983083f0 0x55979830a810 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3df8c51036f5bd2 server_cookie=ff1db54a725f5edb in_seq=0 out_seq=0 2024-03-28T05:31:02.627 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.621+0000 7fb598ff9640 10 client.16074 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:31:02.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.799+0000 7fb598ff9640 1 -- 172.21.15.23:0/2539221245 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5597981fe3a0 con 0x5597983083f0 2024-03-28T05:31:02.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.799+0000 7fb598ff9640 10 client.16074 handle_command_reply: tid=0 2024-03-28T05:31:02.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.799+0000 7fb5ba231740 1 client.16074 shutdown 2024-03-28T05:31:02.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.799+0000 7fb5ba231740 1 -- 172.21.15.23:0/2539221245 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559798293030 msgr2=0x5597982af530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:31:02.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.799+0000 7fb5ba231740 1 --2- 172.21.15.23:0/2539221245 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559798293030 0x5597982af530 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac037040 tx=0x7fb5ac022d10 comp rx=0 tx=0).stop 2024-03-28T05:31:02.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.799+0000 7fb5ba231740 1 -- 172.21.15.23:0/2539221245 shutdown_connections 2024-03-28T05:31:02.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.799+0000 7fb5ba231740 1 --2- 172.21.15.23:0/2539221245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55979829be00 0x559798306630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.799+0000 7fb5ba231740 1 --2- 172.21.15.23:0/2539221245 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559798293030 0x5597982af530 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.799+0000 7fb5ba231740 1 --2- 172.21.15.23:0/2539221245 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x5597983083f0 0x55979830a810 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.799+0000 7fb5ba231740 1 --2- 172.21.15.23:0/2539221245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559798292710 0x5597982aeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.799+0000 7fb598ff9640 0 client.16074 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:31:02.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.799+0000 7fb5ba231740 1 -- 172.21.15.23:0/2539221245 >> 172.21.15.23:0/2539221245 conn(0x55979828d030 msgr2=0x5597982a7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:31:02.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.799+0000 7fb5ba231740 1 -- 172.21.15.23:0/2539221245 shutdown_connections 2024-03-28T05:31:02.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.799+0000 7fb5ba231740 1 -- 172.21.15.23:0/2539221245 wait complete. 2024-03-28T05:31:02.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.800+0000 7fb5ba231740 20 client.16074 trim_cache size 0 max 16384 2024-03-28T05:31:02.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:31:02.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:31:02.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:31:02.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:31:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.801+0000 7fb5b764d640 1 -- 172.21.15.23:0/4119185225 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fb59c0a2a10 msgr2=0x7fb59c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:31:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.801+0000 7fb5b764d640 1 --2- 172.21.15.23:0/4119185225 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fb59c0a2a10 0x7fb59c0a4ed0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0069ce0 tx=0x7fb5a8009e90 comp rx=0 tx=0).stop 2024-03-28T05:31:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.801+0000 7fb5b764d640 1 -- 172.21.15.23:0/4119185225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5b0100400 msgr2=0x7fb5b019a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:31:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.801+0000 7fb5b764d640 1 --2- 172.21.15.23:0/4119185225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5b0100400 0x7fb5b019a000 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0005940 tx=0x7fb5a002cb30 comp rx=0 tx=0).stop 2024-03-28T05:31:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.801+0000 7fb5b764d640 1 -- 172.21.15.23:0/4119185225 shutdown_connections 2024-03-28T05:31:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.801+0000 7fb5b764d640 1 --2- 172.21.15.23:0/4119185225 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5b010dfb0 0x7fb5b019e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.801+0000 7fb5b764d640 1 --2- 172.21.15.23:0/4119185225 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fb59c0a2a10 0x7fb59c0a4ed0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.801+0000 7fb5b764d640 1 --2- 172.21.15.23:0/4119185225 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b0100db0 0x7fb5b019a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.801+0000 7fb5b764d640 1 --2- 172.21.15.23:0/4119185225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5b0100400 0x7fb5b019a000 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.801+0000 7fb5b764d640 1 -- 172.21.15.23:0/4119185225 >> 172.21.15.23:0/4119185225 conn(0x7fb5b00776d0 msgr2=0x7fb5b010f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:31:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.801+0000 7fb5b764d640 1 -- 172.21.15.23:0/4119185225 shutdown_connections 2024-03-28T05:31:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:02.801+0000 7fb5b764d640 1 -- 172.21.15.23:0/4119185225 wait complete. 2024-03-28T05:31:04.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:03 smithi113 ceph-mon[37164]: pgmap v229: 129 pgs: 129 active+clean; 245 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 1.3 MiB/s wr, 607 op/s 2024-03-28T05:31:04.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:03 smithi023 ceph-mon[31954]: pgmap v229: 129 pgs: 129 active+clean; 245 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 1.3 MiB/s wr, 607 op/s 2024-03-28T05:31:04.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:03 smithi098 ceph-mon[36087]: pgmap v229: 129 pgs: 129 active+clean; 245 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 1.3 MiB/s wr, 607 op/s 2024-03-28T05:31:06.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:05 smithi113 ceph-mon[37164]: pgmap v230: 129 pgs: 129 active+clean; 245 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 1.1 MiB/s wr, 453 op/s 2024-03-28T05:31:06.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:05 smithi023 ceph-mon[31954]: pgmap v230: 129 pgs: 129 active+clean; 245 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 1.1 MiB/s wr, 453 op/s 2024-03-28T05:31:06.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:05 smithi098 ceph-mon[36087]: pgmap v230: 129 pgs: 129 active+clean; 245 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 1.1 MiB/s wr, 453 op/s 2024-03-28T05:31:07.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:06 smithi113 ceph-mon[37164]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T05:31:07.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:06 smithi023 ceph-mon[31954]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T05:31:07.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:06 smithi098 ceph-mon[36087]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T05:31:08.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:07 smithi113 ceph-mon[37164]: pgmap v232: 129 pgs: 129 active+clean; 237 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 576 op/s 2024-03-28T05:31:08.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:07 smithi023 ceph-mon[31954]: pgmap v232: 129 pgs: 129 active+clean; 237 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 576 op/s 2024-03-28T05:31:08.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:07 smithi098 ceph-mon[36087]: pgmap v232: 129 pgs: 129 active+clean; 237 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 576 op/s 2024-03-28T05:31:09.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:09 smithi023 ceph-mon[31954]: pgmap v233: 129 pgs: 129 active+clean; 237 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 576 op/s 2024-03-28T05:31:09.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:09 smithi098 ceph-mon[36087]: pgmap v233: 129 pgs: 129 active+clean; 237 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 576 op/s 2024-03-28T05:31:09.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:09 smithi113 ceph-mon[37164]: pgmap v233: 129 pgs: 129 active+clean; 237 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 576 op/s 2024-03-28T05:31:11.494 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:11 smithi113 ceph-mon[37164]: pgmap v234: 129 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 108 active+clean; 239 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 1.0 MiB/s wr, 485 op/s 2024-03-28T05:31:11.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:11 smithi023 ceph-mon[31954]: pgmap v234: 129 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 108 active+clean; 239 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 1.0 MiB/s wr, 485 op/s 2024-03-28T05:31:11.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:11 smithi098 ceph-mon[36087]: pgmap v234: 129 pgs: 8 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 108 active+clean; 239 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 1.0 MiB/s wr, 485 op/s 2024-03-28T05:31:13.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:13 smithi113 ceph-mon[37164]: pgmap v235: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 240 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 969 KiB/s wr, 406 op/s 2024-03-28T05:31:13.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:13 smithi023 ceph-mon[31954]: pgmap v235: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 240 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 969 KiB/s wr, 406 op/s 2024-03-28T05:31:13.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:13 smithi098 ceph-mon[36087]: pgmap v235: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 240 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 969 KiB/s wr, 406 op/s 2024-03-28T05:31:15.494 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:15 smithi113 ceph-mon[37164]: pgmap v236: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 240 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 969 KiB/s wr, 406 op/s 2024-03-28T05:31:15.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:15 smithi023 ceph-mon[31954]: pgmap v236: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 240 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 969 KiB/s wr, 406 op/s 2024-03-28T05:31:15.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:15 smithi098 ceph-mon[36087]: pgmap v236: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 240 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 969 KiB/s wr, 406 op/s 2024-03-28T05:31:15.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.939+0000 7f9120177640 1 -- 172.21.15.23:0/3712091536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9118112680 msgr2=0x7f9118114a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:31:15.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.939+0000 7f9120177640 1 --2- 172.21.15.23:0/3712091536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9118112680 0x7f9118114a70 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f9110009ca0 tx=0x7f911002c1b0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.940+0000 7f9120177640 1 -- 172.21.15.23:0/3712091536 shutdown_connections 2024-03-28T05:31:15.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.940+0000 7f9120177640 1 --2- 172.21.15.23:0/3712091536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9118112680 0x7f9118114a70 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.940+0000 7f9120177640 1 --2- 172.21.15.23:0/3712091536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9118074360 0x7f9118112140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.940+0000 7f9120177640 1 --2- 172.21.15.23:0/3712091536 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9118073a40 0x7f9118073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.940+0000 7f9120177640 1 -- 172.21.15.23:0/3712091536 >> 172.21.15.23:0/3712091536 conn(0x7f91180fd180 msgr2=0x7f91180ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:31:15.945 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.942+0000 7f9120177640 1 -- 172.21.15.23:0/3712091536 shutdown_connections 2024-03-28T05:31:15.945 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.945+0000 7f9120177640 1 -- 172.21.15.23:0/3712091536 wait complete. 2024-03-28T05:31:15.948 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.946+0000 7f9120177640 1 Processor -- start 2024-03-28T05:31:15.948 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.947+0000 7f9120177640 1 -- start start 2024-03-28T05:31:15.948 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.947+0000 7f9120177640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9118073a40 0x7f9118072dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:15.949 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.947+0000 7f9120177640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9118074360 0x7f9118073310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:15.949 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.947+0000 7f9120177640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9118112680 0x7f911806c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:15.949 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.947+0000 7f9120177640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91181180c0 con 0x7f9118112680 2024-03-28T05:31:15.949 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.947+0000 7f9120177640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9118117f40 con 0x7f9118074360 2024-03-28T05:31:15.949 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.947+0000 7f9120177640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9118118240 con 0x7f9118073a40 2024-03-28T05:31:15.949 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.947+0000 7f911df13640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9118073a40 0x7f9118072dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:15.949 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.947+0000 7f911df13640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9118073a40 0x7f9118072dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:48518/0 (socket says 172.21.15.23:48518) 2024-03-28T05:31:15.949 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.947+0000 7f911df13640 1 -- 172.21.15.23:0/3574072821 learned_addr learned my addr 172.21.15.23:0/3574072821 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:31:15.950 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.948+0000 7f911df13640 1 -- 172.21.15.23:0/3574072821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9118074360 msgr2=0x7f9118073310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:31:15.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.948+0000 7f911df13640 1 --2- 172.21.15.23:0/3574072821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9118074360 0x7f9118073310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.948+0000 7f911df13640 1 -- 172.21.15.23:0/3574072821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9118112680 msgr2=0x7f911806c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:31:15.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.948+0000 7f911df13640 1 --2- 172.21.15.23:0/3574072821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9118112680 0x7f911806c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.948+0000 7f911df13640 1 -- 172.21.15.23:0/3574072821 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911806c8f0 con 0x7f9118073a40 2024-03-28T05:31:15.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.949+0000 7f911df13640 1 --2- 172.21.15.23:0/3574072821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9118073a40 0x7f9118072dd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f910800b450 tx=0x7f910800b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:31:15.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.949+0000 7f910effd640 1 -- 172.21.15.23:0/3574072821 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9108015020 con 0x7f9118073a40 2024-03-28T05:31:15.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.949+0000 7f9120177640 1 -- 172.21.15.23:0/3574072821 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f911806cbe0 con 0x7f9118073a40 2024-03-28T05:31:15.951 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.949+0000 7f9120177640 1 -- 172.21.15.23:0/3574072821 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9118070b80 con 0x7f9118073a40 2024-03-28T05:31:15.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.951+0000 7f910effd640 1 -- 172.21.15.23:0/3574072821 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9108011d70 con 0x7f9118073a40 2024-03-28T05:31:15.952 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.951+0000 7f910effd640 1 -- 172.21.15.23:0/3574072821 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910800f340 con 0x7f9118073a40 2024-03-28T05:31:15.954 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.953+0000 7f9122d67740 1 Processor -- start 2024-03-28T05:31:15.954 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.953+0000 7f910effd640 1 -- 172.21.15.23:0/3574072821 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f9108018020 con 0x7f9118073a40 2024-03-28T05:31:15.955 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.954+0000 7f9122d67740 1 -- start start 2024-03-28T05:31:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.955+0000 7f9122d67740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e68349f710 0x55e68349faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.955+0000 7f9122d67740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6834a0030 0x55e6834a88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.955+0000 7f9122d67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6834a8e00 0x55e6834ab1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.955+0000 7f9122d67740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e683403b10 con 0x55e6834a0030 2024-03-28T05:31:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.955+0000 7f9122d67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6833ac900 con 0x55e6834a8e00 2024-03-28T05:31:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.955+0000 7f9122d67740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6833ffbb0 con 0x55e68349f710 2024-03-28T05:31:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.955+0000 7f910effd640 1 --2- 172.21.15.23:0/3574072821 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f91040a2c40 0x7f91040a5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.955+0000 7f911e714640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e68349f710 0x55e68349faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.955+0000 7f911e714640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e68349f710 0x55e68349faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:48522/0 (socket says 172.21.15.23:48522) 2024-03-28T05:31:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.955+0000 7f911e714640 1 -- 172.21.15.23:0/1876772274 learned_addr learned my addr 172.21.15.23:0/1876772274 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:31:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.955+0000 7f911e714640 1 --2- 172.21.15.23:0/3574072821 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f91040a2c40 0x7f91040a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.955+0000 7f911d712640 1 --2- 172.21.15.23:0/1876772274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6834a8e00 0x55e6834ab1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:15.959 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.955+0000 7f911df13640 1 --2- 172.21.15.23:0/1876772274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6834a0030 0x55e6834a88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:15.959 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.955+0000 7f910effd640 1 -- 172.21.15.23:0/3574072821 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9613+0+0 (secure 0 0 0) 0x7f91080ceba0 con 0x7f9118073a40 2024-03-28T05:31:15.959 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.955+0000 7f911d712640 1 -- 172.21.15.23:0/1876772274 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e68349f710 msgr2=0x55e68349faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:31:15.959 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.955+0000 7f911d712640 1 --2- 172.21.15.23:0/1876772274 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e68349f710 0x55e68349faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.959 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.955+0000 7f911d712640 1 -- 172.21.15.23:0/1876772274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6834a0030 msgr2=0x55e6834a88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:31:15.959 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.955+0000 7f911d712640 1 --2- 172.21.15.23:0/1876772274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6834a0030 0x55e6834a88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.959 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.955+0000 7f911d712640 1 -- 172.21.15.23:0/1876772274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e683118210 con 0x55e6834a8e00 2024-03-28T05:31:15.959 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.956+0000 7f911e714640 1 --2- 172.21.15.23:0/3574072821 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f91040a2c40 0x7f91040a5100 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9110037040 tx=0x7f91100095c0 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:31:15.959 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.956+0000 7f911d712640 1 --2- 172.21.15.23:0/1876772274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6834a8e00 0x55e6834ab1f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f911806d6a0 tx=0x7f911400ae40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6c00358953a09ba server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:31:15.959 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.956+0000 7f9103fff640 1 -- 172.21.15.23:0/1876772274 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911400cd80 con 0x55e6834a8e00 2024-03-28T05:31:15.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.956+0000 7f9103fff640 1 -- 172.21.15.23:0/1876772274 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f911400f930 con 0x55e6834a8e00 2024-03-28T05:31:15.960 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.956+0000 7f9103fff640 1 -- 172.21.15.23:0/1876772274 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114018a70 con 0x55e6834a8e00 2024-03-28T05:31:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.957+0000 7f9122d67740 1 -- 172.21.15.23:0/1876772274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6834a8e00 msgr2=0x55e6834ab1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:31:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.957+0000 7f9122d67740 1 --2- 172.21.15.23:0/1876772274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6834a8e00 0x55e6834ab1f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f911806d6a0 tx=0x7f911400ae40 comp rx=0 tx=0).stop 2024-03-28T05:31:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.957+0000 7f9122d67740 1 -- 172.21.15.23:0/1876772274 shutdown_connections 2024-03-28T05:31:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.957+0000 7f9122d67740 1 --2- 172.21.15.23:0/1876772274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6834a8e00 0x55e6834ab1f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.957+0000 7f9122d67740 1 --2- 172.21.15.23:0/1876772274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6834a0030 0x55e6834a88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.957+0000 7f9122d67740 1 --2- 172.21.15.23:0/1876772274 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e68349f710 0x55e68349faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.957+0000 7f9122d67740 1 -- 172.21.15.23:0/1876772274 >> 172.21.15.23:0/1876772274 conn(0x55e68349a030 msgr2=0x55e68349a410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:31:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.957+0000 7f9122d67740 1 -- 172.21.15.23:0/1876772274 shutdown_connections 2024-03-28T05:31:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.957+0000 7f9122d67740 1 -- 172.21.15.23:0/1876772274 wait complete. 2024-03-28T05:31:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.958+0000 7f9122d67740 1 -- start start 2024-03-28T05:31:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.958+0000 7f9122d67740 1 Processor -- start 2024-03-28T05:31:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.958+0000 7f9122d67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e68349f710 0x55e6834a5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.958+0000 7f9122d67740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6834a0030 0x55e6834a6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.958+0000 7f9122d67740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6834a8e00 0x55e6834a68a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.958+0000 7f9122d67740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6834aba30 con 0x55e6834a8e00 2024-03-28T05:31:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.958+0000 7f9122d67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6833bfd50 con 0x55e68349f710 2024-03-28T05:31:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.958+0000 7f9122d67740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6834abbb0 con 0x55e6834a0030 2024-03-28T05:31:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.958+0000 7f9122d67740 10 client.0 _ll_register_callbacks cb 0x55e6833bf720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:31:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.959+0000 7f911d712640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6834a0030 0x55e6834a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.959+0000 7f911df13640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e68349f710 0x55e6834a5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.959+0000 7f911d712640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6834a0030 0x55e6834a6360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:48534/0 (socket says 172.21.15.23:48534) 2024-03-28T05:31:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.959+0000 7f911d712640 1 -- 172.21.15.23:0/2730019810 learned_addr learned my addr 172.21.15.23:0/2730019810 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:31:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.959+0000 7f911e714640 1 --2- 172.21.15.23:0/2730019810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6834a8e00 0x55e6834a68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.959+0000 7f911d712640 1 -- 172.21.15.23:0/2730019810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e68349f710 msgr2=0x55e6834a5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:31:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.959+0000 7f911d712640 1 --2- 172.21.15.23:0/2730019810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e68349f710 0x55e6834a5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.959+0000 7f911d712640 1 -- 172.21.15.23:0/2730019810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6834a8e00 msgr2=0x55e6834a68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:31:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.959+0000 7f911d712640 1 --2- 172.21.15.23:0/2730019810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6834a8e00 0x55e6834a68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.959+0000 7f911d712640 1 -- 172.21.15.23:0/2730019810 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e6831e4620 con 0x55e6834a0030 2024-03-28T05:31:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.959+0000 7f911e714640 1 --2- 172.21.15.23:0/2730019810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6834a8e00 0x55e6834a68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:31:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.959+0000 7f91017fa640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-28T05:31:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.959+0000 7f911d712640 1 --2- 172.21.15.23:0/2730019810 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6834a0030 0x55e6834a6360 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9114005b50 tx=0x7f911400fd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=496124196b8ecdbd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:31:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.960+0000 7f91017fa640 1 -- 172.21.15.23:0/2730019810 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114005020 con 0x55e6834a0030 2024-03-28T05:31:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.960+0000 7f91017fa640 1 -- 172.21.15.23:0/2730019810 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9613+0+0 (secure 0 0 0) 0x7f911400f660 con 0x55e6834a0030 2024-03-28T05:31:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.960+0000 7f9122d67740 1 -- 172.21.15.23:0/2730019810 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f911400f930 con 0x55e6834a0030 2024-03-28T05:31:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.960+0000 7f91017fa640 1 -- 172.21.15.23:0/2730019810 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9114005020 con 0x55e6834a0030 2024-03-28T05:31:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.960+0000 7f9122d67740 10 client.25909 fetch_fsmap learned FSMap version 20 2024-03-28T05:31:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.960+0000 7f9122d67740 1 -- 172.21.15.23:0/2730019810 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55e6834b7ce0 con 0x55e6834a0030 2024-03-28T05:31:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.960+0000 7f91017fa640 1 -- 172.21.15.23:0/2730019810 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f911400f930 con 0x55e6834a0030 2024-03-28T05:31:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.961+0000 7f91017fa640 1 -- 172.21.15.23:0/2730019810 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f9114004b30 con 0x55e6834a0030 2024-03-28T05:31:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.961+0000 7f9122d67740 10 client.25909 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:31:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.961+0000 7f9122d67740 10 client.25909 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:31:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.961+0000 7f9122d67740 20 client.25909 populate_metadata read hostname 'smithi023' 2024-03-28T05:31:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.961+0000 7f9122d67740 1 --2- 172.21.15.23:0/2730019810 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55e6834b90e0 0x55e6834bb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.961+0000 7f9122d67740 4 client.25909 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T05:31:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.961+0000 7f9122d67740 1 -- 172.21.15.23:0/2730019810 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e683411050 con 0x55e6834b90e0 2024-03-28T05:31:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.961+0000 7f911df13640 1 --2- 172.21.15.23:0/2730019810 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55e6834b90e0 0x55e6834bb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.962+0000 7f91017fa640 10 client.25909 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:31:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.962+0000 7f911df13640 1 --2- 172.21.15.23:0/2730019810 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55e6834b90e0 0x55e6834bb4d0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=969317ddcbbde62b server_cookie=52f4cdc4401063fe in_seq=0 out_seq=0 2024-03-28T05:31:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.965+0000 7f91017fa640 1 -- 172.21.15.23:0/2730019810 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e683411050 con 0x55e6834b90e0 2024-03-28T05:31:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.965+0000 7f91017fa640 10 client.25909 handle_command_reply: tid=0 2024-03-28T05:31:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.965+0000 7f9122d67740 1 client.25909 shutdown 2024-03-28T05:31:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.965+0000 7f9122d67740 1 -- 172.21.15.23:0/2730019810 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6834a0030 msgr2=0x55e6834a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:31:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.965+0000 7f9122d67740 1 --2- 172.21.15.23:0/2730019810 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6834a0030 0x55e6834a6360 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9114005b50 tx=0x7f911400fd10 comp rx=0 tx=0).stop 2024-03-28T05:31:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.965+0000 7f9122d67740 1 -- 172.21.15.23:0/2730019810 shutdown_connections 2024-03-28T05:31:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.965+0000 7f9122d67740 1 --2- 172.21.15.23:0/2730019810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6834a8e00 0x55e6834a68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.965+0000 7f9122d67740 1 --2- 172.21.15.23:0/2730019810 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6834a0030 0x55e6834a6360 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.965+0000 7f9122d67740 1 --2- 172.21.15.23:0/2730019810 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55e6834b90e0 0x55e6834bb4d0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.965+0000 7f91017fa640 0 client.25909 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:31:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.965+0000 7f9122d67740 1 --2- 172.21.15.23:0/2730019810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e68349f710 0x55e6834a5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.965+0000 7f9122d67740 1 -- 172.21.15.23:0/2730019810 >> 172.21.15.23:0/2730019810 conn(0x55e68349a030 msgr2=0x55e68349bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:31:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.965+0000 7f9122d67740 1 -- 172.21.15.23:0/2730019810 shutdown_connections 2024-03-28T05:31:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.965+0000 7f9122d67740 1 -- 172.21.15.23:0/2730019810 wait complete. 2024-03-28T05:31:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.965+0000 7f9122d67740 20 client.25909 trim_cache size 0 max 16384 2024-03-28T05:31:15.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.983+0000 7f9122d67740 1 Processor -- start 2024-03-28T05:31:15.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.984+0000 7f9122d67740 1 -- start start 2024-03-28T05:31:15.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.984+0000 7f9122d67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e68349f710 0x55e6834a8550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:15.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.984+0000 7f9122d67740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6834a0030 0x55e68349bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:15.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.984+0000 7f9122d67740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6834a8e00 0x55e6834b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:15.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.984+0000 7f9122d67740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc00e1a0 con 0x55e6834a8e00 2024-03-28T05:31:15.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.985+0000 7f9122d67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc0107a0 con 0x55e68349f710 2024-03-28T05:31:15.986 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.985+0000 7f9122d67740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc00e7c0 con 0x55e6834a0030 2024-03-28T05:31:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.985+0000 7f911d712640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6834a0030 0x55e68349bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.985+0000 7f911d712640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6834a0030 0x55e68349bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:48538/0 (socket says 172.21.15.23:48538) 2024-03-28T05:31:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.985+0000 7f911d712640 1 -- 172.21.15.23:0/1289008723 learned_addr learned my addr 172.21.15.23:0/1289008723 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:31:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.985+0000 7f911df13640 1 --2- 172.21.15.23:0/1289008723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e68349f710 0x55e6834a8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.985+0000 7f911e714640 1 --2- 172.21.15.23:0/1289008723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6834a8e00 0x55e6834b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.985+0000 7f911d712640 1 -- 172.21.15.23:0/1289008723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e68349f710 msgr2=0x55e6834a8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:31:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.985+0000 7f911d712640 1 --2- 172.21.15.23:0/1289008723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e68349f710 0x55e6834a8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.985+0000 7f911d712640 1 -- 172.21.15.23:0/1289008723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6834a8e00 msgr2=0x55e6834b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:31:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.985+0000 7f911d712640 1 --2- 172.21.15.23:0/1289008723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6834a8e00 0x55e6834b42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.985+0000 7f911d712640 1 -- 172.21.15.23:0/1289008723 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e683495450 con 0x55e6834a0030 2024-03-28T05:31:15.992 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.986+0000 7f911d712640 1 --2- 172.21.15.23:0/1289008723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6834a0030 0x55e68349bb40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f911400e990 tx=0x7f911400ae40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7759ce58748eb1c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:31:15.992 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.986+0000 7f91037fe640 1 -- 172.21.15.23:0/1289008723 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114022370 con 0x55e6834a0030 2024-03-28T05:31:15.992 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.986+0000 7f91037fe640 1 -- 172.21.15.23:0/1289008723 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f911401e070 con 0x55e6834a0030 2024-03-28T05:31:15.992 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.986+0000 7f91037fe640 1 -- 172.21.15.23:0/1289008723 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911402cdd0 con 0x55e6834a0030 2024-03-28T05:31:15.992 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.987+0000 7f9122d67740 1 -- 172.21.15.23:0/1289008723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6834a0030 msgr2=0x55e68349bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:31:15.992 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.987+0000 7f9122d67740 1 --2- 172.21.15.23:0/1289008723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6834a0030 0x55e68349bb40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f911400e990 tx=0x7f911400ae40 comp rx=0 tx=0).stop 2024-03-28T05:31:15.993 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.987+0000 7f9122d67740 1 -- 172.21.15.23:0/1289008723 shutdown_connections 2024-03-28T05:31:15.993 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.987+0000 7f9122d67740 1 --2- 172.21.15.23:0/1289008723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6834a8e00 0x55e6834b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.993 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.987+0000 7f9122d67740 1 --2- 172.21.15.23:0/1289008723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6834a0030 0x55e68349bb40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.993 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.987+0000 7f9122d67740 1 --2- 172.21.15.23:0/1289008723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e68349f710 0x55e6834a8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.993 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.987+0000 7f9122d67740 1 -- 172.21.15.23:0/1289008723 >> 172.21.15.23:0/1289008723 conn(0x55e68349a030 msgr2=0x55e68349b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:31:15.993 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.987+0000 7f9122d67740 1 -- 172.21.15.23:0/1289008723 shutdown_connections 2024-03-28T05:31:15.993 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.987+0000 7f9122d67740 1 -- 172.21.15.23:0/1289008723 wait complete. 2024-03-28T05:31:15.993 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.988+0000 7f9122d67740 1 -- start start 2024-03-28T05:31:15.993 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.988+0000 7f9122d67740 1 Processor -- start 2024-03-28T05:31:15.993 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.988+0000 7f9122d67740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e68349f710 0x55e6834b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:15.993 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.988+0000 7f9122d67740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6834a0030 0x55e6834abf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:15.993 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.989+0000 7f9122d67740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6834a8e00 0x55e6834ac440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:15.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.989+0000 7f9122d67740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc00ee20 con 0x55e68349f710 2024-03-28T05:31:15.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.989+0000 7f9122d67740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc010140 con 0x55e6834a8e00 2024-03-28T05:31:15.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.989+0000 7f9122d67740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc00f480 con 0x55e6834a0030 2024-03-28T05:31:15.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.989+0000 7f911d712640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6834a0030 0x55e6834abf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:15.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.989+0000 7f911d712640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6834a0030 0x55e6834abf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:48548/0 (socket says 172.21.15.23:48548) 2024-03-28T05:31:15.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.989+0000 7f911d712640 1 -- 172.21.15.23:0/617108628 learned_addr learned my addr 172.21.15.23:0/617108628 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:31:15.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.989+0000 7f911df13640 1 --2- 172.21.15.23:0/617108628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e68349f710 0x55e6834b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:15.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.989+0000 7f9122d67740 10 client.0 _ll_register_callbacks cb 0x7f911400a830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T05:31:15.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.989+0000 7f911e714640 1 --2- 172.21.15.23:0/617108628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6834a8e00 0x55e6834ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:15.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.989+0000 7f911d712640 1 -- 172.21.15.23:0/617108628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6834a8e00 msgr2=0x55e6834ac440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:31:15.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.989+0000 7f911d712640 1 --2- 172.21.15.23:0/617108628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6834a8e00 0x55e6834ac440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.994 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.989+0000 7f911d712640 1 -- 172.21.15.23:0/617108628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e68349f710 msgr2=0x55e6834b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T05:31:15.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.989+0000 7f911d712640 1 --2- 172.21.15.23:0/617108628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e68349f710 0x55e6834b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:15.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.989+0000 7f911d712640 1 -- 172.21.15.23:0/617108628 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e6834acbf0 con 0x55e6834a0030 2024-03-28T05:31:15.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.989+0000 7f911d712640 1 --2- 172.21.15.23:0/617108628 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6834a0030 0x55e6834abf00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9114005db0 tx=0x7f9114005b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1540fddee6b2d6ea server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:31:15.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.989+0000 7f91017fa640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-28T05:31:15.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.990+0000 7f91017fa640 1 -- 172.21.15.23:0/617108628 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911402c350 con 0x55e6834a0030 2024-03-28T05:31:15.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.990+0000 7f91017fa640 1 -- 172.21.15.23:0/617108628 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9613+0+0 (secure 0 0 0) 0x7f9114022d90 con 0x55e6834a0030 2024-03-28T05:31:15.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.990+0000 7f9122d67740 1 -- 172.21.15.23:0/617108628 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f911401e070 con 0x55e6834a0030 2024-03-28T05:31:15.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.990+0000 7f91017fa640 1 -- 172.21.15.23:0/617108628 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9114046050 con 0x55e6834a0030 2024-03-28T05:31:15.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.990+0000 7f9122d67740 10 client.25915 fetch_fsmap learned FSMap version 20 2024-03-28T05:31:15.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.990+0000 7f9122d67740 1 -- 172.21.15.23:0/617108628 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55e683514680 con 0x55e6834a0030 2024-03-28T05:31:15.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.991+0000 7f91017fa640 1 -- 172.21.15.23:0/617108628 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f911401e070 con 0x55e6834a0030 2024-03-28T05:31:15.995 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.991+0000 7f91017fa640 1 -- 172.21.15.23:0/617108628 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 20) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f9114024ce0 con 0x55e6834a0030 2024-03-28T05:31:15.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.991+0000 7f9122d67740 10 client.25915 fetch_fsmap finished waiting for FSMap version 20 2024-03-28T05:31:15.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.991+0000 7f9122d67740 10 client.25915 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.i 2024-03-28T05:31:15.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.991+0000 7f9122d67740 20 client.25915 populate_metadata read hostname 'smithi023' 2024-03-28T05:31:15.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.991+0000 7f9122d67740 1 --2- 172.21.15.23:0/617108628 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55e683514c50 0x55e683517070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:31:15.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.991+0000 7f9122d67740 4 client.25915 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T05:31:15.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.991+0000 7f9122d67740 1 -- 172.21.15.23:0/617108628 --> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e68340b3a0 con 0x55e683514c50 2024-03-28T05:31:15.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.991+0000 7f911df13640 1 --2- 172.21.15.23:0/617108628 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55e683514c50 0x55e683517070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:31:15.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.992+0000 7f91017fa640 10 client.25915 ms_handle_connect on v2:172.21.15.113:6838/212060489 2024-03-28T05:31:15.996 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:15.992+0000 7f911df13640 1 --2- 172.21.15.23:0/617108628 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55e683514c50 0x55e683517070 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e23f5d13d2fb40a server_cookie=1357ef9b1707e94c in_seq=0 out_seq=0 2024-03-28T05:31:16.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.149+0000 7f91017fa640 1 -- 172.21.15.23:0/617108628 <== mds.0 v2:172.21.15.113:6838/212060489 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e68340b3a0 con 0x55e683514c50 2024-03-28T05:31:16.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.149+0000 7f91017fa640 10 client.25915 handle_command_reply: tid=0 2024-03-28T05:31:16.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.149+0000 7f9122d67740 1 client.25915 shutdown 2024-03-28T05:31:16.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.150+0000 7f9122d67740 1 -- 172.21.15.23:0/617108628 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6834a0030 msgr2=0x55e6834abf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:31:16.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.150+0000 7f9122d67740 1 --2- 172.21.15.23:0/617108628 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6834a0030 0x55e6834abf00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9114005db0 tx=0x7f9114005b10 comp rx=0 tx=0).stop 2024-03-28T05:31:16.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.150+0000 7f9122d67740 1 -- 172.21.15.23:0/617108628 shutdown_connections 2024-03-28T05:31:16.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.150+0000 7f9122d67740 1 --2- 172.21.15.23:0/617108628 >> [v2:172.21.15.113:6838/212060489,v1:172.21.15.113:6839/212060489] conn(0x55e683514c50 0x55e683517070 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:16.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.150+0000 7f91017fa640 0 client.25915 ms_handle_reset on v2:172.21.15.113:6838/212060489 2024-03-28T05:31:16.152 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.150+0000 7f9122d67740 1 --2- 172.21.15.23:0/617108628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e6834a8e00 0x55e6834ac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.150+0000 7f9122d67740 1 --2- 172.21.15.23:0/617108628 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6834a0030 0x55e6834abf00 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.150+0000 7f9122d67740 1 --2- 172.21.15.23:0/617108628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e68349f710 0x55e6834b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.150+0000 7f9122d67740 1 -- 172.21.15.23:0/617108628 >> 172.21.15.23:0/617108628 conn(0x55e68349a030 msgr2=0x55e683499020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:31:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.150+0000 7f9122d67740 1 -- 172.21.15.23:0/617108628 shutdown_connections 2024-03-28T05:31:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.150+0000 7f9122d67740 1 -- 172.21.15.23:0/617108628 wait complete. 2024-03-28T05:31:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.150+0000 7f9122d67740 20 client.25915 trim_cache size 0 max 16384 2024-03-28T05:31:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T05:31:16.153 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T05:31:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T05:31:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T05:31:16.154 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.151+0000 7f9120177640 1 -- 172.21.15.23:0/3574072821 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f91040a2c40 msgr2=0x7f91040a5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:31:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.151+0000 7f9120177640 1 --2- 172.21.15.23:0/3574072821 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f91040a2c40 0x7f91040a5100 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9110037040 tx=0x7f91100095c0 comp rx=0 tx=0).stop 2024-03-28T05:31:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.151+0000 7f9120177640 1 -- 172.21.15.23:0/3574072821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9118073a40 msgr2=0x7f9118072dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:31:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.151+0000 7f9120177640 1 --2- 172.21.15.23:0/3574072821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9118073a40 0x7f9118072dd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f910800b450 tx=0x7f910800b910 comp rx=0 tx=0).stop 2024-03-28T05:31:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.151+0000 7f9120177640 1 -- 172.21.15.23:0/3574072821 shutdown_connections 2024-03-28T05:31:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.151+0000 7f9120177640 1 --2- 172.21.15.23:0/3574072821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9118112680 0x7f911806c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.151+0000 7f9120177640 1 --2- 172.21.15.23:0/3574072821 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f91040a2c40 0x7f91040a5100 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.151+0000 7f9120177640 1 --2- 172.21.15.23:0/3574072821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9118074360 0x7f9118073310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.151+0000 7f9120177640 1 --2- 172.21.15.23:0/3574072821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9118073a40 0x7f9118072dd0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:31:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.151+0000 7f9120177640 1 -- 172.21.15.23:0/3574072821 >> 172.21.15.23:0/3574072821 conn(0x7f91180fd180 msgr2=0x7f91180fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:31:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.152+0000 7f9120177640 1 -- 172.21.15.23:0/3574072821 shutdown_connections 2024-03-28T05:31:16.155 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T05:31:16.152+0000 7f9120177640 1 -- 172.21.15.23:0/3574072821 wait complete. 2024-03-28T05:31:17.494 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:17 smithi113 ceph-mon[37164]: pgmap v237: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 244 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 779 KiB/s wr, 209 op/s 2024-03-28T05:31:17.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:17 smithi023 ceph-mon[31954]: pgmap v237: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 244 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 779 KiB/s wr, 209 op/s 2024-03-28T05:31:17.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:17 smithi098 ceph-mon[36087]: pgmap v237: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 244 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 779 KiB/s wr, 209 op/s 2024-03-28T05:31:19.494 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:19 smithi113 ceph-mon[37164]: pgmap v238: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 244 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 658 KiB/s wr, 176 op/s 2024-03-28T05:31:19.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:19 smithi023 ceph-mon[31954]: pgmap v238: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 244 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 658 KiB/s wr, 176 op/s 2024-03-28T05:31:19.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:19 smithi098 ceph-mon[36087]: pgmap v238: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 244 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 658 KiB/s wr, 176 op/s 2024-03-28T05:31:21.494 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:21 smithi113 ceph-mon[37164]: pgmap v239: 129 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 118 active+clean; 243 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 1015 KiB/s wr, 290 op/s 2024-03-28T05:31:21.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:21 smithi023 ceph-mon[31954]: pgmap v239: 129 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 118 active+clean; 243 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 1015 KiB/s wr, 290 op/s 2024-03-28T05:31:21.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:21 smithi098 ceph-mon[36087]: pgmap v239: 129 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 118 active+clean; 243 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 1015 KiB/s wr, 290 op/s 2024-03-28T05:31:23.418 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:23 smithi098 ceph-mon[36087]: pgmap v240: 129 pgs: 129 active+clean; 238 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 1.1 MiB/s wr, 370 op/s 2024-03-28T05:31:23.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:23 smithi113 ceph-mon[37164]: pgmap v240: 129 pgs: 129 active+clean; 238 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 1.1 MiB/s wr, 370 op/s 2024-03-28T05:31:23.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:23 smithi023 ceph-mon[31954]: pgmap v240: 129 pgs: 129 active+clean; 238 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 1.1 MiB/s wr, 370 op/s 2024-03-28T05:31:25.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:24 smithi098 ceph-mon[36087]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T05:31:25.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:24 smithi113 ceph-mon[37164]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T05:31:25.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:24 smithi023 ceph-mon[31954]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T05:31:26.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:25 smithi098 ceph-mon[36087]: pgmap v242: 129 pgs: 129 active+clean; 238 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 1.1 MiB/s wr, 378 op/s 2024-03-28T05:31:26.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:25 smithi113 ceph-mon[37164]: pgmap v242: 129 pgs: 129 active+clean; 238 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 1.1 MiB/s wr, 378 op/s 2024-03-28T05:31:26.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:25 smithi023 ceph-mon[31954]: pgmap v242: 129 pgs: 129 active+clean; 238 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 1.1 MiB/s wr, 378 op/s 2024-03-28T05:31:26.952 INFO:tasks.workunit:Stopping ['suites/blogbench.sh'] on client.0... 2024-03-28T05:31:26.952 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-28T05:31:27.353 DEBUG:teuthology.parallel:result is None 2024-03-28T05:31:27.353 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-28T05:31:27.386 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-28T05:31:27.386 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-28T05:31:27.401 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi023.front.sepia.ceph.com 2024-03-28T05:31:27.434 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-28T05:31:27.443 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-28T05:31:27.443 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T05:31:27.443 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-28T05:31:27.443 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-28T05:31:27.502 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-28T05:31:28.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:27 smithi098 ceph-mon[36087]: pgmap v243: 129 pgs: 129 active+clean; 245 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 548 op/s 2024-03-28T05:31:28.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:27 smithi113 ceph-mon[37164]: pgmap v243: 129 pgs: 129 active+clean; 245 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 548 op/s 2024-03-28T05:31:28.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:27 smithi023 ceph-mon[31954]: pgmap v243: 129 pgs: 129 active+clean; 245 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 548 op/s 2024-03-28T05:31:30.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:29 smithi098 ceph-mon[36087]: pgmap v244: 129 pgs: 129 active+clean; 245 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 548 op/s 2024-03-28T05:31:30.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:29 smithi113 ceph-mon[37164]: pgmap v244: 129 pgs: 129 active+clean; 245 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 548 op/s 2024-03-28T05:31:30.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:29 smithi023 ceph-mon[31954]: pgmap v244: 129 pgs: 129 active+clean; 245 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 548 op/s 2024-03-28T05:31:32.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:31 smithi098 ceph-mon[36087]: pgmap v245: 129 pgs: 129 active+clean; 245 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 1.1 MiB/s wr, 452 op/s 2024-03-28T05:31:32.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:31 smithi113 ceph-mon[37164]: pgmap v245: 129 pgs: 129 active+clean; 245 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 1.1 MiB/s wr, 452 op/s 2024-03-28T05:31:32.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:31 smithi023 ceph-mon[31954]: pgmap v245: 129 pgs: 129 active+clean; 245 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 1.1 MiB/s wr, 452 op/s 2024-03-28T05:31:34.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:33 smithi098 ceph-mon[36087]: pgmap v246: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 397 op/s 2024-03-28T05:31:34.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:33 smithi113 ceph-mon[37164]: pgmap v246: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 397 op/s 2024-03-28T05:31:34.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:33 smithi023 ceph-mon[31954]: pgmap v246: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 397 op/s 2024-03-28T05:31:36.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:35 smithi098 ceph-mon[36087]: pgmap v247: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1013 KiB/s wr, 385 op/s 2024-03-28T05:31:36.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:35 smithi113 ceph-mon[37164]: pgmap v247: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1013 KiB/s wr, 385 op/s 2024-03-28T05:31:36.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:35 smithi023 ceph-mon[31954]: pgmap v247: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1013 KiB/s wr, 385 op/s 2024-03-28T05:31:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:37 smithi098 ceph-mon[36087]: pgmap v248: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 873 KiB/s wr, 336 op/s 2024-03-28T05:31:38.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:37 smithi113 ceph-mon[37164]: pgmap v248: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 873 KiB/s wr, 336 op/s 2024-03-28T05:31:38.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:37 smithi023 ceph-mon[31954]: pgmap v248: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 873 KiB/s wr, 336 op/s 2024-03-28T05:31:40.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:39 smithi098 ceph-mon[36087]: pgmap v249: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 242 KiB/s wr, 114 op/s 2024-03-28T05:31:40.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:39 smithi113 ceph-mon[37164]: pgmap v249: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 242 KiB/s wr, 114 op/s 2024-03-28T05:31:40.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:39 smithi023 ceph-mon[31954]: pgmap v249: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 242 KiB/s wr, 114 op/s 2024-03-28T05:31:42.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:41 smithi098 ceph-mon[36087]: pgmap v250: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 242 KiB/s wr, 114 op/s 2024-03-28T05:31:42.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:41 smithi113 ceph-mon[37164]: pgmap v250: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 242 KiB/s wr, 114 op/s 2024-03-28T05:31:42.322 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:41 smithi023 ceph-mon[31954]: pgmap v250: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 242 KiB/s wr, 114 op/s 2024-03-28T05:31:44.084 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:43 smithi023 ceph-mon[31954]: pgmap v251: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 248 KiB/s wr, 82 op/s 2024-03-28T05:31:44.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:43 smithi098 ceph-mon[36087]: pgmap v251: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 248 KiB/s wr, 82 op/s 2024-03-28T05:31:44.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:43 smithi113 ceph-mon[37164]: pgmap v251: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 248 KiB/s wr, 82 op/s 2024-03-28T05:31:46.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:45 smithi098 ceph-mon[36087]: pgmap v252: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 8.9 KiB/s wr, 7 op/s 2024-03-28T05:31:46.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:45 smithi113 ceph-mon[37164]: pgmap v252: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 8.9 KiB/s wr, 7 op/s 2024-03-28T05:31:46.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:45 smithi023 ceph-mon[31954]: pgmap v252: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 8.9 KiB/s wr, 7 op/s 2024-03-28T05:31:48.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:47 smithi098 ceph-mon[36087]: pgmap v253: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.9 KiB/s wr, 8 op/s 2024-03-28T05:31:48.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:47 smithi113 ceph-mon[37164]: pgmap v253: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.9 KiB/s wr, 8 op/s 2024-03-28T05:31:48.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:47 smithi023 ceph-mon[31954]: pgmap v253: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.9 KiB/s wr, 8 op/s 2024-03-28T05:31:49.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:48 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:31:49.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:48 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:31:49.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:48 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:31:49.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:48 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:31:49.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:48 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:31:49.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:48 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:31:49.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:48 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:31:49.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:48 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:31:49.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:48 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:31:49.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:48 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:31:49.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:48 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:31:49.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:48 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:31:50.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:49 smithi098 ceph-mon[36087]: pgmap v254: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.4 KiB/s wr, 2 op/s 2024-03-28T05:31:50.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:49 smithi113 ceph-mon[37164]: pgmap v254: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.4 KiB/s wr, 2 op/s 2024-03-28T05:31:50.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:49 smithi023 ceph-mon[31954]: pgmap v254: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.4 KiB/s wr, 2 op/s 2024-03-28T05:31:52.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:51 smithi098 ceph-mon[36087]: pgmap v255: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.4 KiB/s wr, 3 op/s 2024-03-28T05:31:52.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:51 smithi113 ceph-mon[37164]: pgmap v255: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.4 KiB/s wr, 3 op/s 2024-03-28T05:31:52.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:51 smithi023 ceph-mon[31954]: pgmap v255: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.4 KiB/s wr, 3 op/s 2024-03-28T05:31:54.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:53 smithi098 ceph-mon[36087]: pgmap v256: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.8 KiB/s wr, 3 op/s 2024-03-28T05:31:54.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:53 smithi113 ceph-mon[37164]: pgmap v256: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.8 KiB/s wr, 3 op/s 2024-03-28T05:31:54.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:53 smithi023 ceph-mon[31954]: pgmap v256: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.8 KiB/s wr, 3 op/s 2024-03-28T05:31:56.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:55 smithi098 ceph-mon[36087]: pgmap v257: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-28T05:31:56.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:55 smithi113 ceph-mon[37164]: pgmap v257: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-28T05:31:56.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:55 smithi023 ceph-mon[31954]: pgmap v257: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-28T05:31:58.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:57 smithi098 ceph-mon[36087]: pgmap v258: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-28T05:31:58.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:57 smithi113 ceph-mon[37164]: pgmap v258: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-28T05:31:58.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:57 smithi023 ceph-mon[31954]: pgmap v258: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-28T05:32:00.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:31:59 smithi098 ceph-mon[36087]: pgmap v259: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-28T05:32:00.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:31:59 smithi113 ceph-mon[37164]: pgmap v259: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-28T05:32:00.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:31:59 smithi023 ceph-mon[31954]: pgmap v259: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-28T05:32:02.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:01 smithi113 ceph-mon[37164]: pgmap v260: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-28T05:32:02.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:01 smithi023 ceph-mon[31954]: pgmap v260: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-28T05:32:02.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:01 smithi098 ceph-mon[36087]: pgmap v260: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-28T05:32:04.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:03 smithi113 ceph-mon[37164]: pgmap v261: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-28T05:32:04.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:03 smithi023 ceph-mon[31954]: pgmap v261: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-28T05:32:04.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:03 smithi098 ceph-mon[36087]: pgmap v261: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-28T05:32:06.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:05 smithi113 ceph-mon[37164]: pgmap v262: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:32:06.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:05 smithi023 ceph-mon[31954]: pgmap v262: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:32:06.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:05 smithi098 ceph-mon[36087]: pgmap v262: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:32:08.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:07 smithi113 ceph-mon[37164]: pgmap v263: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-28T05:32:08.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:07 smithi023 ceph-mon[31954]: pgmap v263: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-28T05:32:08.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:07 smithi098 ceph-mon[36087]: pgmap v263: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-28T05:32:10.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:09 smithi113 ceph-mon[37164]: pgmap v264: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 18 op/s 2024-03-28T05:32:10.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:09 smithi023 ceph-mon[31954]: pgmap v264: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 18 op/s 2024-03-28T05:32:10.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:09 smithi098 ceph-mon[36087]: pgmap v264: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 18 op/s 2024-03-28T05:32:12.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:11 smithi113 ceph-mon[37164]: pgmap v265: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-28T05:32:12.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:11 smithi113 ceph-mon[37164]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T05:32:12.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:11 smithi023 ceph-mon[31954]: pgmap v265: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-28T05:32:12.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:11 smithi023 ceph-mon[31954]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T05:32:12.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:11 smithi098 ceph-mon[36087]: pgmap v265: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-28T05:32:12.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:11 smithi098 ceph-mon[36087]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T05:32:14.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:13 smithi113 ceph-mon[37164]: pgmap v267: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-03-28T05:32:14.281 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:13 smithi023 ceph-mon[31954]: pgmap v267: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-03-28T05:32:14.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:13 smithi098 ceph-mon[36087]: pgmap v267: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-03-28T05:32:16.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:15 smithi113 ceph-mon[37164]: pgmap v268: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-03-28T05:32:16.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:15 smithi023 ceph-mon[31954]: pgmap v268: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-03-28T05:32:16.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:15 smithi098 ceph-mon[36087]: pgmap v268: 129 pgs: 129 active+clean; 248 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-03-28T05:32:18.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:17 smithi113 ceph-mon[37164]: pgmap v269: 129 pgs: 20 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 55 active+clean; 246 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:32:18.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:17 smithi023 ceph-mon[31954]: pgmap v269: 129 pgs: 20 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 55 active+clean; 246 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:32:18.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:17 smithi098 ceph-mon[36087]: pgmap v269: 129 pgs: 20 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 55 active+clean; 246 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:32:20.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:19 smithi113 ceph-mon[37164]: pgmap v270: 129 pgs: 20 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 55 active+clean; 246 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:32:20.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:19 smithi023 ceph-mon[31954]: pgmap v270: 129 pgs: 20 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 55 active+clean; 246 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:32:20.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:19 smithi098 ceph-mon[36087]: pgmap v270: 129 pgs: 20 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 55 active+clean; 246 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:32:22.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:21 smithi113 ceph-mon[37164]: pgmap v271: 129 pgs: 14 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 68 active+clean; 245 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:32:22.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:21 smithi023 ceph-mon[31954]: pgmap v271: 129 pgs: 14 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 68 active+clean; 245 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:32:22.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:21 smithi098 ceph-mon[36087]: pgmap v271: 129 pgs: 14 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 68 active+clean; 245 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:32:24.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:23 smithi113 ceph-mon[37164]: pgmap v272: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 243 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 183 B/s wr, 2 op/s 2024-03-28T05:32:24.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:23 smithi023 ceph-mon[31954]: pgmap v272: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 243 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 183 B/s wr, 2 op/s 2024-03-28T05:32:24.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:23 smithi098 ceph-mon[36087]: pgmap v272: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 243 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 183 B/s wr, 2 op/s 2024-03-28T05:32:26.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:25 smithi113 ceph-mon[37164]: pgmap v273: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 243 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-28T05:32:26.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:25 smithi023 ceph-mon[31954]: pgmap v273: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 243 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-28T05:32:26.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:25 smithi098 ceph-mon[36087]: pgmap v273: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 243 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-28T05:32:28.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:27 smithi113 ceph-mon[37164]: pgmap v274: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-28T05:32:28.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:27 smithi023 ceph-mon[31954]: pgmap v274: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-28T05:32:28.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:27 smithi098 ceph-mon[36087]: pgmap v274: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-28T05:32:30.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:29 smithi098 ceph-mon[36087]: pgmap v275: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2024-03-28T05:32:30.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:29 smithi098 ceph-mon[36087]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T05:32:30.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:29 smithi113 ceph-mon[37164]: pgmap v275: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2024-03-28T05:32:30.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:29 smithi113 ceph-mon[37164]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T05:32:30.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:29 smithi023 ceph-mon[31954]: pgmap v275: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2024-03-28T05:32:30.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:29 smithi023 ceph-mon[31954]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T05:32:32.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:31 smithi098 ceph-mon[36087]: pgmap v277: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-28T05:32:32.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:31 smithi113 ceph-mon[37164]: pgmap v277: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-28T05:32:32.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:31 smithi023 ceph-mon[31954]: pgmap v277: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-28T05:32:34.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:33 smithi098 ceph-mon[36087]: pgmap v278: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:32:34.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:33 smithi113 ceph-mon[37164]: pgmap v278: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:32:34.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:33 smithi023 ceph-mon[31954]: pgmap v278: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:32:36.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:35 smithi098 ceph-mon[36087]: pgmap v279: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:32:36.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:35 smithi113 ceph-mon[37164]: pgmap v279: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:32:36.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:35 smithi023 ceph-mon[31954]: pgmap v279: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:32:38.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:37 smithi098 ceph-mon[36087]: pgmap v280: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:32:38.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:37 smithi113 ceph-mon[37164]: pgmap v280: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:32:38.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:37 smithi023 ceph-mon[31954]: pgmap v280: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:32:40.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:39 smithi098 ceph-mon[36087]: pgmap v281: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:32:40.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:39 smithi113 ceph-mon[37164]: pgmap v281: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:32:40.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:39 smithi023 ceph-mon[31954]: pgmap v281: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:32:42.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:41 smithi098 ceph-mon[36087]: pgmap v282: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-28T05:32:42.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:41 smithi113 ceph-mon[37164]: pgmap v282: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-28T05:32:42.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:41 smithi023 ceph-mon[31954]: pgmap v282: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-28T05:32:44.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:43 smithi098 ceph-mon[36087]: pgmap v283: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-03-28T05:32:44.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:43 smithi113 ceph-mon[37164]: pgmap v283: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-03-28T05:32:44.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:43 smithi023 ceph-mon[31954]: pgmap v283: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-03-28T05:32:46.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:45 smithi098 ceph-mon[36087]: pgmap v284: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:32:46.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:45 smithi113 ceph-mon[37164]: pgmap v284: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:32:46.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:45 smithi023 ceph-mon[31954]: pgmap v284: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:32:48.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:47 smithi098 ceph-mon[36087]: pgmap v285: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:32:48.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:47 smithi113 ceph-mon[37164]: pgmap v285: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:32:48.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:47 smithi023 ceph-mon[31954]: pgmap v285: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:32:49.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:48 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:32:49.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:48 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:32:49.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:48 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:32:50.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:49 smithi098 ceph-mon[36087]: pgmap v286: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:32:50.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:49 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:32:50.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:49 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:32:50.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:49 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:32:50.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:49 smithi113 ceph-mon[37164]: pgmap v286: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:32:50.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:49 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:32:50.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:49 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:32:50.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:49 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:32:50.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:49 smithi023 ceph-mon[31954]: pgmap v286: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:32:50.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:49 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:32:50.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:49 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:32:50.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:49 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:32:52.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:51 smithi113 ceph-mon[37164]: pgmap v287: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:32:52.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:51 smithi023 ceph-mon[31954]: pgmap v287: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:32:52.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:51 smithi098 ceph-mon[36087]: pgmap v287: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:32:54.169 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:53 smithi098 ceph-mon[36087]: pgmap v288: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-03-28T05:32:54.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:53 smithi113 ceph-mon[37164]: pgmap v288: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-03-28T05:32:54.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:53 smithi023 ceph-mon[31954]: pgmap v288: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-03-28T05:32:56.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:55 smithi113 ceph-mon[37164]: pgmap v289: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:32:56.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:55 smithi023 ceph-mon[31954]: pgmap v289: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:32:56.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:55 smithi098 ceph-mon[36087]: pgmap v289: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:32:58.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:57 smithi113 ceph-mon[37164]: pgmap v290: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:32:58.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:57 smithi023 ceph-mon[31954]: pgmap v290: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:32:58.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:57 smithi098 ceph-mon[36087]: pgmap v290: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:33:00.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:32:59 smithi113 ceph-mon[37164]: pgmap v291: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:33:00.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:32:59 smithi023 ceph-mon[31954]: pgmap v291: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:33:00.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:32:59 smithi098 ceph-mon[36087]: pgmap v291: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:33:02.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:01 smithi113 ceph-mon[37164]: pgmap v292: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:33:02.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:01 smithi023 ceph-mon[31954]: pgmap v292: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:33:02.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:01 smithi098 ceph-mon[36087]: pgmap v292: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:33:04.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:03 smithi113 ceph-mon[37164]: pgmap v293: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-03-28T05:33:04.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:03 smithi023 ceph-mon[31954]: pgmap v293: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-03-28T05:33:04.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:03 smithi098 ceph-mon[36087]: pgmap v293: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-03-28T05:33:06.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:05 smithi113 ceph-mon[37164]: pgmap v294: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:33:06.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:05 smithi023 ceph-mon[31954]: pgmap v294: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:33:06.367 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:05 smithi098 ceph-mon[36087]: pgmap v294: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:33:08.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:07 smithi113 ceph-mon[37164]: pgmap v295: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-28T05:33:08.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:07 smithi023 ceph-mon[31954]: pgmap v295: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-28T05:33:08.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:07 smithi098 ceph-mon[36087]: pgmap v295: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-28T05:33:10.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:09 smithi113 ceph-mon[37164]: pgmap v296: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 18 op/s 2024-03-28T05:33:10.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:09 smithi023 ceph-mon[31954]: pgmap v296: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 18 op/s 2024-03-28T05:33:10.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:09 smithi098 ceph-mon[36087]: pgmap v296: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 18 op/s 2024-03-28T05:33:12.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:11 smithi113 ceph-mon[37164]: pgmap v297: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-28T05:33:12.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:11 smithi113 ceph-mon[37164]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T05:33:12.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:11 smithi023 ceph-mon[31954]: pgmap v297: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-28T05:33:12.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:11 smithi023 ceph-mon[31954]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T05:33:12.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:11 smithi098 ceph-mon[36087]: pgmap v297: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-28T05:33:12.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:11 smithi098 ceph-mon[36087]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T05:33:14.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:13 smithi113 ceph-mon[37164]: pgmap v299: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-03-28T05:33:14.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:13 smithi023 ceph-mon[31954]: pgmap v299: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-03-28T05:33:14.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:13 smithi098 ceph-mon[36087]: pgmap v299: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-03-28T05:33:16.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:15 smithi113 ceph-mon[37164]: pgmap v300: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-03-28T05:33:16.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:15 smithi023 ceph-mon[31954]: pgmap v300: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-03-28T05:33:16.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:15 smithi098 ceph-mon[36087]: pgmap v300: 129 pgs: 129 active+clean; 242 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-03-28T05:33:18.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:17 smithi113 ceph-mon[37164]: pgmap v301: 129 pgs: 21 active+clean+snaptrim, 61 active+clean+snaptrim_wait, 47 active+clean; 240 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:33:18.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:17 smithi023 ceph-mon[31954]: pgmap v301: 129 pgs: 21 active+clean+snaptrim, 61 active+clean+snaptrim_wait, 47 active+clean; 240 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:33:18.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:17 smithi098 ceph-mon[36087]: pgmap v301: 129 pgs: 21 active+clean+snaptrim, 61 active+clean+snaptrim_wait, 47 active+clean; 240 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:33:20.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:19 smithi113 ceph-mon[37164]: pgmap v302: 129 pgs: 21 active+clean+snaptrim, 61 active+clean+snaptrim_wait, 47 active+clean; 240 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:33:20.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:19 smithi023 ceph-mon[31954]: pgmap v302: 129 pgs: 21 active+clean+snaptrim, 61 active+clean+snaptrim_wait, 47 active+clean; 240 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:33:20.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:19 smithi098 ceph-mon[36087]: pgmap v302: 129 pgs: 21 active+clean+snaptrim, 61 active+clean+snaptrim_wait, 47 active+clean; 240 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:33:22.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:21 smithi113 ceph-mon[37164]: pgmap v303: 129 pgs: 15 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 67 active+clean; 240 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:33:22.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:21 smithi023 ceph-mon[31954]: pgmap v303: 129 pgs: 15 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 67 active+clean; 240 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:33:22.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:21 smithi098 ceph-mon[36087]: pgmap v303: 129 pgs: 15 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 67 active+clean; 240 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:33:24.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:23 smithi113 ceph-mon[37164]: pgmap v304: 129 pgs: 10 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 95 active+clean; 238 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 183 B/s wr, 2 op/s 2024-03-28T05:33:24.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:23 smithi023 ceph-mon[31954]: pgmap v304: 129 pgs: 10 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 95 active+clean; 238 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 183 B/s wr, 2 op/s 2024-03-28T05:33:24.367 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:23 smithi098 ceph-mon[36087]: pgmap v304: 129 pgs: 10 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 95 active+clean; 238 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 183 B/s wr, 2 op/s 2024-03-28T05:33:26.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:25 smithi113 ceph-mon[37164]: pgmap v305: 129 pgs: 10 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 95 active+clean; 238 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-28T05:33:26.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:25 smithi023 ceph-mon[31954]: pgmap v305: 129 pgs: 10 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 95 active+clean; 238 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-28T05:33:26.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:25 smithi098 ceph-mon[36087]: pgmap v305: 129 pgs: 10 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 95 active+clean; 238 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-28T05:33:28.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:27 smithi113 ceph-mon[37164]: pgmap v306: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-28T05:33:28.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:27 smithi023 ceph-mon[31954]: pgmap v306: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-28T05:33:28.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:27 smithi098 ceph-mon[36087]: pgmap v306: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-28T05:33:30.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:29 smithi098 ceph-mon[36087]: pgmap v307: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2024-03-28T05:33:30.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:29 smithi098 ceph-mon[36087]: osdmap e102: 12 total, 12 up, 12 in 2024-03-28T05:33:30.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:29 smithi113 ceph-mon[37164]: pgmap v307: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2024-03-28T05:33:30.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:29 smithi113 ceph-mon[37164]: osdmap e102: 12 total, 12 up, 12 in 2024-03-28T05:33:30.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:29 smithi023 ceph-mon[31954]: pgmap v307: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2024-03-28T05:33:30.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:29 smithi023 ceph-mon[31954]: osdmap e102: 12 total, 12 up, 12 in 2024-03-28T05:33:32.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:31 smithi098 ceph-mon[36087]: pgmap v309: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-28T05:33:32.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:31 smithi113 ceph-mon[37164]: pgmap v309: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-28T05:33:32.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:31 smithi023 ceph-mon[31954]: pgmap v309: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-28T05:33:34.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:33 smithi098 ceph-mon[36087]: pgmap v310: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:33:34.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:33 smithi113 ceph-mon[37164]: pgmap v310: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:33:34.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:33 smithi023 ceph-mon[31954]: pgmap v310: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:33:36.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:35 smithi098 ceph-mon[36087]: pgmap v311: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:33:36.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:35 smithi113 ceph-mon[37164]: pgmap v311: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:33:36.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:35 smithi023 ceph-mon[31954]: pgmap v311: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:33:36.575 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-28T05:33:36.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.690+0000 7fb231e36640 1 -- 172.21.15.23:0/2608425808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb22c107660 msgr2=0x7fb22c107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:36.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.690+0000 7fb231e36640 1 --2- 172.21.15.23:0/2608425808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb22c107660 0x7fb22c107a40 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb21c009930 tx=0x7fb21c02bc30 comp rx=0 tx=0).stop 2024-03-28T05:33:36.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.690+0000 7fb231e36640 1 -- 172.21.15.23:0/2608425808 shutdown_connections 2024-03-28T05:33:36.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.690+0000 7fb231e36640 1 --2- 172.21.15.23:0/2608425808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb22c102010 0x7fb22c10e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:36.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.690+0000 7fb231e36640 1 --2- 172.21.15.23:0/2608425808 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb22c101650 0x7fb22c101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:36.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.690+0000 7fb231e36640 1 --2- 172.21.15.23:0/2608425808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb22c107660 0x7fb22c107a40 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:36.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.690+0000 7fb231e36640 1 -- 172.21.15.23:0/2608425808 >> 172.21.15.23:0/2608425808 conn(0x7fb22c0fd160 msgr2=0x7fb22c0ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:33:36.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.690+0000 7fb231e36640 1 -- 172.21.15.23:0/2608425808 shutdown_connections 2024-03-28T05:33:36.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.690+0000 7fb231e36640 1 -- 172.21.15.23:0/2608425808 wait complete. 2024-03-28T05:33:36.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.691+0000 7fb231e36640 1 Processor -- start 2024-03-28T05:33:36.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.691+0000 7fb231e36640 1 -- start start 2024-03-28T05:33:36.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.691+0000 7fb231e36640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb22c101650 0x7fb22c197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:36.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.691+0000 7fb231e36640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb22c102010 0x7fb22c198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:36.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.691+0000 7fb231e36640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb22c107660 0x7fb22c19c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:36.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.691+0000 7fb231e36640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c111a30 con 0x7fb22c107660 2024-03-28T05:33:36.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.691+0000 7fb231e36640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c1118b0 con 0x7fb22c102010 2024-03-28T05:33:36.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.691+0000 7fb231e36640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c111bb0 con 0x7fb22c101650 2024-03-28T05:33:36.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.691+0000 7fb22bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb22c107660 0x7fb22c19c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:36.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.692+0000 7fb22bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb22c107660 0x7fb22c19c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51796/0 (socket says 172.21.15.23:51796) 2024-03-28T05:33:36.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.692+0000 7fb22bfff640 1 -- 172.21.15.23:0/848455660 learned_addr learned my addr 172.21.15.23:0/848455660 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:33:36.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.692+0000 7fb22b7fe640 1 --2- 172.21.15.23:0/848455660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb22c101650 0x7fb22c197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:36.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.692+0000 7fb22affd640 1 --2- 172.21.15.23:0/848455660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb22c102010 0x7fb22c198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:36.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.692+0000 7fb22bfff640 1 -- 172.21.15.23:0/848455660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb22c101650 msgr2=0x7fb22c197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:36.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.692+0000 7fb22bfff640 1 --2- 172.21.15.23:0/848455660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb22c101650 0x7fb22c197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:36.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.692+0000 7fb22bfff640 1 -- 172.21.15.23:0/848455660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb22c102010 msgr2=0x7fb22c198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:36.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.692+0000 7fb22bfff640 1 --2- 172.21.15.23:0/848455660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb22c102010 0x7fb22c198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:36.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.692+0000 7fb22bfff640 1 -- 172.21.15.23:0/848455660 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c19ce40 con 0x7fb22c107660 2024-03-28T05:33:36.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.692+0000 7fb22bfff640 1 --2- 172.21.15.23:0/848455660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb22c107660 0x7fb22c19c710 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb22400eb10 tx=0x7fb22400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:33:36.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.692+0000 7fb228ff9640 1 -- 172.21.15.23:0/848455660 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb224004b20 con 0x7fb22c107660 2024-03-28T05:33:36.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.692+0000 7fb231e36640 1 -- 172.21.15.23:0/848455660 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb22c19d130 con 0x7fb22c107660 2024-03-28T05:33:36.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.692+0000 7fb228ff9640 1 -- 172.21.15.23:0/848455660 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb224004cc0 con 0x7fb22c107660 2024-03-28T05:33:36.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.692+0000 7fb228ff9640 1 -- 172.21.15.23:0/848455660 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb224010760 con 0x7fb22c107660 2024-03-28T05:33:36.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.692+0000 7fb231e36640 1 -- 172.21.15.23:0/848455660 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb22c1a49c0 con 0x7fb22c107660 2024-03-28T05:33:36.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.693+0000 7fb231e36640 1 -- 172.21.15.23:0/848455660 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb22c04f400 con 0x7fb22c107660 2024-03-28T05:33:36.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.694+0000 7fb228ff9640 1 -- 172.21.15.23:0/848455660 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fb224010900 con 0x7fb22c107660 2024-03-28T05:33:36.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.695+0000 7fb228ff9640 1 --2- 172.21.15.23:0/848455660 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fb2140a2a10 0x7fb2140a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:36.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.695+0000 7fb228ff9640 1 -- 172.21.15.23:0/848455660 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fb2240cfa40 con 0x7fb22c107660 2024-03-28T05:33:36.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.695+0000 7fb22b7fe640 1 --2- 172.21.15.23:0/848455660 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fb2140a2a10 0x7fb2140a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:36.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.695+0000 7fb22b7fe640 1 --2- 172.21.15.23:0/848455660 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fb2140a2a10 0x7fb2140a4ed0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb21c005ab0 tx=0x7fb21c005fd0 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:33:36.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.699+0000 7fb228ff9640 1 -- 172.21.15.23:0/848455660 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb224053740 con 0x7fb22c107660 2024-03-28T05:33:36.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.904+0000 7fb231e36640 1 -- 172.21.15.23:0/848455660 --> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7fb22c061ae0 con 0x7fb2140a2a10 2024-03-28T05:33:36.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.912+0000 7fb228ff9640 1 -- 172.21.15.23:0/848455660 <== mgr.15402 v2:172.21.15.23:6800/1385031613 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+350 (secure 0 0 0) 0x7fb22c061ae0 con 0x7fb2140a2a10 2024-03-28T05:33:36.913 INFO:teuthology.orchestra.run.smithi023.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-28T00:00:00", "created": "2024-03-28T05:23:34", "first": "2024-03-28T05:24:00", "last": "2024-03-28T05:33:00", "last_pruned": "2024-03-28T05:33:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-03-28T05:33:36.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.914+0000 7fb231e36640 1 -- 172.21.15.23:0/848455660 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fb2140a2a10 msgr2=0x7fb2140a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:36.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.914+0000 7fb231e36640 1 --2- 172.21.15.23:0/848455660 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fb2140a2a10 0x7fb2140a4ed0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb21c005ab0 tx=0x7fb21c005fd0 comp rx=0 tx=0).stop 2024-03-28T05:33:36.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.914+0000 7fb231e36640 1 -- 172.21.15.23:0/848455660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb22c107660 msgr2=0x7fb22c19c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:36.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.914+0000 7fb231e36640 1 --2- 172.21.15.23:0/848455660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb22c107660 0x7fb22c19c710 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb22400eb10 tx=0x7fb22400c2d0 comp rx=0 tx=0).stop 2024-03-28T05:33:36.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.914+0000 7fb231e36640 1 -- 172.21.15.23:0/848455660 shutdown_connections 2024-03-28T05:33:36.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.914+0000 7fb231e36640 1 --2- 172.21.15.23:0/848455660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb22c107660 0x7fb22c19c710 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:36.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.914+0000 7fb231e36640 1 --2- 172.21.15.23:0/848455660 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fb2140a2a10 0x7fb2140a4ed0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:36.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.914+0000 7fb231e36640 1 --2- 172.21.15.23:0/848455660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb22c102010 0x7fb22c198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:36.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.914+0000 7fb231e36640 1 --2- 172.21.15.23:0/848455660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb22c101650 0x7fb22c197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:36.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.914+0000 7fb231e36640 1 -- 172.21.15.23:0/848455660 >> 172.21.15.23:0/848455660 conn(0x7fb22c0fd160 msgr2=0x7fb22c0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:33:36.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.914+0000 7fb231e36640 1 -- 172.21.15.23:0/848455660 shutdown_connections 2024-03-28T05:33:36.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:36.914+0000 7fb231e36640 1 -- 172.21.15.23:0/848455660 wait complete. 2024-03-28T05:33:36.925 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-28T05:33:37.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.029+0000 7f56699b2640 1 -- 172.21.15.23:0/4247412273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5664106700 msgr2=0x7f5664106ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:37.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.029+0000 7f56699b2640 1 --2- 172.21.15.23:0/4247412273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5664106700 0x7f5664106ae0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f5654009900 tx=0x7f565402bc30 comp rx=0 tx=0).stop 2024-03-28T05:33:37.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.030+0000 7f56699b2640 1 -- 172.21.15.23:0/4247412273 shutdown_connections 2024-03-28T05:33:37.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.030+0000 7f56699b2640 1 --2- 172.21.15.23:0/4247412273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5664100f30 0x7f566410d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.030+0000 7f56699b2640 1 --2- 172.21.15.23:0/4247412273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5664100570 0x7f56641009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.030+0000 7f56699b2640 1 --2- 172.21.15.23:0/4247412273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5664106700 0x7f5664106ae0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.030+0000 7f56699b2640 1 -- 172.21.15.23:0/4247412273 >> 172.21.15.23:0/4247412273 conn(0x7f56640776d0 msgr2=0x7f5664077ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:33:37.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.030+0000 7f56699b2640 1 -- 172.21.15.23:0/4247412273 shutdown_connections 2024-03-28T05:33:37.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.030+0000 7f56699b2640 1 -- 172.21.15.23:0/4247412273 wait complete. 2024-03-28T05:33:37.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.030+0000 7f56699b2640 1 Processor -- start 2024-03-28T05:33:37.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.030+0000 7f56699b2640 1 -- start start 2024-03-28T05:33:37.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.030+0000 7f56699b2640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5664100570 0x7f5664198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:37.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.030+0000 7f56699b2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5664100f30 0x7f56641991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:37.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.030+0000 7f56699b2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5664106700 0x7f56641a3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:37.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.030+0000 7f56699b2640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56641109b0 con 0x7f5664106700 2024-03-28T05:33:37.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.030+0000 7f56699b2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5664110830 con 0x7f5664100f30 2024-03-28T05:33:37.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.030+0000 7f56699b2640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5664110b30 con 0x7f5664100570 2024-03-28T05:33:37.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.031+0000 7f56637fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5664106700 0x7f56641a3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:37.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.031+0000 7f5662ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5664100570 0x7f5664198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:37.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.031+0000 7f5662ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5664100570 0x7f5664198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:57264/0 (socket says 172.21.15.23:57264) 2024-03-28T05:33:37.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.031+0000 7f5662ffd640 1 -- 172.21.15.23:0/1554015558 learned_addr learned my addr 172.21.15.23:0/1554015558 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:33:37.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.031+0000 7f56637fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5664106700 0x7f56641a3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51820/0 (socket says 172.21.15.23:51820) 2024-03-28T05:33:37.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.031+0000 7f56627fc640 1 --2- 172.21.15.23:0/1554015558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5664100f30 0x7f56641991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:37.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.031+0000 7f5662ffd640 1 -- 172.21.15.23:0/1554015558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5664100f30 msgr2=0x7f56641991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:37.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.031+0000 7f5662ffd640 1 --2- 172.21.15.23:0/1554015558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5664100f30 0x7f56641991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.031+0000 7f5662ffd640 1 -- 172.21.15.23:0/1554015558 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5664106700 msgr2=0x7f56641a3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:37.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.031+0000 7f5662ffd640 1 --2- 172.21.15.23:0/1554015558 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5664106700 0x7f56641a3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.031+0000 7f5662ffd640 1 -- 172.21.15.23:0/1554015558 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56641a4230 con 0x7f5664100570 2024-03-28T05:33:37.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.031+0000 7f56627fc640 1 --2- 172.21.15.23:0/1554015558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5664100f30 0x7f56641991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:33:37.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.031+0000 7f5662ffd640 1 --2- 172.21.15.23:0/1554015558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5664100570 0x7f5664198c70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5654005940 tx=0x7f56540042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:33:37.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.031+0000 7f56689b0640 1 -- 172.21.15.23:0/1554015558 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5654041070 con 0x7f5664100570 2024-03-28T05:33:37.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.031+0000 7f56699b2640 1 -- 172.21.15.23:0/1554015558 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56641a4400 con 0x7f5664100570 2024-03-28T05:33:37.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.031+0000 7f56689b0640 1 -- 172.21.15.23:0/1554015558 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f565402ccc0 con 0x7f5664100570 2024-03-28T05:33:37.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.031+0000 7f56689b0640 1 -- 172.21.15.23:0/1554015558 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5654003d80 con 0x7f5664100570 2024-03-28T05:33:37.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.031+0000 7f56699b2640 1 -- 172.21.15.23:0/1554015558 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56641a4910 con 0x7f5664100570 2024-03-28T05:33:37.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.032+0000 7f56699b2640 1 -- 172.21.15.23:0/1554015558 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5664101cb0 con 0x7f5664100570 2024-03-28T05:33:37.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.033+0000 7f56689b0640 1 -- 172.21.15.23:0/1554015558 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f5654044070 con 0x7f5664100570 2024-03-28T05:33:37.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.034+0000 7f56689b0640 1 --2- 172.21.15.23:0/1554015558 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f564c0a2a10 0x7f564c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:37.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.034+0000 7f56627fc640 1 --2- 172.21.15.23:0/1554015558 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f564c0a2a10 0x7f564c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:37.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.034+0000 7f56689b0640 1 -- 172.21.15.23:0/1554015558 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f56540f0f40 con 0x7f5664100570 2024-03-28T05:33:37.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.034+0000 7f56627fc640 1 --2- 172.21.15.23:0/1554015558 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f564c0a2a10 0x7f564c0a4ed0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f566406aa50 tx=0x7f5658009500 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:33:37.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.041+0000 7f56689b0640 1 -- 172.21.15.23:0/1554015558 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f56540bd9c0 con 0x7f5664100570 2024-03-28T05:33:37.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.246+0000 7f56699b2640 1 -- 172.21.15.23:0/1554015558 --> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f5664061ae0 con 0x7f564c0a2a10 2024-03-28T05:33:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.253+0000 7f56689b0640 1 -- 172.21.15.23:0/1554015558 <== mgr.15402 v2:172.21.15.23:6800/1385031613 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f5664061ae0 con 0x7f564c0a2a10 2024-03-28T05:33:37.254 INFO:teuthology.orchestra.run.smithi023.stdout:/ 1m 6m3h 2024-03-28T05:33:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.255+0000 7f56699b2640 1 -- 172.21.15.23:0/1554015558 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f564c0a2a10 msgr2=0x7f564c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.255+0000 7f56699b2640 1 --2- 172.21.15.23:0/1554015558 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f564c0a2a10 0x7f564c0a4ed0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f566406aa50 tx=0x7f5658009500 comp rx=0 tx=0).stop 2024-03-28T05:33:37.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.255+0000 7f56699b2640 1 -- 172.21.15.23:0/1554015558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5664100570 msgr2=0x7f5664198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:37.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.255+0000 7f56699b2640 1 --2- 172.21.15.23:0/1554015558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5664100570 0x7f5664198c70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5654005940 tx=0x7f56540042f0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.255+0000 7f56699b2640 1 -- 172.21.15.23:0/1554015558 shutdown_connections 2024-03-28T05:33:37.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.255+0000 7f56699b2640 1 --2- 172.21.15.23:0/1554015558 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5664106700 0x7f56641a3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.255+0000 7f56699b2640 1 --2- 172.21.15.23:0/1554015558 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f564c0a2a10 0x7f564c0a4ed0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.255+0000 7f56699b2640 1 --2- 172.21.15.23:0/1554015558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5664100f30 0x7f56641991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.255+0000 7f56699b2640 1 --2- 172.21.15.23:0/1554015558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5664100570 0x7f5664198c70 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.255+0000 7f56699b2640 1 -- 172.21.15.23:0/1554015558 >> 172.21.15.23:0/1554015558 conn(0x7f56640776d0 msgr2=0x7f56640ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:33:37.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.255+0000 7f56699b2640 1 -- 172.21.15.23:0/1554015558 shutdown_connections 2024-03-28T05:33:37.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.255+0000 7f56699b2640 1 -- 172.21.15.23:0/1554015558 wait complete. 2024-03-28T05:33:37.269 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-28T05:33:37.281 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T05:33:37.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.383+0000 7f868c84b640 1 -- 172.21.15.23:0/935279478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86881076a0 msgr2=0x7f8688107a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:37.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.383+0000 7f868c84b640 1 --2- 172.21.15.23:0/935279478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86881076a0 0x7f8688107a80 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f8680009900 tx=0x7f868002bcd0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.384+0000 7f868c84b640 1 -- 172.21.15.23:0/935279478 shutdown_connections 2024-03-28T05:33:37.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.384+0000 7f868c84b640 1 --2- 172.21.15.23:0/935279478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f868806a730 0x7f868810e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.384+0000 7f868c84b640 1 --2- 172.21.15.23:0/935279478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8688069d90 0x7f868806a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.384+0000 7f868c84b640 1 --2- 172.21.15.23:0/935279478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86881076a0 0x7f8688107a80 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.384+0000 7f868c84b640 1 -- 172.21.15.23:0/935279478 >> 172.21.15.23:0/935279478 conn(0x7f8688077640 msgr2=0x7f8688077a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:33:37.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.384+0000 7f868c84b640 1 -- 172.21.15.23:0/935279478 shutdown_connections 2024-03-28T05:33:37.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.384+0000 7f868c84b640 1 -- 172.21.15.23:0/935279478 wait complete. 2024-03-28T05:33:37.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.384+0000 7f868c84b640 1 Processor -- start 2024-03-28T05:33:37.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.384+0000 7f868c84b640 1 -- start start 2024-03-28T05:33:37.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.385+0000 7f868c84b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8688069d90 0x7f86881005d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:37.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.385+0000 7f868c84b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f868806a730 0x7f8688100b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:37.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.385+0000 7f868c84b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86881076a0 0x7f8688101050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:37.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.385+0000 7f868c84b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8688111a60 con 0x7f86881076a0 2024-03-28T05:33:37.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.385+0000 7f868c84b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86881118e0 con 0x7f8688069d90 2024-03-28T05:33:37.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.385+0000 7f868c84b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8688111be0 con 0x7f868806a730 2024-03-28T05:33:37.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.385+0000 7f8686d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86881076a0 0x7f8688101050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:37.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.385+0000 7f8686d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86881076a0 0x7f8688101050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51826/0 (socket says 172.21.15.23:51826) 2024-03-28T05:33:37.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.385+0000 7f8686d9d640 1 -- 172.21.15.23:0/3780729298 learned_addr learned my addr 172.21.15.23:0/3780729298 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:33:37.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.385+0000 7f8685d9b640 1 --2- 172.21.15.23:0/3780729298 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f868806a730 0x7f8688100b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:37.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.385+0000 7f868659c640 1 --2- 172.21.15.23:0/3780729298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8688069d90 0x7f86881005d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:37.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.385+0000 7f8686d9d640 1 -- 172.21.15.23:0/3780729298 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f868806a730 msgr2=0x7f8688100b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:37.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.385+0000 7f8686d9d640 1 --2- 172.21.15.23:0/3780729298 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f868806a730 0x7f8688100b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.385+0000 7f8686d9d640 1 -- 172.21.15.23:0/3780729298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8688069d90 msgr2=0x7f86881005d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:37.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.385+0000 7f8686d9d640 1 --2- 172.21.15.23:0/3780729298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8688069d90 0x7f86881005d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.385+0000 7f8686d9d640 1 -- 172.21.15.23:0/3780729298 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86881a63c0 con 0x7f86881076a0 2024-03-28T05:33:37.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.385+0000 7f8686d9d640 1 --2- 172.21.15.23:0/3780729298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86881076a0 0x7f8688101050 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f867800eaa0 tx=0x7f867800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:33:37.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.385+0000 7f867f7fe640 1 -- 172.21.15.23:0/3780729298 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867800cdb0 con 0x7f86881076a0 2024-03-28T05:33:37.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.385+0000 7f867f7fe640 1 -- 172.21.15.23:0/3780729298 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8678005020 con 0x7f86881076a0 2024-03-28T05:33:37.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.385+0000 7f868c84b640 1 -- 172.21.15.23:0/3780729298 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86881a66b0 con 0x7f86881076a0 2024-03-28T05:33:37.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.385+0000 7f867f7fe640 1 -- 172.21.15.23:0/3780729298 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86780116b0 con 0x7f86881076a0 2024-03-28T05:33:37.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.385+0000 7f868c84b640 1 -- 172.21.15.23:0/3780729298 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86881a6bb0 con 0x7f86881076a0 2024-03-28T05:33:37.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.386+0000 7f868c84b640 1 -- 172.21.15.23:0/3780729298 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86880688e0 con 0x7f86881076a0 2024-03-28T05:33:37.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.388+0000 7f867f7fe640 1 -- 172.21.15.23:0/3780729298 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f8678005560 con 0x7f86881076a0 2024-03-28T05:33:37.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.388+0000 7f867f7fe640 1 --2- 172.21.15.23:0/3780729298 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f86740a2970 0x7f86740a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:37.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.388+0000 7f868659c640 1 --2- 172.21.15.23:0/3780729298 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f86740a2970 0x7f86740a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:37.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.388+0000 7f867f7fe640 1 -- 172.21.15.23:0/3780729298 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f86780cf840 con 0x7f86881076a0 2024-03-28T05:33:37.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.389+0000 7f868659c640 1 --2- 172.21.15.23:0/3780729298 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f86740a2970 0x7f86740a4e30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8680000c00 tx=0x7f868002c710 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:33:37.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.392+0000 7f867f7fe640 1 -- 172.21.15.23:0/3780729298 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8678016030 con 0x7f86881076a0 2024-03-28T05:33:37.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.578+0000 7f868c84b640 1 -- 172.21.15.23:0/3780729298 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f868802d070 con 0x7f86881076a0 2024-03-28T05:33:37.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.580+0000 7f867f7fe640 1 -- 172.21.15.23:0/3780729298 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+25254 (secure 0 0 0) 0x7f867809c1b0 con 0x7f86881076a0 2024-03-28T05:33:37.584 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:33:37.584 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":102,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","created":"2024-03-28T05:13:13.453653+0000","modified":"2024-03-28T05:33:28.817596+0000","last_up_change":"2024-03-28T05:20:20.221527+0000","last_in_change":"2024-03-28T05:20:00.365478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T05:16:32.986974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T05:21:06.103894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T05:21:06.827190+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T05:21:10.507418+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"18150942-59a4-4c3b-83d4-c1fc3ccf1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6803","nonce":1219956881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6805","nonce":1219956881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6809","nonce":1219956881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6807","nonce":1219956881}]},"public_addr":"172.21.15.23:6803/1219956881","cluster_addr":"172.21.15.23:6805/1219956881","heartbeat_back_addr":"172.21.15.23:6809/1219956881","heartbeat_front_addr":"172.21.15.23:6807/1219956881","state":["exists","up"]},{"osd":1,"uuid":"ba6d1a95-5135-4670-95f9-7878c0e7bd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6801","nonce":555491234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6803","nonce":555491234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6807","nonce":555491234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6805","nonce":555491234}]},"public_addr":"172.21.15.98:6801/555491234","cluster_addr":"172.21.15.98:6803/555491234","heartbeat_back_addr":"172.21.15.98:6807/555491234","heartbeat_front_addr":"172.21.15.98:6805/555491234","state":["exists","up"]},{"osd":2,"uuid":"873bdd60-2a47-403b-b1f4-ee9226a95067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6801","nonce":1535856914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6803","nonce":1535856914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6807","nonce":1535856914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6805","nonce":1535856914}]},"public_addr":"172.21.15.113:6801/1535856914","cluster_addr":"172.21.15.113:6803/1535856914","heartbeat_back_addr":"172.21.15.113:6807/1535856914","heartbeat_front_addr":"172.21.15.113:6805/1535856914","state":["exists","up"]},{"osd":3,"uuid":"83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6811","nonce":2949415020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6813","nonce":2949415020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6817","nonce":2949415020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6815","nonce":2949415020}]},"public_addr":"172.21.15.23:6811/2949415020","cluster_addr":"172.21.15.23:6813/2949415020","heartbeat_back_addr":"172.21.15.23:6817/2949415020","heartbeat_front_addr":"172.21.15.23:6815/2949415020","state":["exists","up"]},{"osd":4,"uuid":"76a90bac-b221-40a3-8238-40e8049cec36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6809","nonce":2615902819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6811","nonce":2615902819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6815","nonce":2615902819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6813","nonce":2615902819}]},"public_addr":"172.21.15.98:6809/2615902819","cluster_addr":"172.21.15.98:6811/2615902819","heartbeat_back_addr":"172.21.15.98:6815/2615902819","heartbeat_front_addr":"172.21.15.98:6813/2615902819","state":["exists","up"]},{"osd":5,"uuid":"b53fdb55-a8df-40a8-bd1b-8bcef634e26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6809","nonce":625271528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6811","nonce":625271528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6815","nonce":625271528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6813","nonce":625271528}]},"public_addr":"172.21.15.113:6809/625271528","cluster_addr":"172.21.15.113:6811/625271528","heartbeat_back_addr":"172.21.15.113:6815/625271528","heartbeat_front_addr":"172.21.15.113:6813/625271528","state":["exists","up"]},{"osd":6,"uuid":"f00283d2-8d67-4e97-a575-9a9d3b55d5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6819","nonce":4188464176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6821","nonce":4188464176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6825","nonce":4188464176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6823","nonce":4188464176}]},"public_addr":"172.21.15.23:6819/4188464176","cluster_addr":"172.21.15.23:6821/4188464176","heartbeat_back_addr":"172.21.15.23:6825/4188464176","heartbeat_front_addr":"172.21.15.23:6823/4188464176","state":["exists","up"]},{"osd":7,"uuid":"edc008a7-b07b-40a8-88b7-1614ba38a775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6817","nonce":1985546028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6819","nonce":1985546028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6823","nonce":1985546028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6821","nonce":1985546028}]},"public_addr":"172.21.15.98:6817/1985546028","cluster_addr":"172.21.15.98:6819/1985546028","heartbeat_back_addr":"172.21.15.98:6823/1985546028","heartbeat_front_addr":"172.21.15.98:6821/1985546028","state":["exists","up"]},{"osd":8,"uuid":"56e7b2d1-81f0-47b2-8638-f29832bed8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6817","nonce":3534876081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6819","nonce":3534876081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6823","nonce":3534876081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6821","nonce":3534876081}]},"public_addr":"172.21.15.113:6817/3534876081","cluster_addr":"172.21.15.113:6819/3534876081","heartbeat_back_addr":"172.21.15.113:6823/3534876081","heartbeat_front_addr":"172.21.15.113:6821/3534876081","state":["exists","up"]},{"osd":9,"uuid":"a2e90caa-6e9c-42bf-9f18-81f7f47880cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6827","nonce":672264207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6829","nonce":672264207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6833","nonce":672264207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6831","nonce":672264207}]},"public_addr":"172.21.15.23:6827/672264207","cluster_addr":"172.21.15.23:6829/672264207","heartbeat_back_addr":"172.21.15.23:6833/672264207","heartbeat_front_addr":"172.21.15.23:6831/672264207","state":["exists","up"]},{"osd":10,"uuid":"0eda917f-020c-437f-95ae-b26203f6c68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6825","nonce":2699920595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6827","nonce":2699920595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6831","nonce":2699920595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6829","nonce":2699920595}]},"public_addr":"172.21.15.98:6825/2699920595","cluster_addr":"172.21.15.98:6827/2699920595","heartbeat_back_addr":"172.21.15.98:6831/2699920595","heartbeat_front_addr":"172.21.15.98:6829/2699920595","state":["exists","up"]},{"osd":11,"uuid":"aae78501-78a8-469a-be20-3006f5ec5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6825","nonce":2113644020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6827","nonce":2113644020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6831","nonce":2113644020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6829","nonce":2113644020}]},"public_addr":"172.21.15.113:6825/2113644020","cluster_addr":"172.21.15.113:6827/2113644020","heartbeat_back_addr":"172.21.15.113:6831/2113644020","heartbeat_front_addr":"172.21.15.113:6829/2113644020","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:15:43.527475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:07.006538+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:30.594176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:55.810999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:20.595811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:45.474138+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:10.500409+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:35.787638+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:01.001188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:26.761115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:52.952752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:20:18.559832+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":7}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2772113246":"2024-03-29T05:23:33.950262+0000","172.21.15.23:0/84692814":"2024-03-29T05:23:33.950262+0000","172.21.15.23:6801/222178108":"2024-03-29T05:23:33.950262+0000","172.21.15.23:6801/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:6800/222178108":"2024-03-29T05:23:33.950262+0000","172.21.15.23:6800/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1876451381":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2253559862":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2653562762":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/3337463830":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/944304909":"2024-03-29T05:23:33.950262+0000","172.21.15.23:6800/1735221490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/2784771635":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1135235867":"2024-03-29T05:23:33.950262+0000","172.21.15.23:0/2389902490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6801/1735221490":"2024-03-29T05:13:42.579607+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T05:33:37.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.582+0000 7f868c84b640 1 -- 172.21.15.23:0/3780729298 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f86740a2970 msgr2=0x7f86740a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:37.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.582+0000 7f868c84b640 1 --2- 172.21.15.23:0/3780729298 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f86740a2970 0x7f86740a4e30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8680000c00 tx=0x7f868002c710 comp rx=0 tx=0).stop 2024-03-28T05:33:37.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.582+0000 7f868c84b640 1 -- 172.21.15.23:0/3780729298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86881076a0 msgr2=0x7f8688101050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:37.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.582+0000 7f868c84b640 1 --2- 172.21.15.23:0/3780729298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86881076a0 0x7f8688101050 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f867800eaa0 tx=0x7f867800ef60 comp rx=0 tx=0).stop 2024-03-28T05:33:37.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.582+0000 7f868c84b640 1 -- 172.21.15.23:0/3780729298 shutdown_connections 2024-03-28T05:33:37.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.582+0000 7f868c84b640 1 --2- 172.21.15.23:0/3780729298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86881076a0 0x7f8688101050 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.582+0000 7f868c84b640 1 --2- 172.21.15.23:0/3780729298 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f868806a730 0x7f8688100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.582+0000 7f868c84b640 1 --2- 172.21.15.23:0/3780729298 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f86740a2970 0x7f86740a4e30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.582+0000 7f868c84b640 1 --2- 172.21.15.23:0/3780729298 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8688069d90 0x7f86881005d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.582+0000 7f868c84b640 1 -- 172.21.15.23:0/3780729298 >> 172.21.15.23:0/3780729298 conn(0x7f8688077640 msgr2=0x7f86880ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:33:37.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.582+0000 7f868c84b640 1 -- 172.21.15.23:0/3780729298 shutdown_connections 2024-03-28T05:33:37.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.582+0000 7f868c84b640 1 -- 172.21.15.23:0/3780729298 wait complete. 2024-03-28T05:33:37.594 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T05:16:32.986974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T05:21:06.103894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T05:21:06.827190+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T05:21:10.507418+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T05:33:37.595 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T05:33:37.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.700+0000 7f878759e640 1 -- 172.21.15.23:0/464446872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f878810bb90 msgr2=0x7f878810e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:37.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.700+0000 7f878759e640 1 --2- 172.21.15.23:0/464446872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f878810bb90 0x7f878810e010 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f8780009ca0 tx=0x7f878002c180 comp rx=0 tx=0).stop 2024-03-28T05:33:37.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.700+0000 7f878759e640 1 -- 172.21.15.23:0/464446872 shutdown_connections 2024-03-28T05:33:37.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.700+0000 7f878759e640 1 --2- 172.21.15.23:0/464446872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f878810bb90 0x7f878810e010 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.700+0000 7f878759e640 1 --2- 172.21.15.23:0/464446872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87881030e0 0x7f878810b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.700+0000 7f878759e640 1 --2- 172.21.15.23:0/464446872 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f87881027c0 0x7f8788102ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.700+0000 7f878759e640 1 -- 172.21.15.23:0/464446872 >> 172.21.15.23:0/464446872 conn(0x7f87880fbfa0 msgr2=0x7f87880fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:33:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.700+0000 7f878759e640 1 -- 172.21.15.23:0/464446872 shutdown_connections 2024-03-28T05:33:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.700+0000 7f878759e640 1 -- 172.21.15.23:0/464446872 wait complete. 2024-03-28T05:33:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.701+0000 7f878759e640 1 Processor -- start 2024-03-28T05:33:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.701+0000 7f878759e640 1 -- start start 2024-03-28T05:33:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.701+0000 7f878759e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87881027c0 0x7f8788197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.701+0000 7f878759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87881030e0 0x7f87881983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.701+0000 7f878759e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f878810bb90 0x7f878819c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.701+0000 7f878759e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87881119b0 con 0x7f87881027c0 2024-03-28T05:33:37.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.701+0000 7f878759e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8788111830 con 0x7f87881030e0 2024-03-28T05:33:37.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.701+0000 7f878759e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8788111b30 con 0x7f878810bb90 2024-03-28T05:33:37.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.701+0000 7f878659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87881027c0 0x7f8788197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:37.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.701+0000 7f878659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87881027c0 0x7f8788197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51838/0 (socket says 172.21.15.23:51838) 2024-03-28T05:33:37.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.701+0000 7f878659c640 1 -- 172.21.15.23:0/3825872786 learned_addr learned my addr 172.21.15.23:0/3825872786 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:33:37.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.701+0000 7f8786d9d640 1 --2- 172.21.15.23:0/3825872786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f878810bb90 0x7f878819c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:37.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.701+0000 7f877ffff640 1 --2- 172.21.15.23:0/3825872786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87881030e0 0x7f87881983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:37.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.701+0000 7f878659c640 1 -- 172.21.15.23:0/3825872786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f878810bb90 msgr2=0x7f878819c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:37.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.701+0000 7f878659c640 1 --2- 172.21.15.23:0/3825872786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f878810bb90 0x7f878819c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.701+0000 7f878659c640 1 -- 172.21.15.23:0/3825872786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87881030e0 msgr2=0x7f87881983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:37.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.701+0000 7f878659c640 1 --2- 172.21.15.23:0/3825872786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87881030e0 0x7f87881983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.701+0000 7f878659c640 1 -- 172.21.15.23:0/3825872786 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878819ce50 con 0x7f87881027c0 2024-03-28T05:33:37.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.702+0000 7f8786d9d640 1 --2- 172.21.15.23:0/3825872786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f878810bb90 0x7f878819c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:33:37.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.702+0000 7f878659c640 1 --2- 172.21.15.23:0/3825872786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87881027c0 0x7f8788197eb0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f877800e9f0 tx=0x7f877800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:33:37.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.702+0000 7f877f7fe640 1 -- 172.21.15.23:0/3825872786 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877800cd80 con 0x7f87881027c0 2024-03-28T05:33:37.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.702+0000 7f878759e640 1 -- 172.21.15.23:0/3825872786 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f878819d140 con 0x7f87881027c0 2024-03-28T05:33:37.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.702+0000 7f877f7fe640 1 -- 172.21.15.23:0/3825872786 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8778010910 con 0x7f87881027c0 2024-03-28T05:33:37.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.702+0000 7f877f7fe640 1 -- 172.21.15.23:0/3825872786 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87780188f0 con 0x7f87881027c0 2024-03-28T05:33:37.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.702+0000 7f878759e640 1 -- 172.21.15.23:0/3825872786 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87881a49d0 con 0x7f87881027c0 2024-03-28T05:33:37.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.703+0000 7f878759e640 1 -- 172.21.15.23:0/3825872786 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8788068800 con 0x7f87881027c0 2024-03-28T05:33:37.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.704+0000 7f877f7fe640 1 -- 172.21.15.23:0/3825872786 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f8778010ac0 con 0x7f87881027c0 2024-03-28T05:33:37.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.704+0000 7f877f7fe640 1 --2- 172.21.15.23:0/3825872786 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f87740a2970 0x7f87740a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:37.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.704+0000 7f877f7fe640 1 -- 172.21.15.23:0/3825872786 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f87780cf6b0 con 0x7f87881027c0 2024-03-28T05:33:37.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.704+0000 7f877ffff640 1 --2- 172.21.15.23:0/3825872786 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f87740a2970 0x7f87740a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:37.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.705+0000 7f877ffff640 1 --2- 172.21.15.23:0/3825872786 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f87740a2970 0x7f87740a4e30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f87880fe5c0 tx=0x7f8770008040 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:33:37.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.709+0000 7f877f7fe640 1 -- 172.21.15.23:0/3825872786 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f877809c020 con 0x7f87881027c0 2024-03-28T05:33:37.842 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:37 smithi023 ceph-mon[31954]: pgmap v312: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:33:37.842 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:37 smithi023 ceph-mon[31954]: from='client.16095 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:33:37.842 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:37 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3780729298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:33:37.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.894+0000 7f878759e640 1 -- 172.21.15.23:0/3825872786 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8788106c80 con 0x7f87881027c0 2024-03-28T05:33:37.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.895+0000 7f877f7fe640 1 -- 172.21.15.23:0/3825872786 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) v1 ==== 93+0+10 (secure 0 0 0) 0x7f877809b920 con 0x7f87881027c0 2024-03-28T05:33:37.896 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-28T05:33:37.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.897+0000 7f878759e640 1 -- 172.21.15.23:0/3825872786 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f87740a2970 msgr2=0x7f87740a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:37.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.897+0000 7f878759e640 1 --2- 172.21.15.23:0/3825872786 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f87740a2970 0x7f87740a4e30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f87880fe5c0 tx=0x7f8770008040 comp rx=0 tx=0).stop 2024-03-28T05:33:37.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.897+0000 7f878759e640 1 -- 172.21.15.23:0/3825872786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87881027c0 msgr2=0x7f8788197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:37.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.897+0000 7f878759e640 1 --2- 172.21.15.23:0/3825872786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87881027c0 0x7f8788197eb0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f877800e9f0 tx=0x7f877800eeb0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.897+0000 7f878759e640 1 -- 172.21.15.23:0/3825872786 shutdown_connections 2024-03-28T05:33:37.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.897+0000 7f878759e640 1 --2- 172.21.15.23:0/3825872786 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f878810bb90 0x7f878819c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.897+0000 7f878759e640 1 --2- 172.21.15.23:0/3825872786 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f87740a2970 0x7f87740a4e30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.897+0000 7f878759e640 1 --2- 172.21.15.23:0/3825872786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87881030e0 0x7f87881983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.897+0000 7f878759e640 1 --2- 172.21.15.23:0/3825872786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f87881027c0 0x7f8788197eb0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:37.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.897+0000 7f878759e640 1 -- 172.21.15.23:0/3825872786 >> 172.21.15.23:0/3825872786 conn(0x7f87880fbfa0 msgr2=0x7f87880fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:33:37.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.897+0000 7f878759e640 1 -- 172.21.15.23:0/3825872786 shutdown_connections 2024-03-28T05:33:37.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:37.897+0000 7f878759e640 1 -- 172.21.15.23:0/3825872786 wait complete. 2024-03-28T05:33:37.909 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T05:33:38.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.013+0000 7f6ef771d640 1 -- 172.21.15.23:0/2380550144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef0103230 msgr2=0x7f6ef010b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:38.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.013+0000 7f6ef771d640 1 --2- 172.21.15.23:0/2380550144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef0103230 0x7f6ef010b710 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0009900 tx=0x7f6ee002bcd0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.013+0000 7f6ef771d640 1 -- 172.21.15.23:0/2380550144 shutdown_connections 2024-03-28T05:33:38.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.013+0000 7f6ef771d640 1 --2- 172.21.15.23:0/2380550144 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ef010bce0 0x7f6ef010e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.013+0000 7f6ef771d640 1 --2- 172.21.15.23:0/2380550144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef0103230 0x7f6ef010b710 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.013+0000 7f6ef771d640 1 --2- 172.21.15.23:0/2380550144 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ef0102910 0x7f6ef0102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.013+0000 7f6ef771d640 1 -- 172.21.15.23:0/2380550144 >> 172.21.15.23:0/2380550144 conn(0x7f6ef0077700 msgr2=0x7f6ef0077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:33:38.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.013+0000 7f6ef771d640 1 -- 172.21.15.23:0/2380550144 shutdown_connections 2024-03-28T05:33:38.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.013+0000 7f6ef771d640 1 -- 172.21.15.23:0/2380550144 wait complete. 2024-03-28T05:33:38.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.014+0000 7f6ef771d640 1 Processor -- start 2024-03-28T05:33:38.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.014+0000 7f6ef771d640 1 -- start start 2024-03-28T05:33:38.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.014+0000 7f6ef771d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ef0102910 0x7f6ef0198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:38.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.014+0000 7f6ef771d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ef0103230 0x7f6ef0198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:38.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.014+0000 7f6ef771d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef010bce0 0x7f6ef019c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:38.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.014+0000 7f6ef771d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef0111b10 con 0x7f6ef010bce0 2024-03-28T05:33:38.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.014+0000 7f6ef771d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef0111990 con 0x7f6ef0102910 2024-03-28T05:33:38.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.014+0000 7f6ef771d640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef0111c90 con 0x7f6ef0103230 2024-03-28T05:33:38.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.014+0000 7f6ef5cba640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef010bce0 0x7f6ef019c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:38.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.014+0000 7f6ef5cba640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef010bce0 0x7f6ef019c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51870/0 (socket says 172.21.15.23:51870) 2024-03-28T05:33:38.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.014+0000 7f6ef5cba640 1 -- 172.21.15.23:0/2104701992 learned_addr learned my addr 172.21.15.23:0/2104701992 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:33:38.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.014+0000 7f6ef4cb8640 1 --2- 172.21.15.23:0/2104701992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ef0103230 0x7f6ef0198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:38.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.014+0000 7f6ef54b9640 1 --2- 172.21.15.23:0/2104701992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ef0102910 0x7f6ef0198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:38.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.014+0000 7f6ef5cba640 1 -- 172.21.15.23:0/2104701992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ef0103230 msgr2=0x7f6ef0198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:38.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.014+0000 7f6ef5cba640 1 --2- 172.21.15.23:0/2104701992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ef0103230 0x7f6ef0198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.014+0000 7f6ef5cba640 1 -- 172.21.15.23:0/2104701992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ef0102910 msgr2=0x7f6ef0198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:38.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.014+0000 7f6ef5cba640 1 --2- 172.21.15.23:0/2104701992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ef0102910 0x7f6ef0198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.014+0000 7f6ef5cba640 1 -- 172.21.15.23:0/2104701992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef019cff0 con 0x7f6ef010bce0 2024-03-28T05:33:38.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.014+0000 7f6ef54b9640 1 --2- 172.21.15.23:0/2104701992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ef0102910 0x7f6ef0198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:33:38.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.015+0000 7f6ef5cba640 1 --2- 172.21.15.23:0/2104701992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef010bce0 0x7f6ef019c8c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f6ee800b970 tx=0x7f6ee800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:33:38.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.015+0000 7f6ee67fc640 1 -- 172.21.15.23:0/2104701992 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee8019070 con 0x7f6ef010bce0 2024-03-28T05:33:38.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.015+0000 7f6ee67fc640 1 -- 172.21.15.23:0/2104701992 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6ee800cb40 con 0x7f6ef010bce0 2024-03-28T05:33:38.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.015+0000 7f6ee67fc640 1 -- 172.21.15.23:0/2104701992 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee8007650 con 0x7f6ef010bce0 2024-03-28T05:33:38.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.015+0000 7f6ef771d640 1 -- 172.21.15.23:0/2104701992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ef019d2e0 con 0x7f6ef010bce0 2024-03-28T05:33:38.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.015+0000 7f6ef771d640 1 -- 172.21.15.23:0/2104701992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ef01a4b70 con 0x7f6ef010bce0 2024-03-28T05:33:38.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.016+0000 7f6ef771d640 1 -- 172.21.15.23:0/2104701992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ef00688e0 con 0x7f6ef010bce0 2024-03-28T05:33:38.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.017+0000 7f6ee67fc640 1 -- 172.21.15.23:0/2104701992 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f6ee801c070 con 0x7f6ef010bce0 2024-03-28T05:33:38.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.017+0000 7f6ee67fc640 1 --2- 172.21.15.23:0/2104701992 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f6edc0a2970 0x7f6edc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:38.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.017+0000 7f6ef54b9640 1 --2- 172.21.15.23:0/2104701992 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f6edc0a2970 0x7f6edc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:38.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.017+0000 7f6ee67fc640 1 -- 172.21.15.23:0/2104701992 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f6ee80ce6f0 con 0x7f6ef010bce0 2024-03-28T05:33:38.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.018+0000 7f6ef54b9640 1 --2- 172.21.15.23:0/2104701992 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f6edc0a2970 0x7f6edc0a4e30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6eec009c20 tx=0x7f6eec005f90 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:33:38.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.022+0000 7f6ee67fc640 1 -- 172.21.15.23:0/2104701992 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6ee809ab20 con 0x7f6ef010bce0 2024-03-28T05:33:38.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:37 smithi098 ceph-mon[36087]: pgmap v312: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:33:38.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:37 smithi098 ceph-mon[36087]: from='client.16095 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:33:38.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:37 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3780729298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:33:38.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.203+0000 7f6ef771d640 1 -- 172.21.15.23:0/2104701992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6ef0106e30 con 0x7f6ef010bce0 2024-03-28T05:33:38.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.204+0000 7f6ee67fc640 1 -- 172.21.15.23:0/2104701992 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v102) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6ee809ebc0 con 0x7f6ef010bce0 2024-03-28T05:33:38.205 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T05:33:38.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.206+0000 7f6ef771d640 1 -- 172.21.15.23:0/2104701992 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f6edc0a2970 msgr2=0x7f6edc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:38.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.206+0000 7f6ef771d640 1 --2- 172.21.15.23:0/2104701992 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f6edc0a2970 0x7f6edc0a4e30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6eec009c20 tx=0x7f6eec005f90 comp rx=0 tx=0).stop 2024-03-28T05:33:38.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.206+0000 7f6ef771d640 1 -- 172.21.15.23:0/2104701992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef010bce0 msgr2=0x7f6ef019c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:38.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.206+0000 7f6ef771d640 1 --2- 172.21.15.23:0/2104701992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef010bce0 0x7f6ef019c8c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f6ee800b970 tx=0x7f6ee800be30 comp rx=0 tx=0).stop 2024-03-28T05:33:38.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.206+0000 7f6ef771d640 1 -- 172.21.15.23:0/2104701992 shutdown_connections 2024-03-28T05:33:38.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.206+0000 7f6ef771d640 1 --2- 172.21.15.23:0/2104701992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ef010bce0 0x7f6ef019c8c0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.206+0000 7f6ef771d640 1 --2- 172.21.15.23:0/2104701992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ef0103230 0x7f6ef0198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.206+0000 7f6ef771d640 1 --2- 172.21.15.23:0/2104701992 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f6edc0a2970 0x7f6edc0a4e30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.206+0000 7f6ef771d640 1 --2- 172.21.15.23:0/2104701992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ef0102910 0x7f6ef0198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.206+0000 7f6ef771d640 1 -- 172.21.15.23:0/2104701992 >> 172.21.15.23:0/2104701992 conn(0x7f6ef0077700 msgr2=0x7f6ef00fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:33:38.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.206+0000 7f6ef771d640 1 -- 172.21.15.23:0/2104701992 shutdown_connections 2024-03-28T05:33:38.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.206+0000 7f6ef771d640 1 -- 172.21.15.23:0/2104701992 wait complete. 2024-03-28T05:33:38.217 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T05:33:38.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:37 smithi113 ceph-mon[37164]: pgmap v312: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:33:38.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:37 smithi113 ceph-mon[37164]: from='client.16095 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:33:38.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:37 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3780729298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:33:38.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.323+0000 7fef2cebf640 1 -- 172.21.15.23:0/3821689902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef281065f0 msgr2=0x7fef281069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:38.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.323+0000 7fef2cebf640 1 --2- 172.21.15.23:0/3821689902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef281065f0 0x7fef281069d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fef18009890 tx=0x7fef1802bb70 comp rx=0 tx=0).stop 2024-03-28T05:33:38.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.323+0000 7fef2cebf640 1 -- 172.21.15.23:0/3821689902 shutdown_connections 2024-03-28T05:33:38.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.323+0000 7fef2cebf640 1 --2- 172.21.15.23:0/3821689902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef28100ef0 0x7fef2810d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.323+0000 7fef2cebf640 1 --2- 172.21.15.23:0/3821689902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef28100530 0x7fef281009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.323+0000 7fef2cebf640 1 --2- 172.21.15.23:0/3821689902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef281065f0 0x7fef281069d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.323+0000 7fef2cebf640 1 -- 172.21.15.23:0/3821689902 >> 172.21.15.23:0/3821689902 conn(0x7fef280776a0 msgr2=0x7fef28077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:33:38.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.323+0000 7fef2cebf640 1 -- 172.21.15.23:0/3821689902 shutdown_connections 2024-03-28T05:33:38.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.323+0000 7fef2cebf640 1 -- 172.21.15.23:0/3821689902 wait complete. 2024-03-28T05:33:38.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.323+0000 7fef2cebf640 1 Processor -- start 2024-03-28T05:33:38.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.323+0000 7fef2cebf640 1 -- start start 2024-03-28T05:33:38.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.324+0000 7fef2cebf640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef28100530 0x7fef28197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:38.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.324+0000 7fef2cebf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef28100ef0 0x7fef28198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:38.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.324+0000 7fef2cebf640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef281065f0 0x7fef2819c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:38.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.324+0000 7fef2cebf640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef281108d0 con 0x7fef28100530 2024-03-28T05:33:38.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.324+0000 7fef2cebf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef28110750 con 0x7fef28100ef0 2024-03-28T05:33:38.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.324+0000 7fef2cebf640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef28110a50 con 0x7fef281065f0 2024-03-28T05:33:38.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.324+0000 7fef2659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef28100530 0x7fef28197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:38.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.324+0000 7fef2659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef28100530 0x7fef28197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51900/0 (socket says 172.21.15.23:51900) 2024-03-28T05:33:38.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.324+0000 7fef2659c640 1 -- 172.21.15.23:0/3523010930 learned_addr learned my addr 172.21.15.23:0/3523010930 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:33:38.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.324+0000 7fef26d9d640 1 --2- 172.21.15.23:0/3523010930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef281065f0 0x7fef2819c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:38.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.324+0000 7fef2659c640 1 -- 172.21.15.23:0/3523010930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef281065f0 msgr2=0x7fef2819c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:38.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.324+0000 7fef2659c640 1 --2- 172.21.15.23:0/3523010930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef281065f0 0x7fef2819c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.324+0000 7fef2659c640 1 -- 172.21.15.23:0/3523010930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef28100ef0 msgr2=0x7fef28198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:38.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.324+0000 7fef2659c640 1 --2- 172.21.15.23:0/3523010930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef28100ef0 0x7fef28198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.324+0000 7fef2659c640 1 -- 172.21.15.23:0/3523010930 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2819cf00 con 0x7fef28100530 2024-03-28T05:33:38.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.324+0000 7fef2659c640 1 --2- 172.21.15.23:0/3523010930 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef28100530 0x7fef28197f10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fef180043c0 tx=0x7fef180043f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:33:38.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.324+0000 7fef177fe640 1 -- 172.21.15.23:0/3523010930 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef18041020 con 0x7fef28100530 2024-03-28T05:33:38.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.324+0000 7fef2cebf640 1 -- 172.21.15.23:0/3523010930 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef2819d190 con 0x7fef28100530 2024-03-28T05:33:38.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.324+0000 7fef177fe640 1 -- 172.21.15.23:0/3523010930 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fef1802c9f0 con 0x7fef28100530 2024-03-28T05:33:38.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.324+0000 7fef2cebf640 1 -- 172.21.15.23:0/3523010930 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef281a4a80 con 0x7fef28100530 2024-03-28T05:33:38.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.324+0000 7fef177fe640 1 -- 172.21.15.23:0/3523010930 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef18039040 con 0x7fef28100530 2024-03-28T05:33:38.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.325+0000 7fef2cebf640 1 -- 172.21.15.23:0/3523010930 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef28068490 con 0x7fef28100530 2024-03-28T05:33:38.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.327+0000 7fef177fe640 1 -- 172.21.15.23:0/3523010930 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fef18044020 con 0x7fef28100530 2024-03-28T05:33:38.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.327+0000 7fef177fe640 1 --2- 172.21.15.23:0/3523010930 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fef100a2970 0x7fef100a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:38.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.327+0000 7fef25d9b640 1 --2- 172.21.15.23:0/3523010930 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fef100a2970 0x7fef100a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:38.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.327+0000 7fef177fe640 1 -- 172.21.15.23:0/3523010930 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fef180f0ef0 con 0x7fef28100530 2024-03-28T05:33:38.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.328+0000 7fef25d9b640 1 --2- 172.21.15.23:0/3523010930 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fef100a2970 0x7fef100a4e30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fef1c005ab0 tx=0x7fef1c005a20 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:33:38.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.331+0000 7fef177fe640 1 -- 172.21.15.23:0/3523010930 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fef1804b020 con 0x7fef28100530 2024-03-28T05:33:38.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.514+0000 7fef2cebf640 1 -- 172.21.15.23:0/3523010930 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fef281a4cb0 con 0x7fef28100530 2024-03-28T05:33:38.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.514+0000 7fef177fe640 1 -- 172.21.15.23:0/3523010930 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v102) v1 ==== 100+0+11 (secure 0 0 0) 0x7fef180bd970 con 0x7fef28100530 2024-03-28T05:33:38.515 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-28T05:33:38.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.516+0000 7fef2cebf640 1 -- 172.21.15.23:0/3523010930 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fef100a2970 msgr2=0x7fef100a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:38.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.516+0000 7fef2cebf640 1 --2- 172.21.15.23:0/3523010930 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fef100a2970 0x7fef100a4e30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fef1c005ab0 tx=0x7fef1c005a20 comp rx=0 tx=0).stop 2024-03-28T05:33:38.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.516+0000 7fef2cebf640 1 -- 172.21.15.23:0/3523010930 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef28100530 msgr2=0x7fef28197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:38.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.516+0000 7fef2cebf640 1 --2- 172.21.15.23:0/3523010930 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef28100530 0x7fef28197f10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fef180043c0 tx=0x7fef180043f0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.516+0000 7fef2cebf640 1 -- 172.21.15.23:0/3523010930 shutdown_connections 2024-03-28T05:33:38.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.516+0000 7fef2cebf640 1 --2- 172.21.15.23:0/3523010930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef281065f0 0x7fef2819c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.516+0000 7fef2cebf640 1 --2- 172.21.15.23:0/3523010930 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fef100a2970 0x7fef100a4e30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.516+0000 7fef2cebf640 1 --2- 172.21.15.23:0/3523010930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef28100ef0 0x7fef28198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.516+0000 7fef2cebf640 1 --2- 172.21.15.23:0/3523010930 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef28100530 0x7fef28197f10 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.516+0000 7fef2cebf640 1 -- 172.21.15.23:0/3523010930 >> 172.21.15.23:0/3523010930 conn(0x7fef280776a0 msgr2=0x7fef28104740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:33:38.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.516+0000 7fef2cebf640 1 -- 172.21.15.23:0/3523010930 shutdown_connections 2024-03-28T05:33:38.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.516+0000 7fef2cebf640 1 -- 172.21.15.23:0/3523010930 wait complete. 2024-03-28T05:33:38.528 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T05:33:38.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.633+0000 7f269b59e640 1 -- 172.21.15.23:0/2441175477 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f269c100400 msgr2=0x7f269c100880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:38.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.633+0000 7f269b59e640 1 --2- 172.21.15.23:0/2441175477 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f269c100400 0x7f269c100880 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2684009890 tx=0x7f268402bb70 comp rx=0 tx=0).stop 2024-03-28T05:33:38.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.633+0000 7f269b59e640 1 -- 172.21.15.23:0/2441175477 shutdown_connections 2024-03-28T05:33:38.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.633+0000 7f269b59e640 1 --2- 172.21.15.23:0/2441175477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f269c100dc0 0x7f269c10d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.633+0000 7f269b59e640 1 --2- 172.21.15.23:0/2441175477 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f269c100400 0x7f269c100880 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.633+0000 7f269b59e640 1 --2- 172.21.15.23:0/2441175477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f269c1064a0 0x7f269c106880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.633+0000 7f269b59e640 1 -- 172.21.15.23:0/2441175477 >> 172.21.15.23:0/2441175477 conn(0x7f269c0fbef0 msgr2=0x7f269c0fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:33:38.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.633+0000 7f269b59e640 1 -- 172.21.15.23:0/2441175477 shutdown_connections 2024-03-28T05:33:38.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.633+0000 7f269b59e640 1 -- 172.21.15.23:0/2441175477 wait complete. 2024-03-28T05:33:38.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.634+0000 7f269b59e640 1 Processor -- start 2024-03-28T05:33:38.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.634+0000 7f269b59e640 1 -- start start 2024-03-28T05:33:38.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.634+0000 7f269b59e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f269c100400 0x7f269c197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:38.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.634+0000 7f269b59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f269c100dc0 0x7f269c1982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:38.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.634+0000 7f269b59e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f269c1064a0 0x7f269c19c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:38.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.634+0000 7f269b59e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c1107b0 con 0x7f269c100400 2024-03-28T05:33:38.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.634+0000 7f269b59e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c110630 con 0x7f269c100dc0 2024-03-28T05:33:38.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.634+0000 7f269b59e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c110930 con 0x7f269c1064a0 2024-03-28T05:33:38.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.634+0000 7f269a59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f269c100400 0x7f269c197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:38.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.634+0000 7f269a59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f269c100400 0x7f269c197d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51906/0 (socket says 172.21.15.23:51906) 2024-03-28T05:33:38.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.634+0000 7f269a59c640 1 -- 172.21.15.23:0/3981627898 learned_addr learned my addr 172.21.15.23:0/3981627898 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:33:38.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.634+0000 7f269ad9d640 1 --2- 172.21.15.23:0/3981627898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f269c1064a0 0x7f269c19c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:38.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.634+0000 7f2693fff640 1 --2- 172.21.15.23:0/3981627898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f269c100dc0 0x7f269c1982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:38.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.635+0000 7f269a59c640 1 -- 172.21.15.23:0/3981627898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f269c1064a0 msgr2=0x7f269c19c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:38.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.635+0000 7f269a59c640 1 --2- 172.21.15.23:0/3981627898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f269c1064a0 0x7f269c19c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.635+0000 7f269a59c640 1 -- 172.21.15.23:0/3981627898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f269c100dc0 msgr2=0x7f269c1982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:38.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.635+0000 7f269a59c640 1 --2- 172.21.15.23:0/3981627898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f269c100dc0 0x7f269c1982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.635+0000 7f269a59c640 1 -- 172.21.15.23:0/3981627898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f269c19cd60 con 0x7f269c100400 2024-03-28T05:33:38.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.635+0000 7f2693fff640 1 --2- 172.21.15.23:0/3981627898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f269c100dc0 0x7f269c1982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:33:38.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.635+0000 7f269a59c640 1 --2- 172.21.15.23:0/3981627898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f269c100400 0x7f269c197d70 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f268c00c920 tx=0x7f268c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:33:38.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.635+0000 7f26937fe640 1 -- 172.21.15.23:0/3981627898 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c007c70 con 0x7f269c100400 2024-03-28T05:33:38.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.635+0000 7f269b59e640 1 -- 172.21.15.23:0/3981627898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f269c19d050 con 0x7f269c100400 2024-03-28T05:33:38.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.635+0000 7f26937fe640 1 -- 172.21.15.23:0/3981627898 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f268c007e10 con 0x7f269c100400 2024-03-28T05:33:38.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.635+0000 7f26937fe640 1 -- 172.21.15.23:0/3981627898 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c018890 con 0x7f269c100400 2024-03-28T05:33:38.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.635+0000 7f269b59e640 1 -- 172.21.15.23:0/3981627898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f269c1a48e0 con 0x7f269c100400 2024-03-28T05:33:38.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.636+0000 7f269b59e640 1 -- 172.21.15.23:0/3981627898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f269c068800 con 0x7f269c100400 2024-03-28T05:33:38.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.637+0000 7f26937fe640 1 -- 172.21.15.23:0/3981627898 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f268c018a50 con 0x7f269c100400 2024-03-28T05:33:38.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.638+0000 7f26937fe640 1 --2- 172.21.15.23:0/3981627898 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f26880a2970 0x7f26880a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:38.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.638+0000 7f2693fff640 1 --2- 172.21.15.23:0/3981627898 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f26880a2970 0x7f26880a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:38.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.638+0000 7f26937fe640 1 -- 172.21.15.23:0/3981627898 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f268c0cf5d0 con 0x7f269c100400 2024-03-28T05:33:38.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.638+0000 7f2693fff640 1 --2- 172.21.15.23:0/3981627898 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f26880a2970 0x7f26880a4e30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2684009860 tx=0x7f26840058b0 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:33:38.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.641+0000 7f26937fe640 1 -- 172.21.15.23:0/3981627898 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f268c014030 con 0x7f269c100400 2024-03-28T05:33:38.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.825+0000 7f269b59e640 1 -- 172.21.15.23:0/3981627898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f269c1a4b10 con 0x7f269c100400 2024-03-28T05:33:38.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.825+0000 7f26937fe640 1 -- 172.21.15.23:0/3981627898 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v102) v1 ==== 103+0+11 (secure 0 0 0) 0x7f268c09bfd0 con 0x7f269c100400 2024-03-28T05:33:38.827 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T05:33:38.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.827+0000 7f269b59e640 1 -- 172.21.15.23:0/3981627898 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f26880a2970 msgr2=0x7f26880a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:38.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.828+0000 7f269b59e640 1 --2- 172.21.15.23:0/3981627898 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f26880a2970 0x7f26880a4e30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2684009860 tx=0x7f26840058b0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.828+0000 7f269b59e640 1 -- 172.21.15.23:0/3981627898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f269c100400 msgr2=0x7f269c197d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:38.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.828+0000 7f269b59e640 1 --2- 172.21.15.23:0/3981627898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f269c100400 0x7f269c197d70 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f268c00c920 tx=0x7f268c00cde0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.828+0000 7f269b59e640 1 -- 172.21.15.23:0/3981627898 shutdown_connections 2024-03-28T05:33:38.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.828+0000 7f269b59e640 1 --2- 172.21.15.23:0/3981627898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f269c1064a0 0x7f269c19c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.828+0000 7f269b59e640 1 --2- 172.21.15.23:0/3981627898 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f26880a2970 0x7f26880a4e30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.828+0000 7f269b59e640 1 --2- 172.21.15.23:0/3981627898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f269c100dc0 0x7f269c1982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.828+0000 7f269b59e640 1 --2- 172.21.15.23:0/3981627898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f269c100400 0x7f269c197d70 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.828+0000 7f269b59e640 1 -- 172.21.15.23:0/3981627898 >> 172.21.15.23:0/3981627898 conn(0x7f269c0fbef0 msgr2=0x7f269c104240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:33:38.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.828+0000 7f269b59e640 1 -- 172.21.15.23:0/3981627898 shutdown_connections 2024-03-28T05:33:38.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.828+0000 7f269b59e640 1 -- 172.21.15.23:0/3981627898 wait complete. 2024-03-28T05:33:38.840 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-28T05:33:38.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.942+0000 7f10c3b87640 1 -- 172.21.15.23:0/3824292953 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10bc069d50 msgr2=0x7f10bc06a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:38.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.942+0000 7f10c3b87640 1 --2- 172.21.15.23:0/3824292953 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10bc069d50 0x7f10bc06a1b0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f10ac009900 tx=0x7f10ac02bcd0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.942+0000 7f10c3b87640 1 -- 172.21.15.23:0/3824292953 shutdown_connections 2024-03-28T05:33:38.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.942+0000 7f10c3b87640 1 --2- 172.21.15.23:0/3824292953 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10bc06a6f0 0x7f10bc10e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.942+0000 7f10c3b87640 1 --2- 172.21.15.23:0/3824292953 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10bc069d50 0x7f10bc06a1b0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.942+0000 7f10c3b87640 1 --2- 172.21.15.23:0/3824292953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10bc1076a0 0x7f10bc107a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.942+0000 7f10c3b87640 1 -- 172.21.15.23:0/3824292953 >> 172.21.15.23:0/3824292953 conn(0x7f10bc077640 msgr2=0x7f10bc077a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:33:38.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.942+0000 7f10c3b87640 1 -- 172.21.15.23:0/3824292953 shutdown_connections 2024-03-28T05:33:38.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.942+0000 7f10c3b87640 1 -- 172.21.15.23:0/3824292953 wait complete. 2024-03-28T05:33:38.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.943+0000 7f10c3b87640 1 Processor -- start 2024-03-28T05:33:38.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.943+0000 7f10c3b87640 1 -- start start 2024-03-28T05:33:38.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.943+0000 7f10c3b87640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10bc069d50 0x7f10bc1005d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:38.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.943+0000 7f10c3b87640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10bc06a6f0 0x7f10bc100b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:38.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.943+0000 7f10c3b87640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10bc1076a0 0x7f10bc101050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:38.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.943+0000 7f10c3b87640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc111a20 con 0x7f10bc1076a0 2024-03-28T05:33:38.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.943+0000 7f10c3b87640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc1118a0 con 0x7f10bc069d50 2024-03-28T05:33:38.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.943+0000 7f10c3b87640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc111ba0 con 0x7f10bc06a6f0 2024-03-28T05:33:38.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.943+0000 7f10c1122640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10bc06a6f0 0x7f10bc100b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:38.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.943+0000 7f10c2124640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10bc1076a0 0x7f10bc101050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:38.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.943+0000 7f10c1122640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10bc06a6f0 0x7f10bc100b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:57366/0 (socket says 172.21.15.23:57366) 2024-03-28T05:33:38.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.943+0000 7f10c1122640 1 -- 172.21.15.23:0/1354037424 learned_addr learned my addr 172.21.15.23:0/1354037424 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:33:38.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.943+0000 7f10c1923640 1 --2- 172.21.15.23:0/1354037424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10bc069d50 0x7f10bc1005d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:38.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.944+0000 7f10c1122640 1 -- 172.21.15.23:0/1354037424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10bc069d50 msgr2=0x7f10bc1005d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:38.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.944+0000 7f10c1122640 1 --2- 172.21.15.23:0/1354037424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10bc069d50 0x7f10bc1005d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.944+0000 7f10c1122640 1 -- 172.21.15.23:0/1354037424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10bc1076a0 msgr2=0x7f10bc101050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:38.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.944+0000 7f10c1122640 1 --2- 172.21.15.23:0/1354037424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10bc1076a0 0x7f10bc101050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:38.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.944+0000 7f10c1122640 1 -- 172.21.15.23:0/1354037424 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10bc1a63a0 con 0x7f10bc06a6f0 2024-03-28T05:33:38.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.944+0000 7f10c2124640 1 --2- 172.21.15.23:0/1354037424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10bc1076a0 0x7f10bc101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T05:33:38.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.944+0000 7f10c1122640 1 --2- 172.21.15.23:0/1354037424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10bc06a6f0 0x7f10bc100b10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f10ac005940 tx=0x7f10ac02cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:33:38.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.944+0000 7f10b2ffd640 1 -- 172.21.15.23:0/1354037424 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ac041070 con 0x7f10bc06a6f0 2024-03-28T05:33:38.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.944+0000 7f10b2ffd640 1 -- 172.21.15.23:0/1354037424 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f10ac004450 con 0x7f10bc06a6f0 2024-03-28T05:33:38.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.944+0000 7f10c3b87640 1 -- 172.21.15.23:0/1354037424 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10bc1a6630 con 0x7f10bc06a6f0 2024-03-28T05:33:38.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.944+0000 7f10b2ffd640 1 -- 172.21.15.23:0/1354037424 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ac003df0 con 0x7f10bc06a6f0 2024-03-28T05:33:38.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.944+0000 7f10c3b87640 1 -- 172.21.15.23:0/1354037424 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10bc1a6b20 con 0x7f10bc06a6f0 2024-03-28T05:33:38.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.945+0000 7f10c3b87640 1 -- 172.21.15.23:0/1354037424 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10bc102dc0 con 0x7f10bc06a6f0 2024-03-28T05:33:38.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.946+0000 7f10b2ffd640 1 -- 172.21.15.23:0/1354037424 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f10ac044070 con 0x7f10bc06a6f0 2024-03-28T05:33:38.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.946+0000 7f10b2ffd640 1 --2- 172.21.15.23:0/1354037424 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f10a80a29a0 0x7f10a80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:33:38.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.946+0000 7f10c1923640 1 --2- 172.21.15.23:0/1354037424 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f10a80a29a0 0x7f10a80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:33:38.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.946+0000 7f10b2ffd640 1 -- 172.21.15.23:0/1354037424 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f10ac0f0c40 con 0x7f10bc06a6f0 2024-03-28T05:33:38.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.947+0000 7f10c1923640 1 --2- 172.21.15.23:0/1354037424 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f10a80a29a0 0x7f10a80a4e60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f10b8005d00 tx=0x7f10b800b410 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:33:38.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:38.951+0000 7f10b2ffd640 1 -- 172.21.15.23:0/1354037424 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f10ac0bd070 con 0x7f10bc06a6f0 2024-03-28T05:33:39.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:38 smithi098 ceph-mon[36087]: from='client.25927 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:33:39.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:38 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3825872786' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:33:39.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:38 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2104701992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:33:39.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:38 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3523010930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:33:39.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:38 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3981627898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:33:39.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:39.184+0000 7f10c3b87640 1 -- 172.21.15.23:0/1354037424 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f10bc109b40 con 0x7f10bc06a6f0 2024-03-28T05:33:39.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:39.192+0000 7f10b2ffd640 1 -- 172.21.15.23:0/1354037424 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) v1 ==== 81+0+740360 (secure 0 0 0) 0x7f10ac0c1110 con 0x7f10bc06a6f0 2024-03-28T05:33:39.206 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:33:39.207 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T05:33:39.207 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 26, 2024-03-28T05:33:39.207 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.207 INFO:teuthology.orchestra.run.smithi023.stdout: "active_gid": 15402, 2024-03-28T05:33:39.207 INFO:teuthology.orchestra.run.smithi023.stdout: "active_name": "x", 2024-03-28T05:33:39.207 INFO:teuthology.orchestra.run.smithi023.stdout: "active_addrs": { 2024-03-28T05:33:39.207 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-28T05:33:39.207 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.207 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-28T05:33:39.207 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6800", 2024-03-28T05:33:39.207 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1385031613 2024-03-28T05:33:39.208 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.208 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.208 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-28T05:33:39.208 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6801", 2024-03-28T05:33:39.208 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1385031613 2024-03-28T05:33:39.208 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.208 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T05:33:39.208 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.208 INFO:teuthology.orchestra.run.smithi023.stdout: "active_addr": "172.21.15.23:6801/1385031613", 2024-03-28T05:33:39.208 INFO:teuthology.orchestra.run.smithi023.stdout: "active_change": "2024-03-28T05:23:33.950502+0000", 2024-03-28T05:33:39.208 INFO:teuthology.orchestra.run.smithi023.stdout: "active_mgr_features": 4540701547738038271, 2024-03-28T05:33:39.209 INFO:teuthology.orchestra.run.smithi023.stdout: "available": true, 2024-03-28T05:33:39.209 INFO:teuthology.orchestra.run.smithi023.stdout: "standbys": [ 2024-03-28T05:33:39.209 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.209 INFO:teuthology.orchestra.run.smithi023.stdout: "gid": 24971, 2024-03-28T05:33:39.209 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "y", 2024-03-28T05:33:39.209 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_features": 4540701547738038271, 2024-03-28T05:33:39.209 INFO:teuthology.orchestra.run.smithi023.stdout: "available_modules": [ 2024-03-28T05:33:39.209 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.209 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alerts", 2024-03-28T05:33:39.209 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.209 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.209 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.210 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T05:33:39.210 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T05:33:39.210 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.210 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.210 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.210 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T05:33:39.210 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.210 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.210 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.210 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T05:33:39.211 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.213 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.214 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.214 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.215 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.215 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.215 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.215 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.215 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.215 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.215 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.215 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.215 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.215 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.216 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.216 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.216 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.216 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.216 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.216 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.216 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.216 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.216 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.216 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.216 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.217 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.217 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.217 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.217 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.217 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.217 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.217 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.217 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.217 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.217 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.217 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.217 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.218 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.218 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.218 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.218 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.218 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.218 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.218 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.218 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.218 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.218 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.218 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.219 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.219 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.219 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.219 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.219 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.219 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.219 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.219 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.219 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.219 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.219 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.219 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.220 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.220 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.220 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.220 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.220 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.220 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.220 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.220 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.220 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.220 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.220 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.220 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.221 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.221 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.221 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_destination": { 2024-03-28T05:33:39.221 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_destination", 2024-03-28T05:33:39.221 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.221 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.221 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.221 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.221 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.221 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.221 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.221 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email address to send alerts to", 2024-03-28T05:33:39.221 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.222 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.222 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.222 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.222 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_from_name": { 2024-03-28T05:33:39.222 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_from_name", 2024-03-28T05:33:39.222 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.222 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.222 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.222 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "Ceph", 2024-03-28T05:33:39.222 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.222 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.222 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.223 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email From: name", 2024-03-28T05:33:39.223 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.223 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.223 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.223 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.223 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_host": { 2024-03-28T05:33:39.223 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_host", 2024-03-28T05:33:39.223 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.223 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.223 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.223 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.223 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.223 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.224 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.224 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP server", 2024-03-28T05:33:39.224 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.224 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.224 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.224 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.224 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_password": { 2024-03-28T05:33:39.224 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_password", 2024-03-28T05:33:39.224 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.224 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.224 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.224 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.225 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.225 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.225 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.225 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Password to authenticate with", 2024-03-28T05:33:39.225 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.225 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.225 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.225 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.225 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_port": { 2024-03-28T05:33:39.225 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_port", 2024-03-28T05:33:39.225 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.226 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.226 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.226 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "465", 2024-03-28T05:33:39.226 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.226 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.226 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.226 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP port", 2024-03-28T05:33:39.226 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.226 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.226 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.226 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.227 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_sender": { 2024-03-28T05:33:39.227 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_sender", 2024-03-28T05:33:39.227 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.227 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.227 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.227 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.227 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.227 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.227 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.227 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP envelope sender", 2024-03-28T05:33:39.227 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.227 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.228 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.228 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.228 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_ssl": { 2024-03-28T05:33:39.228 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_ssl", 2024-03-28T05:33:39.228 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.228 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.228 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.228 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.228 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.228 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.228 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.228 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T05:33:39.229 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.229 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.229 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.229 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.229 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_user": { 2024-03-28T05:33:39.229 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_user", 2024-03-28T05:33:39.229 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.229 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.229 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.229 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.229 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.229 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.229 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.230 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "User to authenticate as", 2024-03-28T05:33:39.230 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.230 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.230 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.230 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.230 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.230 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.230 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.230 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "balancer", 2024-03-28T05:33:39.230 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.230 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.231 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.231 INFO:teuthology.orchestra.run.smithi023.stdout: "active": { 2024-03-28T05:33:39.231 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "active", 2024-03-28T05:33:39.231 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.231 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.231 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.231 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.231 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.231 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.231 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.231 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T05:33:39.231 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.232 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.232 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.232 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.232 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_time": { 2024-03-28T05:33:39.232 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_time", 2024-03-28T05:33:39.232 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.232 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.232 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.232 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0000", 2024-03-28T05:33:39.232 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.232 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.232 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.233 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T05:33:39.233 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T05:33:39.233 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.233 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.233 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.233 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_weekday": { 2024-03-28T05:33:39.233 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_weekday", 2024-03-28T05:33:39.233 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T05:33:39.233 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.233 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.233 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T05:33:39.233 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-28T05:33:39.234 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-28T05:33:39.234 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.234 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T05:33:39.234 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T05:33:39.234 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.234 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.234 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.234 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_max_iterations": { 2024-03-28T05:33:39.234 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_max_iterations", 2024-03-28T05:33:39.234 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T05:33:39.234 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.234 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.234 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "25", 2024-03-28T05:33:39.235 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T05:33:39.235 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "250", 2024-03-28T05:33:39.235 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.235 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T05:33:39.235 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.235 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.235 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.235 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.235 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_metrics": { 2024-03-28T05:33:39.235 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_metrics", 2024-03-28T05:33:39.235 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.235 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.235 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.236 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T05:33:39.236 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.236 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.236 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.236 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T05:33:39.236 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T05:33:39.236 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.236 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.236 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.236 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_step": { 2024-03-28T05:33:39.236 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_step", 2024-03-28T05:33:39.237 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T05:33:39.237 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.237 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.237 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.5", 2024-03-28T05:33:39.237 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0.001", 2024-03-28T05:33:39.237 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "0.999", 2024-03-28T05:33:39.237 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.237 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "aggressiveness of optimization", 2024-03-28T05:33:39.237 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T05:33:39.237 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.237 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.237 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.237 INFO:teuthology.orchestra.run.smithi023.stdout: "end_time": { 2024-03-28T05:33:39.238 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_time", 2024-03-28T05:33:39.238 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.238 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.238 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.238 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2359", 2024-03-28T05:33:39.238 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.238 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.238 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.238 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T05:33:39.238 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T05:33:39.238 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.238 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.238 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.239 INFO:teuthology.orchestra.run.smithi023.stdout: "end_weekday": { 2024-03-28T05:33:39.239 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_weekday", 2024-03-28T05:33:39.239 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T05:33:39.239 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.239 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.239 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T05:33:39.239 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-28T05:33:39.239 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-28T05:33:39.239 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.239 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T05:33:39.239 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T05:33:39.239 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.239 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.240 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.240 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.240 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.240 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.240 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.240 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.240 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.240 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.240 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.240 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.241 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.241 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.241 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.241 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.241 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.241 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.241 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.241 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.241 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.241 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.241 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.241 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.241 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.241 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.242 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.242 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.242 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.242 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.242 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.242 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.242 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.242 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.242 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.242 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.242 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.242 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.243 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.243 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.243 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.243 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.243 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.243 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.243 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.243 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.243 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.243 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.243 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.243 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.244 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.244 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.244 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.244 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.244 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.244 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.244 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.244 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.244 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.244 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.244 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.244 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.244 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.245 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.245 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.245 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.245 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.245 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.245 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.245 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.245 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.245 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.246 INFO:teuthology.orchestra.run.smithi023.stdout: "min_score": { 2024-03-28T05:33:39.246 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_score", 2024-03-28T05:33:39.246 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T05:33:39.246 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.246 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.246 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T05:33:39.246 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.246 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.246 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.246 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T05:33:39.246 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.246 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.247 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.247 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.247 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-28T05:33:39.247 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-28T05:33:39.247 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.247 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.247 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.247 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "upmap", 2024-03-28T05:33:39.247 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.247 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.247 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.247 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-compat", 2024-03-28T05:33:39.247 INFO:teuthology.orchestra.run.smithi023.stdout: "none", 2024-03-28T05:33:39.247 INFO:teuthology.orchestra.run.smithi023.stdout: "read", 2024-03-28T05:33:39.248 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap", 2024-03-28T05:33:39.248 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap-read" 2024-03-28T05:33:39.248 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.248 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Balancer mode", 2024-03-28T05:33:39.248 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.248 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.248 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.248 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.248 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_ids": { 2024-03-28T05:33:39.248 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_ids", 2024-03-28T05:33:39.248 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.248 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.248 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.249 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.249 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.249 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.249 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.249 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T05:33:39.249 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.249 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.249 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.249 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.249 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T05:33:39.249 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T05:33:39.249 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.250 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.250 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.250 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T05:33:39.250 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.250 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.250 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.250 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T05:33:39.250 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.250 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.250 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.250 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.250 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_deviation": { 2024-03-28T05:33:39.250 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_deviation", 2024-03-28T05:33:39.251 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.251 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.251 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.251 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T05:33:39.251 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T05:33:39.251 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.251 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.251 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T05:33:39.251 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T05:33:39.251 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.251 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.251 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.251 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_optimizations": { 2024-03-28T05:33:39.251 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_optimizations", 2024-03-28T05:33:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T05:33:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T05:33:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.253 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.253 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm", 2024-03-28T05:33:39.253 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.253 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.253 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.253 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_down_multiplier": { 2024-03-28T05:33:39.253 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_down_multiplier", 2024-03-28T05:33:39.253 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T05:33:39.253 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.253 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.253 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-28T05:33:39.253 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.253 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.253 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.253 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T05:33:39.254 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.254 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.254 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.254 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.254 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_refresh_rate": { 2024-03-28T05:33:39.254 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_refresh_rate", 2024-03-28T05:33:39.254 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.254 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.254 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.254 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "20", 2024-03-28T05:33:39.254 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.254 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.254 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.255 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T05:33:39.255 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.255 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.255 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.255 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.255 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_starting_port": { 2024-03-28T05:33:39.255 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_starting_port", 2024-03-28T05:33:39.255 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.255 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.255 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.255 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4721", 2024-03-28T05:33:39.255 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.255 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.255 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.255 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T05:33:39.256 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.256 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.256 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.256 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.256 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_ptrace": { 2024-03-28T05:33:39.256 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_ptrace", 2024-03-28T05:33:39.256 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.256 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.256 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.256 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.256 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.256 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.256 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.257 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T05:33:39.257 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T05:33:39.257 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.257 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.257 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.257 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_interval": { 2024-03-28T05:33:39.257 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_interval", 2024-03-28T05:33:39.257 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.257 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.257 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.257 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T05:33:39.257 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.257 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.257 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.258 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T05:33:39.258 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.258 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.258 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.258 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.258 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_memory_target_ratio": { 2024-03-28T05:33:39.258 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T05:33:39.258 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T05:33:39.258 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.258 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.258 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.7", 2024-03-28T05:33:39.258 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.258 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.258 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.259 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T05:33:39.259 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.259 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.259 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.259 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.259 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm_log_destination": { 2024-03-28T05:33:39.259 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm_log_destination", 2024-03-28T05:33:39.259 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.259 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.259 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.259 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.259 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.259 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.260 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.260 INFO:teuthology.orchestra.run.smithi023.stdout: "file", 2024-03-28T05:33:39.260 INFO:teuthology.orchestra.run.smithi023.stdout: "file,syslog", 2024-03-28T05:33:39.260 INFO:teuthology.orchestra.run.smithi023.stdout: "syslog" 2024-03-28T05:33:39.260 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.260 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T05:33:39.260 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.260 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.260 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.260 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.260 INFO:teuthology.orchestra.run.smithi023.stdout: "cgroups_split": { 2024-03-28T05:33:39.260 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cgroups_split", 2024-03-28T05:33:39.260 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.260 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.261 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.261 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.261 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.261 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.261 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.261 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T05:33:39.261 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.261 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.261 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.261 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.261 INFO:teuthology.orchestra.run.smithi023.stdout: "config_checks_enabled": { 2024-03-28T05:33:39.261 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_checks_enabled", 2024-03-28T05:33:39.261 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.262 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.262 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.262 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.262 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.262 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.262 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.262 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T05:33:39.262 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.262 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.262 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.262 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.262 INFO:teuthology.orchestra.run.smithi023.stdout: "config_dashboard": { 2024-03-28T05:33:39.262 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_dashboard", 2024-03-28T05:33:39.263 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.263 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.263 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.263 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.263 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.263 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.263 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.263 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T05:33:39.263 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.263 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.263 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.263 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.263 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_alertmanager": { 2024-03-28T05:33:39.263 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_alertmanager", 2024-03-28T05:33:39.264 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.264 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.264 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.264 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T05:33:39.264 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.264 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.264 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.264 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T05:33:39.264 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.264 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.264 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.264 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.264 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_base": { 2024-03-28T05:33:39.265 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_base", 2024-03-28T05:33:39.265 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.265 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.265 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.265 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T05:33:39.265 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.265 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.265 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.265 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Container image name, without the tag", 2024-03-28T05:33:39.266 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.266 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.266 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.266 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_elasticsearch": { 2024-03-28T05:33:39.266 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_elasticsearch", 2024-03-28T05:33:39.266 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.266 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.266 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.266 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T05:33:39.266 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.266 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.266 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.267 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "elasticsearch container image", 2024-03-28T05:33:39.267 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.267 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.267 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.267 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.267 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_grafana": { 2024-03-28T05:33:39.267 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_grafana", 2024-03-28T05:33:39.267 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.267 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.267 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.267 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T05:33:39.267 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.267 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.267 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.268 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T05:33:39.268 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.268 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.268 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.268 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.268 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_haproxy": { 2024-03-28T05:33:39.268 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_haproxy", 2024-03-28T05:33:39.268 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.268 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.268 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.268 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T05:33:39.268 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.269 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.269 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.269 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "HAproxy container image", 2024-03-28T05:33:39.269 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.269 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.269 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.269 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.269 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_agent": { 2024-03-28T05:33:39.269 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_agent", 2024-03-28T05:33:39.269 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.269 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.269 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.270 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T05:33:39.270 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.270 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.270 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.270 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger agent container image", 2024-03-28T05:33:39.270 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.270 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.270 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.271 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.271 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_collector": { 2024-03-28T05:33:39.271 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_collector", 2024-03-28T05:33:39.271 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.271 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.271 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.271 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T05:33:39.271 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.271 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.271 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.271 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger collector container image", 2024-03-28T05:33:39.271 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.271 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.272 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.272 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.272 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_query": { 2024-03-28T05:33:39.272 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_query", 2024-03-28T05:33:39.272 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.272 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.272 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.272 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T05:33:39.272 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.272 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.272 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.272 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger query container image", 2024-03-28T05:33:39.272 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.272 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.273 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.273 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.273 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_keepalived": { 2024-03-28T05:33:39.273 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_keepalived", 2024-03-28T05:33:39.273 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.273 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.273 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.273 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T05:33:39.273 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.273 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.273 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.273 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Keepalived container image", 2024-03-28T05:33:39.273 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.274 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.274 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.274 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.274 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_loki": { 2024-03-28T05:33:39.274 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_loki", 2024-03-28T05:33:39.274 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.274 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.274 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.274 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T05:33:39.274 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.274 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.275 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.275 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Loki container image", 2024-03-28T05:33:39.275 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.275 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.275 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.275 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.275 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_node_exporter": { 2024-03-28T05:33:39.275 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_node_exporter", 2024-03-28T05:33:39.275 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.275 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.275 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.275 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T05:33:39.276 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.276 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.276 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.276 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T05:33:39.276 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.276 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.276 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.276 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.276 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_nvmeof": { 2024-03-28T05:33:39.276 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_nvmeof", 2024-03-28T05:33:39.276 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.276 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.276 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.276 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-28T05:33:39.277 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.277 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.277 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.277 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Nvme-of container image", 2024-03-28T05:33:39.277 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.277 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.277 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.277 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.277 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_prometheus": { 2024-03-28T05:33:39.277 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_prometheus", 2024-03-28T05:33:39.277 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.277 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.277 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.277 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T05:33:39.278 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.278 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.278 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.278 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T05:33:39.278 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.278 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.279 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.279 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.279 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_promtail": { 2024-03-28T05:33:39.279 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_promtail", 2024-03-28T05:33:39.279 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.279 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.280 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.280 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T05:33:39.280 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.280 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.280 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.280 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Promtail container image", 2024-03-28T05:33:39.280 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.280 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.280 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.280 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.280 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_snmp_gateway": { 2024-03-28T05:33:39.280 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_snmp_gateway", 2024-03-28T05:33:39.280 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.281 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.281 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.281 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T05:33:39.281 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.281 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.281 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.281 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SNMP Gateway container image", 2024-03-28T05:33:39.281 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.281 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.281 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.281 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.281 INFO:teuthology.orchestra.run.smithi023.stdout: "container_init": { 2024-03-28T05:33:39.281 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_init", 2024-03-28T05:33:39.282 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.282 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.282 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.282 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.282 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.282 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.282 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.282 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T05:33:39.282 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.282 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.282 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.282 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.282 INFO:teuthology.orchestra.run.smithi023.stdout: "daemon_cache_timeout": { 2024-03-28T05:33:39.283 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "daemon_cache_timeout", 2024-03-28T05:33:39.283 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.283 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.283 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.283 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T05:33:39.283 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.283 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.283 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.283 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T05:33:39.283 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.283 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.283 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.284 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.284 INFO:teuthology.orchestra.run.smithi023.stdout: "default_cephadm_command_timeout": { 2024-03-28T05:33:39.284 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T05:33:39.284 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.284 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.284 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.284 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "900", 2024-03-28T05:33:39.284 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.284 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.284 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.285 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T05:33:39.285 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.285 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.285 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.285 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.285 INFO:teuthology.orchestra.run.smithi023.stdout: "default_registry": { 2024-03-28T05:33:39.285 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_registry", 2024-03-28T05:33:39.285 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.285 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.285 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.285 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io", 2024-03-28T05:33:39.285 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.285 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.286 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.286 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T05:33:39.286 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.286 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.286 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.286 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.286 INFO:teuthology.orchestra.run.smithi023.stdout: "device_cache_timeout": { 2024-03-28T05:33:39.286 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_cache_timeout", 2024-03-28T05:33:39.286 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.286 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.286 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.286 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1800", 2024-03-28T05:33:39.287 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.287 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.287 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.287 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache device inventory", 2024-03-28T05:33:39.287 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.287 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.287 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.287 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.287 INFO:teuthology.orchestra.run.smithi023.stdout: "device_enhanced_scan": { 2024-03-28T05:33:39.287 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_enhanced_scan", 2024-03-28T05:33:39.287 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.287 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.287 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.288 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.288 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.288 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.288 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.288 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T05:33:39.288 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.288 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.288 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.288 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.288 INFO:teuthology.orchestra.run.smithi023.stdout: "facts_cache_timeout": { 2024-03-28T05:33:39.288 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "facts_cache_timeout", 2024-03-28T05:33:39.288 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.288 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.288 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.288 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T05:33:39.289 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.289 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.289 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.289 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache host facts data", 2024-03-28T05:33:39.289 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.289 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.289 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.289 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.289 INFO:teuthology.orchestra.run.smithi023.stdout: "host_check_interval": { 2024-03-28T05:33:39.289 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "host_check_interval", 2024-03-28T05:33:39.289 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.289 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.289 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.289 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T05:33:39.289 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.290 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.290 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.290 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to perform a host check", 2024-03-28T05:33:39.290 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.290 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.290 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.290 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.290 INFO:teuthology.orchestra.run.smithi023.stdout: "hw_monitoring": { 2024-03-28T05:33:39.290 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hw_monitoring", 2024-03-28T05:33:39.290 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.290 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.290 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.290 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.291 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.291 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.291 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.291 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T05:33:39.291 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.291 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.291 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.291 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.291 INFO:teuthology.orchestra.run.smithi023.stdout: "inventory_list_all": { 2024-03-28T05:33:39.291 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "inventory_list_all", 2024-03-28T05:33:39.291 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.291 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.291 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.291 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.291 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.292 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.292 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.292 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T05:33:39.292 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.292 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.292 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.292 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.292 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.292 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.292 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.292 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.292 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.292 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.292 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.292 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.292 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.293 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.293 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.293 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.293 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.293 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.293 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.293 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.293 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.293 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.293 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.293 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.293 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.293 INFO:teuthology.orchestra.run.smithi023.stdout: "log_refresh_metadata": { 2024-03-28T05:33:39.293 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_refresh_metadata", 2024-03-28T05:33:39.293 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.294 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.294 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.294 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.294 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.294 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.294 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.294 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T05:33:39.294 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.294 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.294 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.294 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.294 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.294 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.295 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.295 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.295 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.295 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.295 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.295 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.295 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.295 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T05:33:39.295 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.295 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.295 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.295 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.295 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.295 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.295 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.296 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.296 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.296 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.296 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.296 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.296 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.296 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.296 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.296 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.296 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.296 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.296 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.296 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.296 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.296 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.297 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.297 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.297 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.297 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.297 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.297 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.297 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.297 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.297 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.297 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.297 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.297 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.297 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.297 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.297 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.298 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.298 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.298 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T05:33:39.298 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T05:33:39.298 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.298 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.298 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.298 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.298 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.298 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.298 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.298 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T05:33:39.298 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.298 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.299 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.299 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.299 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T05:33:39.299 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T05:33:39.299 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.299 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.299 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.299 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "*", 2024-03-28T05:33:39.299 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.299 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.299 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.299 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T05:33:39.299 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.299 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.299 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.300 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.300 INFO:teuthology.orchestra.run.smithi023.stdout: "max_count_per_host": { 2024-03-28T05:33:39.300 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_count_per_host", 2024-03-28T05:33:39.300 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.300 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.300 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.300 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:39.300 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.300 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.300 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.300 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of daemons per service per host", 2024-03-28T05:33:39.300 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.300 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.300 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.300 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.301 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd_draining_count": { 2024-03-28T05:33:39.301 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_osd_draining_count", 2024-03-28T05:33:39.301 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.301 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.301 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.301 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:39.301 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.301 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.301 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.301 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T05:33:39.301 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.301 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.301 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.301 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.301 INFO:teuthology.orchestra.run.smithi023.stdout: "migration_current": { 2024-03-28T05:33:39.301 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "migration_current", 2024-03-28T05:33:39.302 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.302 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.302 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.302 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.302 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.302 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.302 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.302 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "internal - do not modify", 2024-03-28T05:33:39.302 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.302 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.302 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.302 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.302 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-28T05:33:39.302 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-28T05:33:39.303 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.303 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.303 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.303 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "root", 2024-03-28T05:33:39.303 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.303 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.303 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.303 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm-package", 2024-03-28T05:33:39.303 INFO:teuthology.orchestra.run.smithi023.stdout: "root" 2024-03-28T05:33:39.303 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.303 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T05:33:39.303 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.303 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.303 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.303 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.303 INFO:teuthology.orchestra.run.smithi023.stdout: "oob_default_addr": { 2024-03-28T05:33:39.304 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "oob_default_addr", 2024-03-28T05:33:39.304 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.304 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.304 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.304 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "169.254.1.1", 2024-03-28T05:33:39.304 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.304 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.304 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.304 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T05:33:39.304 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.304 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.304 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.304 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.304 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_alerts_path": { 2024-03-28T05:33:39.304 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_alerts_path", 2024-03-28T05:33:39.305 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.305 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.305 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.305 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T05:33:39.305 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.305 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.305 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.305 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T05:33:39.305 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.305 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.305 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.305 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.305 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_insecure": { 2024-03-28T05:33:39.305 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_insecure", 2024-03-28T05:33:39.305 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.305 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.306 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.306 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.306 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.306 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.306 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.306 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T05:33:39.306 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.306 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.306 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.306 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.306 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_password": { 2024-03-28T05:33:39.306 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_password", 2024-03-28T05:33:39.306 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.306 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.306 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.307 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.307 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.307 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.307 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.307 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T05:33:39.307 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.307 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.307 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.307 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.307 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_url": { 2024-03-28T05:33:39.307 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_url", 2024-03-28T05:33:39.307 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.307 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.307 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.307 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.308 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.308 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.308 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.308 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T05:33:39.308 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.308 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.308 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.308 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.308 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_username": { 2024-03-28T05:33:39.308 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_username", 2024-03-28T05:33:39.308 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.308 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.308 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.308 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.308 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.308 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.309 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.309 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T05:33:39.309 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.309 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.309 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.309 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.309 INFO:teuthology.orchestra.run.smithi023.stdout: "secure_monitoring_stack": { 2024-03-28T05:33:39.309 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "secure_monitoring_stack", 2024-03-28T05:33:39.309 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.309 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.309 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.309 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.309 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.309 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.309 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.309 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T05:33:39.310 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.310 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.310 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.310 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.310 INFO:teuthology.orchestra.run.smithi023.stdout: "service_discovery_port": { 2024-03-28T05:33:39.310 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "service_discovery_port", 2024-03-28T05:33:39.310 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.310 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.310 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.310 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8765", 2024-03-28T05:33:39.310 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.310 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.310 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.311 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "cephadm service discovery port", 2024-03-28T05:33:39.311 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.311 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.311 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.311 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.311 INFO:teuthology.orchestra.run.smithi023.stdout: "ssh_config_file": { 2024-03-28T05:33:39.311 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssh_config_file", 2024-03-28T05:33:39.311 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.311 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.311 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.311 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.311 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.311 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.311 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.311 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T05:33:39.311 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.312 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.312 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.312 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.312 INFO:teuthology.orchestra.run.smithi023.stdout: "use_agent": { 2024-03-28T05:33:39.312 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_agent", 2024-03-28T05:33:39.312 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.312 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.312 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.312 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.312 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.312 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.312 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.312 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T05:33:39.312 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.312 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.312 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.312 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.313 INFO:teuthology.orchestra.run.smithi023.stdout: "use_repo_digest": { 2024-03-28T05:33:39.313 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_repo_digest", 2024-03-28T05:33:39.313 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.313 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.313 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.313 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.313 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.313 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.313 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.313 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T05:33:39.313 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.313 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.313 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.313 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.313 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_failed_host_check": { 2024-03-28T05:33:39.314 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_failed_host_check", 2024-03-28T05:33:39.314 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.314 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.314 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.314 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.314 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.314 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.314 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.314 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T05:33:39.314 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.314 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.315 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.315 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.315 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_daemons": { 2024-03-28T05:33:39.315 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_daemons", 2024-03-28T05:33:39.315 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.315 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.315 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.315 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.315 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.315 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.315 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.315 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T05:33:39.315 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.315 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.315 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.316 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.316 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_hosts": { 2024-03-28T05:33:39.316 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_hosts", 2024-03-28T05:33:39.316 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.316 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.316 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.316 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.316 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.316 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.316 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.316 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T05:33:39.316 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.316 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.316 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.316 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.316 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.317 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.317 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.317 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crash", 2024-03-28T05:33:39.317 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.317 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.317 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.317 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.317 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.317 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.317 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.317 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.317 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.317 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.317 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.317 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.317 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.317 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.318 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.318 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.318 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.318 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.318 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.318 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.318 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.318 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.318 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.318 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.318 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.318 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.318 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.318 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.319 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.319 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.319 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.319 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.319 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.319 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.319 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.319 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.319 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.319 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.319 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.319 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.319 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.319 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.319 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.320 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.320 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.320 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.320 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.320 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.320 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.320 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.320 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.320 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.320 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.320 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.320 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.320 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.320 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.320 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.320 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.321 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.321 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.321 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.321 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.321 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.321 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.321 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.321 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.321 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.321 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.321 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.321 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.321 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.321 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.321 INFO:teuthology.orchestra.run.smithi023.stdout: "retain_interval": { 2024-03-28T05:33:39.321 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retain_interval", 2024-03-28T05:33:39.322 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.322 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.322 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.322 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "31536000", 2024-03-28T05:33:39.322 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.322 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.322 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.322 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T05:33:39.322 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.322 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.322 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.323 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.323 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_recent_interval": { 2024-03-28T05:33:39.323 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_recent_interval", 2024-03-28T05:33:39.323 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.323 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.323 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.323 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1209600", 2024-03-28T05:33:39.323 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.323 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.323 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.323 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T05:33:39.323 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.323 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.323 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.324 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.324 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.324 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.324 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.324 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dashboard", 2024-03-28T05:33:39.324 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.324 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.324 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.324 INFO:teuthology.orchestra.run.smithi023.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T05:33:39.324 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T05:33:39.324 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.324 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.324 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.324 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:39.324 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.324 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.325 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.325 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.325 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.325 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.325 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.325 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.325 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T05:33:39.325 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T05:33:39.325 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.325 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.325 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.325 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.325 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.325 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.325 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.325 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.326 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.326 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.326 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.326 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.326 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T05:33:39.326 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T05:33:39.326 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.326 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.326 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.326 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.326 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.326 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.326 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.326 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.326 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.327 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.327 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.327 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.327 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_ENABLED": { 2024-03-28T05:33:39.327 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T05:33:39.327 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.327 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.327 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.327 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.327 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.327 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.327 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.327 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.327 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.327 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.327 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.328 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.328 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T05:33:39.328 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T05:33:39.328 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.328 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.328 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.328 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.328 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.328 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.328 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.328 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.328 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.328 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.328 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.328 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.328 INFO:teuthology.orchestra.run.smithi023.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T05:33:39.329 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T05:33:39.329 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.329 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.329 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.329 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.329 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.329 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.329 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.329 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.329 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.329 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.329 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.329 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.329 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T05:33:39.329 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T05:33:39.330 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.330 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.330 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.330 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.330 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.330 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.330 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.330 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.330 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.330 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.330 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.330 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.330 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T05:33:39.330 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T05:33:39.331 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.331 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.331 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.331 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.331 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.331 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.331 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.331 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.331 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.331 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.331 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.331 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.331 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T05:33:39.331 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T05:33:39.331 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.331 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.332 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.332 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.332 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.332 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.332 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.332 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.332 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.332 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.332 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.332 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.332 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T05:33:39.332 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T05:33:39.332 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.332 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.332 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.332 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.333 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.333 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.333 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.333 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.333 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.333 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.333 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.333 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.333 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T05:33:39.333 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T05:33:39.333 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.333 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.333 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.333 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.333 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.333 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.333 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.334 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.334 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.334 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.334 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.334 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.334 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T05:33:39.334 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T05:33:39.334 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.334 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.334 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.334 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.334 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.334 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.334 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.335 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.335 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.335 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.335 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.335 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.335 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T05:33:39.335 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T05:33:39.335 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.335 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.335 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.335 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.335 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.335 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.335 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.335 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.335 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.336 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.336 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.336 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.336 INFO:teuthology.orchestra.run.smithi023.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T05:33:39.336 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T05:33:39.336 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.336 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.336 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.336 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.336 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.336 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.336 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.336 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.337 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.337 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.337 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.337 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.337 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T05:33:39.337 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T05:33:39.337 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.337 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.337 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.337 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-28T05:33:39.337 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.337 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.337 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.337 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.337 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.337 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.337 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.338 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.338 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T05:33:39.338 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T05:33:39.338 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.338 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.338 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.338 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.338 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.338 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.338 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.338 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.338 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.338 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.338 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.338 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.339 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_URL": { 2024-03-28T05:33:39.339 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_URL", 2024-03-28T05:33:39.339 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.339 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.339 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.339 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.339 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.339 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.339 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.339 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.339 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.339 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.339 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.339 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.339 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T05:33:39.339 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T05:33:39.340 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.340 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.340 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.340 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-28T05:33:39.340 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.340 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.340 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.340 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.340 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.340 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.340 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.340 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.340 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T05:33:39.340 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T05:33:39.340 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.340 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.341 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.341 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.341 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.341 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.341 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.341 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.341 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.341 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.341 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.341 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.341 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T05:33:39.341 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T05:33:39.341 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.341 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.341 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.341 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.341 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.342 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.342 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.342 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.342 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.342 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.342 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.342 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.342 INFO:teuthology.orchestra.run.smithi023.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T05:33:39.342 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T05:33:39.342 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.342 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.342 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.342 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.342 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.342 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.342 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.343 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.343 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.343 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.343 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.343 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.343 INFO:teuthology.orchestra.run.smithi023.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T05:33:39.343 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T05:33:39.343 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.343 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.343 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.343 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.343 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.343 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.343 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.343 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.344 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.344 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.344 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.344 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.344 INFO:teuthology.orchestra.run.smithi023.stdout: "MULTICLUSTER_CONFIG": { 2024-03-28T05:33:39.344 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-28T05:33:39.344 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.344 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.344 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.344 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "{}", 2024-03-28T05:33:39.344 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.344 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.344 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.344 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.344 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.344 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.344 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.345 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.345 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T05:33:39.345 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T05:33:39.345 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.345 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.345 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.345 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.345 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.345 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.345 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.345 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.345 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.345 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.345 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.345 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.345 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T05:33:39.346 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T05:33:39.346 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.346 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.346 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.346 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.346 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.346 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.346 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.346 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.346 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.346 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.346 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.346 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.346 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T05:33:39.346 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T05:33:39.347 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.347 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.347 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.347 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.347 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.347 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.347 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.347 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.347 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.347 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.347 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.347 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.347 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T05:33:39.347 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T05:33:39.347 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.347 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.347 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.348 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.348 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.348 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.348 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.348 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.348 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.348 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.348 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.348 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.348 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T05:33:39.348 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T05:33:39.348 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.348 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.348 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.348 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.348 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.349 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.349 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.349 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.349 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.349 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.349 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.349 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.349 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T05:33:39.349 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T05:33:39.349 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.349 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.349 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.349 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.349 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.349 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.349 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.349 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.350 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.350 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.350 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.350 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.350 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T05:33:39.350 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T05:33:39.350 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.350 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.350 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.350 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.350 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.350 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.350 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.350 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.351 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.351 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.351 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.351 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.351 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T05:33:39.351 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T05:33:39.351 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.351 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.351 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.351 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.351 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.351 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.351 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.351 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.351 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.351 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.351 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.352 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.352 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T05:33:39.352 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T05:33:39.352 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.352 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.352 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.352 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.352 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.352 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.352 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.352 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.352 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.352 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.352 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.352 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.352 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T05:33:39.352 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T05:33:39.353 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.353 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.353 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.353 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.353 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.353 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.353 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.353 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.353 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.353 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.353 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.353 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.353 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T05:33:39.353 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T05:33:39.353 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.353 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.353 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.354 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T05:33:39.354 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.354 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.354 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.354 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.354 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.354 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.354 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.354 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.354 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T05:33:39.354 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T05:33:39.354 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.354 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.354 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.355 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:39.355 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.355 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.355 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.355 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.355 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.355 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.355 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.355 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.355 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T05:33:39.355 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T05:33:39.355 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.355 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.355 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.355 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8", 2024-03-28T05:33:39.355 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.355 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.356 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.356 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.356 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.356 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.356 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.356 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.356 INFO:teuthology.orchestra.run.smithi023.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T05:33:39.356 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T05:33:39.356 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.356 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.356 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.356 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "45", 2024-03-28T05:33:39.356 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.356 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.356 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.356 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.356 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.357 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.357 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.357 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.357 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T05:33:39.357 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T05:33:39.357 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.357 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.357 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.357 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.357 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.357 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.357 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.357 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.357 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.357 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.357 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.357 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.358 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T05:33:39.358 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T05:33:39.358 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.358 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.358 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.358 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-28T05:33:39.358 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.358 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.358 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.358 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.358 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.358 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.358 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.358 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.358 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T05:33:39.359 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T05:33:39.359 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.359 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.359 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.359 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.359 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.359 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.359 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.359 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.359 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.359 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.359 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.359 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.359 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T05:33:39.359 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T05:33:39.359 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.359 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.360 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.360 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.360 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.360 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.360 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.360 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.360 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.360 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.360 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.360 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.360 INFO:teuthology.orchestra.run.smithi023.stdout: "UNSAFE_TLS_v1_2": { 2024-03-28T05:33:39.360 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-28T05:33:39.360 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.360 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.360 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.361 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.361 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.361 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.361 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.361 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.361 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.361 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.361 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.361 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.361 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T05:33:39.361 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T05:33:39.361 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.361 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.361 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.361 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T05:33:39.361 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.362 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.362 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.362 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.362 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.362 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.362 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.362 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.362 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T05:33:39.362 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T05:33:39.362 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.362 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.362 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.362 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:39.362 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.362 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.363 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.363 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.363 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.363 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.363 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.363 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.363 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T05:33:39.363 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T05:33:39.363 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.363 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.363 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.363 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T05:33:39.363 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.363 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.363 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.363 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.363 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.363 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.364 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.364 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.364 INFO:teuthology.orchestra.run.smithi023.stdout: "cross_origin_url": { 2024-03-28T05:33:39.364 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cross_origin_url", 2024-03-28T05:33:39.364 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.364 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.364 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.364 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.364 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.364 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.364 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.364 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.364 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.364 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.364 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.364 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.364 INFO:teuthology.orchestra.run.smithi023.stdout: "crt_file": { 2024-03-28T05:33:39.365 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crt_file", 2024-03-28T05:33:39.365 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.365 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.365 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.365 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.365 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.365 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.365 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.365 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.365 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.365 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.365 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.365 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.365 INFO:teuthology.orchestra.run.smithi023.stdout: "debug": { 2024-03-28T05:33:39.365 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "debug", 2024-03-28T05:33:39.365 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.365 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.366 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.366 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.366 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.366 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.366 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.366 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable/disable debug options", 2024-03-28T05:33:39.366 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.366 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.366 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.366 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.366 INFO:teuthology.orchestra.run.smithi023.stdout: "jwt_token_ttl": { 2024-03-28T05:33:39.366 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "jwt_token_ttl", 2024-03-28T05:33:39.366 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.366 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.367 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.367 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "28800", 2024-03-28T05:33:39.367 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.367 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.367 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.367 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.367 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.367 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.367 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.367 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.367 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-28T05:33:39.367 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-28T05:33:39.367 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.367 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.367 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.367 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.367 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.368 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.368 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.368 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.368 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.368 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.368 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.368 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.368 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.368 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.368 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.368 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.368 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.368 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.368 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.368 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.368 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.368 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.369 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.369 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.369 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.369 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.369 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.369 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.369 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.369 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.369 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.369 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.369 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.369 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.369 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.369 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.369 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.369 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.369 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.370 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.370 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.370 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.370 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.370 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.370 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.370 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.370 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.370 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.370 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.370 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.370 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.370 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.370 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.370 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.371 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.371 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.371 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.371 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.371 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.371 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.371 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.371 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.371 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.371 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.371 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.371 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.371 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.371 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.371 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.371 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.371 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.371 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.372 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.372 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.372 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.372 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.372 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.372 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.372 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.372 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.372 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.372 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.372 INFO:teuthology.orchestra.run.smithi023.stdout: "motd": { 2024-03-28T05:33:39.372 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "motd", 2024-03-28T05:33:39.372 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.372 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.372 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.372 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.372 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.373 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.373 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.373 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "The message of the day", 2024-03-28T05:33:39.373 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.373 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.373 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.373 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.373 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect_resolve_ip_addr": { 2024-03-28T05:33:39.373 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T05:33:39.373 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.373 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.373 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.373 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.373 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.373 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.373 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.373 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.374 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.374 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.374 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.374 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-28T05:33:39.374 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-28T05:33:39.374 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.374 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.374 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.374 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-28T05:33:39.374 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.374 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.374 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.374 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.374 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.374 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.375 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.375 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.375 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-28T05:33:39.375 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-28T05:33:39.375 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.375 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.375 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.375 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8080", 2024-03-28T05:33:39.375 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.375 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.375 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.375 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.375 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.375 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.375 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.375 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.375 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-28T05:33:39.375 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-28T05:33:39.376 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.376 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.376 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.376 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.376 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.376 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.376 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.376 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.376 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.376 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.376 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.376 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.376 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl_server_port": { 2024-03-28T05:33:39.376 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl_server_port", 2024-03-28T05:33:39.376 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.376 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.376 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.377 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8443", 2024-03-28T05:33:39.377 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.377 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.377 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.377 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.377 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.377 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.377 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.377 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.377 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-28T05:33:39.377 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-28T05:33:39.377 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.377 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.377 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.377 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "redirect", 2024-03-28T05:33:39.378 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.378 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.378 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.378 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.378 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect" 2024-03-28T05:33:39.378 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.378 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.378 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.378 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.378 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.378 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.378 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-28T05:33:39.378 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-28T05:33:39.378 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.378 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.378 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.379 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-28T05:33:39.379 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-28T05:33:39.379 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-28T05:33:39.379 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.379 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.379 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.379 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.379 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.379 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.379 INFO:teuthology.orchestra.run.smithi023.stdout: "url_prefix": { 2024-03-28T05:33:39.379 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url_prefix", 2024-03-28T05:33:39.379 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.379 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.379 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.379 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.379 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.380 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.380 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.380 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.380 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.380 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.380 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.380 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.380 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.380 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.380 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.380 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2024-03-28T05:33:39.380 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.380 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.380 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.380 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_monitoring": { 2024-03-28T05:33:39.380 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_monitoring", 2024-03-28T05:33:39.380 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.381 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.381 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.381 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.381 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.381 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.381 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.381 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "monitor device health metrics", 2024-03-28T05:33:39.381 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.381 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.381 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.381 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.381 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.381 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.381 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.381 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.381 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.381 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.382 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.382 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.382 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.382 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.382 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.382 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.382 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.382 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.382 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.382 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.382 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.382 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.382 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.382 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.382 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.382 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.383 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.383 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.383 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.383 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.383 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.383 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.383 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.383 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.383 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.383 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.383 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.383 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.383 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.383 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.383 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.383 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.383 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.383 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.384 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.384 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.384 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.384 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.384 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.384 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.384 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.384 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.384 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.384 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.384 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.384 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.384 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.384 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.384 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.384 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.384 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.385 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.385 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.385 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.385 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.385 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.385 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.385 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.385 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.385 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.385 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.385 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.385 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.385 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.385 INFO:teuthology.orchestra.run.smithi023.stdout: "mark_out_threshold": { 2024-03-28T05:33:39.385 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mark_out_threshold", 2024-03-28T05:33:39.385 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.385 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.385 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.386 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2419200", 2024-03-28T05:33:39.386 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.386 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.386 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.386 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T05:33:39.386 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.386 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.386 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.386 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.386 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": { 2024-03-28T05:33:39.386 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_name", 2024-03-28T05:33:39.386 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.386 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.386 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.386 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "device_health_metrics", 2024-03-28T05:33:39.386 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.386 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.387 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.387 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T05:33:39.387 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.387 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.387 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.387 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.387 INFO:teuthology.orchestra.run.smithi023.stdout: "retention_period": { 2024-03-28T05:33:39.387 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retention_period", 2024-03-28T05:33:39.387 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.387 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.387 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.387 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15552000", 2024-03-28T05:33:39.387 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.387 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.387 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.387 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain device health metrics", 2024-03-28T05:33:39.387 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.388 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.388 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.388 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.388 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_frequency": { 2024-03-28T05:33:39.388 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_frequency", 2024-03-28T05:33:39.388 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.388 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.388 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.388 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-28T05:33:39.388 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.388 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.388 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.388 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T05:33:39.388 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.388 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.388 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.388 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.389 INFO:teuthology.orchestra.run.smithi023.stdout: "self_heal": { 2024-03-28T05:33:39.389 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "self_heal", 2024-03-28T05:33:39.389 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.389 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.389 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.389 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.389 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.389 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.389 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.389 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T05:33:39.389 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.389 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.389 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.389 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.389 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T05:33:39.389 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T05:33:39.389 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.390 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.390 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.390 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T05:33:39.390 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.390 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.390 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.390 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T05:33:39.390 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.390 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.390 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.390 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.390 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_threshold": { 2024-03-28T05:33:39.390 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_threshold", 2024-03-28T05:33:39.390 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.390 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.390 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.391 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7257600", 2024-03-28T05:33:39.391 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.391 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.391 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.391 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T05:33:39.391 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.391 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.391 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.391 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.391 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.391 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.391 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.391 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "diskprediction_local", 2024-03-28T05:33:39.391 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.391 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.391 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.391 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.392 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.392 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.392 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.392 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.392 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.392 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.392 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.392 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.392 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.392 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.392 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.392 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.392 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.392 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.392 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.392 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.392 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.392 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.393 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.393 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.393 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.393 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.393 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.393 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.393 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.393 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.393 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.393 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.393 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.393 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.393 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.393 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.393 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.393 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.393 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.394 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.394 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.394 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.394 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.394 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.394 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.394 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.394 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.394 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.394 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.394 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.394 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.394 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.394 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.394 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.394 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.394 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.395 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.395 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.395 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.395 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.395 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.395 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.395 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.395 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.395 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.395 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.395 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.395 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.395 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.395 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.395 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.395 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.395 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.395 INFO:teuthology.orchestra.run.smithi023.stdout: "predict_interval": { 2024-03-28T05:33:39.396 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predict_interval", 2024-03-28T05:33:39.396 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.396 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.396 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.396 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-28T05:33:39.396 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.396 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.396 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.396 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.396 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.396 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.396 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.396 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.396 INFO:teuthology.orchestra.run.smithi023.stdout: "predictor_model": { 2024-03-28T05:33:39.396 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predictor_model", 2024-03-28T05:33:39.396 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.396 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.397 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.397 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "prophetstor", 2024-03-28T05:33:39.397 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.397 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.397 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.397 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.397 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.397 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.397 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.397 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.397 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T05:33:39.397 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T05:33:39.397 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.397 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.397 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.397 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T05:33:39.397 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.397 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.398 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.398 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.398 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.398 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.398 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.398 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.398 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.398 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.398 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.398 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "influx", 2024-03-28T05:33:39.398 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": false, 2024-03-28T05:33:39.398 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "influxdb python module not found", 2024-03-28T05:33:39.398 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.398 INFO:teuthology.orchestra.run.smithi023.stdout: "batch_size": { 2024-03-28T05:33:39.398 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "batch_size", 2024-03-28T05:33:39.398 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.398 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.398 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.399 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5000", 2024-03-28T05:33:39.399 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.399 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.399 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.399 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T05:33:39.399 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.399 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.399 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.399 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.399 INFO:teuthology.orchestra.run.smithi023.stdout: "database": { 2024-03-28T05:33:39.399 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "database", 2024-03-28T05:33:39.399 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.399 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.399 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.399 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "ceph", 2024-03-28T05:33:39.399 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.400 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.400 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.400 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T05:33:39.400 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.400 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.400 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.400 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.400 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": { 2024-03-28T05:33:39.400 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hostname", 2024-03-28T05:33:39.400 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.400 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.400 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.400 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.400 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.400 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.400 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.400 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server hostname", 2024-03-28T05:33:39.401 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.401 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.401 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.401 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.401 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T05:33:39.401 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T05:33:39.401 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.401 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.401 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.401 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "30", 2024-03-28T05:33:39.401 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "5", 2024-03-28T05:33:39.401 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.401 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.401 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T05:33:39.401 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.401 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.401 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.402 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.402 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.402 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.402 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.402 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.402 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.402 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.402 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.402 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.402 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.402 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.402 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.402 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.402 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.402 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.402 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.402 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.403 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.403 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.403 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.403 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.403 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.403 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.403 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.403 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.403 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.403 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.403 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.403 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.403 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.403 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.403 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.403 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.403 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.403 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.404 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.404 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.404 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.404 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.404 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.404 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.404 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.404 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.404 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.404 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.404 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.404 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.404 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.404 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.404 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.404 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.404 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.405 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.405 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.405 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.405 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.405 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.405 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.405 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.405 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.405 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.405 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.405 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.405 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.405 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.405 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.405 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.405 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.405 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.405 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.406 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.406 INFO:teuthology.orchestra.run.smithi023.stdout: "password": { 2024-03-28T05:33:39.406 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "password", 2024-03-28T05:33:39.406 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.406 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.406 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.406 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.406 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.406 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.406 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.406 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "password of InfluxDB server user", 2024-03-28T05:33:39.406 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.406 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.406 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.406 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.406 INFO:teuthology.orchestra.run.smithi023.stdout: "port": { 2024-03-28T05:33:39.407 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "port", 2024-03-28T05:33:39.407 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.407 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.407 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.407 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8086", 2024-03-28T05:33:39.407 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.407 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.407 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.407 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server port", 2024-03-28T05:33:39.407 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.407 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.407 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.407 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.407 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-28T05:33:39.407 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-28T05:33:39.407 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.407 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.407 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.408 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "false", 2024-03-28T05:33:39.408 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.408 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.408 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.408 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T05:33:39.408 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.408 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.408 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.408 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.408 INFO:teuthology.orchestra.run.smithi023.stdout: "threads": { 2024-03-28T05:33:39.408 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threads", 2024-03-28T05:33:39.408 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.408 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.408 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.408 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T05:33:39.408 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T05:33:39.408 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "32", 2024-03-28T05:33:39.409 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.409 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T05:33:39.409 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.409 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.409 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.409 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.409 INFO:teuthology.orchestra.run.smithi023.stdout: "username": { 2024-03-28T05:33:39.409 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "username", 2024-03-28T05:33:39.409 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.409 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.409 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.409 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.409 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.409 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.409 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.409 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "username of InfluxDB server user", 2024-03-28T05:33:39.409 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.409 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.410 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.410 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.410 INFO:teuthology.orchestra.run.smithi023.stdout: "verify_ssl": { 2024-03-28T05:33:39.410 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "verify_ssl", 2024-03-28T05:33:39.410 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.410 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.410 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.410 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "true", 2024-03-28T05:33:39.410 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.410 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.410 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.410 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T05:33:39.410 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.410 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.410 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.411 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.411 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.411 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.411 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.411 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "insights", 2024-03-28T05:33:39.411 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.411 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.411 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.411 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.411 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.411 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.411 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.411 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.411 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.411 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.411 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.411 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.411 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.412 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.412 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.412 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.412 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.412 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.412 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.412 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.412 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.412 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.412 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.412 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.412 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.412 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.412 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.412 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.412 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.412 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.413 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.413 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.413 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.413 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.413 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.413 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.413 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.413 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.413 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.413 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.413 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.413 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.413 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.413 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.413 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.413 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.413 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.414 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.414 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.414 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.414 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.414 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.414 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.414 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.414 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.414 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.414 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.414 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.414 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.414 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.414 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.414 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.415 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.415 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.415 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.415 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.415 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.415 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.415 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.415 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.415 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.415 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.415 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.415 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.415 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.415 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.415 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "iostat", 2024-03-28T05:33:39.415 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.416 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.416 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.416 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.416 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.416 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.416 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.416 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.416 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.416 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.416 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.416 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.416 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.416 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.416 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.416 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.416 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.416 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.416 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.417 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.417 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.417 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.417 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.417 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.417 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.417 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.417 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.417 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.417 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.417 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.417 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.417 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.417 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.417 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.417 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.417 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.418 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.418 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.418 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.418 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.418 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.418 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.418 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.418 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.418 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.418 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.418 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.418 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.418 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.418 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.418 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.418 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.418 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.418 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.419 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.419 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.419 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.419 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.419 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.419 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.419 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.419 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.419 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.419 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.419 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.419 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.419 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.419 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.419 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.420 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.420 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.420 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.420 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.420 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.420 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.420 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.420 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "k8sevents", 2024-03-28T05:33:39.420 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.420 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.420 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.420 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_event_retention_days": { 2024-03-28T05:33:39.420 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ceph_event_retention_days", 2024-03-28T05:33:39.420 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.420 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.420 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.420 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7", 2024-03-28T05:33:39.421 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.421 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.421 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.421 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T05:33:39.421 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.421 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.421 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.421 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.421 INFO:teuthology.orchestra.run.smithi023.stdout: "config_check_secs": { 2024-03-28T05:33:39.421 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_check_secs", 2024-03-28T05:33:39.421 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.421 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.421 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.421 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:39.421 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "10", 2024-03-28T05:33:39.421 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.421 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.422 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T05:33:39.422 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.422 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.422 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.422 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.422 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.422 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.422 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.422 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.422 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.422 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.422 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.422 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.422 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.422 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.422 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.423 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.423 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.423 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.423 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.423 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.423 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.423 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.423 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.423 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.423 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.423 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.423 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.423 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.423 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.423 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.423 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.423 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.424 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.424 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.424 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.424 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.424 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.424 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.424 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.424 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.424 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.424 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.424 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.424 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.424 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.424 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.424 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.424 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.424 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.425 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.425 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.425 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.425 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.425 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.425 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.425 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.425 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.425 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.425 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.425 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.425 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.425 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.425 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.425 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.425 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.425 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.426 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.426 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.426 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.426 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.426 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.426 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.426 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.426 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.426 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.426 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.426 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.426 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "localpool", 2024-03-28T05:33:39.426 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.426 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.426 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.426 INFO:teuthology.orchestra.run.smithi023.stdout: "failure_domain": { 2024-03-28T05:33:39.427 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "failure_domain", 2024-03-28T05:33:39.427 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.427 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.427 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.427 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "host", 2024-03-28T05:33:39.427 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.427 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.427 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.427 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "failure domain for any created local pool", 2024-03-28T05:33:39.427 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T05:33:39.427 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.427 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.427 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.427 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.427 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.427 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.427 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.428 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.428 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.428 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.428 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.428 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.428 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.428 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.428 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.428 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.428 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.428 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.428 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.428 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.428 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.428 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.428 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.428 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.428 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.429 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.429 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.429 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.429 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.429 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.429 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.429 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.429 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.429 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.429 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.429 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.429 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.429 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.429 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.429 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.429 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.430 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.430 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.430 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.430 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.430 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.430 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.430 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.430 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.430 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.430 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.430 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.430 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.430 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.430 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.430 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.430 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.430 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.431 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.431 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.431 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.431 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.431 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.431 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.431 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.431 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.431 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.431 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.431 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.431 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.431 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.431 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.431 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.431 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": { 2024-03-28T05:33:39.431 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_size", 2024-03-28T05:33:39.432 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.432 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.432 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.432 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.432 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.432 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.432 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.432 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default min_size for any created local pool", 2024-03-28T05:33:39.432 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T05:33:39.432 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.432 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.432 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.432 INFO:teuthology.orchestra.run.smithi023.stdout: "num_rep": { 2024-03-28T05:33:39.432 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "num_rep", 2024-03-28T05:33:39.432 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.432 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.432 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.432 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3", 2024-03-28T05:33:39.433 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.433 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.433 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.433 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default replica count for any created local pool", 2024-03-28T05:33:39.433 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.433 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.433 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.433 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.433 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": { 2024-03-28T05:33:39.433 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_num", 2024-03-28T05:33:39.433 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.433 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.433 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.433 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "128", 2024-03-28T05:33:39.433 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.433 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.434 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.434 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T05:33:39.434 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.434 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.434 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.434 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.434 INFO:teuthology.orchestra.run.smithi023.stdout: "prefix": { 2024-03-28T05:33:39.434 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prefix", 2024-03-28T05:33:39.434 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.434 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.434 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.434 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.434 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.434 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.434 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.434 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name prefix for any created local pool", 2024-03-28T05:33:39.434 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.435 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.435 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.435 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.435 INFO:teuthology.orchestra.run.smithi023.stdout: "subtree": { 2024-03-28T05:33:39.435 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "subtree", 2024-03-28T05:33:39.435 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.435 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.435 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.435 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "rack", 2024-03-28T05:33:39.435 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.435 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.435 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.435 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T05:33:39.435 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T05:33:39.435 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.435 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.436 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.436 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.436 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.436 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.436 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mds_autoscaler", 2024-03-28T05:33:39.436 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.436 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.436 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.436 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.436 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.436 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.436 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.436 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.436 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.436 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.436 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.436 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.436 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.437 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.437 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.437 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.437 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.437 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.437 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.437 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.437 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.437 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.437 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.437 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.437 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.437 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.437 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.437 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.437 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.438 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.438 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.438 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.438 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.438 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.438 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.438 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.438 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.438 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.438 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.438 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.438 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.438 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.438 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.438 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.438 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.438 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.439 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.439 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.439 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.439 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.439 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.439 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.439 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.439 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.439 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.439 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.439 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.439 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.439 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.439 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.439 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.439 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.439 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.440 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.440 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.440 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.440 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.440 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.440 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.440 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.440 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.440 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.440 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.440 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.440 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.440 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.440 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirroring", 2024-03-28T05:33:39.440 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.440 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.440 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.440 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.441 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.441 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.441 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.441 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.441 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.441 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.441 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.441 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.441 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.441 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.441 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.441 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.441 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.441 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.441 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.441 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.441 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.442 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.442 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.442 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.442 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.442 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.442 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.442 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.442 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.442 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.442 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.442 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.442 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.442 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.442 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.442 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.442 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.442 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.443 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.443 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.443 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.443 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.443 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.443 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.443 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.443 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.443 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.443 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.443 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.443 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.443 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.443 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.443 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.443 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.443 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.443 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.444 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.444 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.444 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.444 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.444 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.444 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.444 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.444 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.444 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.444 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.444 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.444 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.444 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.444 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.444 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.444 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.445 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.445 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.445 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.445 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.445 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "nfs", 2024-03-28T05:33:39.445 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.445 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.445 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.445 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.445 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.445 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.445 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.445 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.445 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.445 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.445 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.445 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.445 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.446 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.446 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.446 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.446 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.446 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.446 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.446 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.446 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.446 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.446 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.446 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.446 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.446 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.446 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.446 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.446 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.446 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.447 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.447 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.447 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.447 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.447 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.447 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.447 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.447 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.447 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.447 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.447 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.447 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.447 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.447 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.447 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.447 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.447 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.447 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.448 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.448 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.448 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.448 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.448 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.448 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.448 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.448 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.448 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.448 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.448 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.448 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.448 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.448 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.448 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.448 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.448 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.449 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.449 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.449 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.449 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.449 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.449 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.449 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.449 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.449 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.449 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.449 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.449 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-28T05:33:39.449 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.449 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.449 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.449 INFO:teuthology.orchestra.run.smithi023.stdout: "fail_fs": { 2024-03-28T05:33:39.450 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "fail_fs", 2024-03-28T05:33:39.450 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.450 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.450 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.450 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.450 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.450 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.450 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.450 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T05:33:39.450 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.450 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.450 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.450 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.450 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.450 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.450 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.450 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.451 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.451 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.451 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.451 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.451 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.451 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.451 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.451 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.451 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.451 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.451 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.451 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.451 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.451 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.451 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.451 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.451 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.452 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.452 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.452 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.452 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.452 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.452 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.452 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.452 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.452 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.452 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.452 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.452 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.452 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.452 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.452 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.452 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.452 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.453 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.453 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.453 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.453 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.453 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.453 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.453 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.453 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.453 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.453 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.453 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.453 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.453 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.453 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.453 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.453 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.453 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.453 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.454 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.454 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.454 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.454 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.454 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.454 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.454 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.454 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.454 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.454 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.454 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.454 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.454 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.454 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.454 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator": { 2024-03-28T05:33:39.454 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-28T05:33:39.454 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.455 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.455 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.455 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.455 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.455 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.455 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.455 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2024-03-28T05:33:39.455 INFO:teuthology.orchestra.run.smithi023.stdout: "rook", 2024-03-28T05:33:39.455 INFO:teuthology.orchestra.run.smithi023.stdout: "test_orchestrator" 2024-03-28T05:33:39.455 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.455 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Orchestrator backend", 2024-03-28T05:33:39.455 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.455 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.455 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.455 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.455 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.455 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.455 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.456 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_perf_query", 2024-03-28T05:33:39.456 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.456 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.456 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.456 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.456 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.456 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.456 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.456 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.456 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.456 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.456 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.456 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.456 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.456 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.456 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.456 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.457 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.457 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.457 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.457 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.457 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.457 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.457 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.457 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.457 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.457 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.457 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.457 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.457 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.457 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.457 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.457 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.457 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.458 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.458 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.458 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.458 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.458 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.458 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.458 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.458 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.458 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.458 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.458 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.458 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.458 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.458 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.458 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.458 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.458 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.458 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.459 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.459 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.459 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.459 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.459 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.459 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.459 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.459 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.459 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.459 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.459 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.459 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.459 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.459 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.459 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.459 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.460 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.460 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.460 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.460 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.460 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.460 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.460 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.460 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.460 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.460 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_support", 2024-03-28T05:33:39.460 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.460 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.460 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.460 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.460 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.460 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.460 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.461 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.461 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.461 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.461 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.461 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.461 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.461 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.461 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.461 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.461 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.461 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.461 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.461 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.461 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.461 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.461 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.461 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.462 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.462 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.462 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.462 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.462 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.462 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.462 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.462 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.462 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.462 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.462 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.462 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.462 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.462 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.462 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.462 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.463 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.463 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.463 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.463 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.463 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.463 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.463 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.463 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.463 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.463 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.463 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.463 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.463 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.463 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.463 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.463 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.463 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.464 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.464 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.464 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.464 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.464 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.464 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.464 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.464 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.464 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.464 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.464 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.464 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.464 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.464 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.464 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.464 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.464 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.464 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.465 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.465 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_autoscaler", 2024-03-28T05:33:39.465 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.465 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.465 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.465 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.465 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.465 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.465 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.465 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.465 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.465 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.465 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.465 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.465 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.465 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.465 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.465 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.466 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.466 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.466 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.466 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.466 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.466 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.466 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.466 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.466 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.466 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.466 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.466 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.466 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.466 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.466 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.466 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.466 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.467 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.467 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.467 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.467 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.467 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.467 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.467 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.467 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.467 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.467 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.467 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.467 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.467 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.467 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.467 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.467 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.467 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.468 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.468 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.468 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.468 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.468 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.468 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.468 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.468 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.468 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.468 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.468 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.468 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.468 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.468 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.468 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.468 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.468 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.469 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.469 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.469 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.469 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.469 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.469 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.469 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T05:33:39.469 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T05:33:39.469 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.469 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.469 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.469 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T05:33:39.469 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.469 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.469 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.469 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.469 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.469 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.470 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.470 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.470 INFO:teuthology.orchestra.run.smithi023.stdout: "threshold": { 2024-03-28T05:33:39.470 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threshold", 2024-03-28T05:33:39.470 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T05:33:39.470 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.470 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.470 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-28T05:33:39.470 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1.0", 2024-03-28T05:33:39.470 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.470 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.470 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "scaling threshold", 2024-03-28T05:33:39.470 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T05:33:39.470 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.470 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.471 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.471 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.471 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.471 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.471 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "progress", 2024-03-28T05:33:39.471 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.471 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.471 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.471 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_pg_recovery_event": { 2024-03-28T05:33:39.471 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_pg_recovery_event", 2024-03-28T05:33:39.471 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.471 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.471 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.471 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.471 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.471 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.471 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.471 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T05:33:39.472 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.472 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.472 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.472 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.472 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-28T05:33:39.472 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-28T05:33:39.472 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.472 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.472 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.472 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.472 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.472 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.472 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.472 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.472 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.472 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.472 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.473 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.473 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.473 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.473 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.473 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.473 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.473 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.473 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.473 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.473 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.473 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.473 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.473 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.473 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.473 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.473 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.473 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.473 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.474 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.474 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.474 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.474 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.474 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.474 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.474 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.474 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.474 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.474 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.474 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.474 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.474 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.474 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.474 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.474 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.474 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.475 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.475 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.475 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.475 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.475 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.475 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.475 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.475 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.475 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.475 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.475 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.475 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.475 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.475 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.475 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.475 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.475 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.475 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.476 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.476 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.476 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.476 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.476 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.476 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.476 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.476 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.476 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.476 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.476 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.476 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.476 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.476 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.476 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.476 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.476 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.476 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.477 INFO:teuthology.orchestra.run.smithi023.stdout: "max_completed_events": { 2024-03-28T05:33:39.477 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_completed_events", 2024-03-28T05:33:39.477 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.477 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.477 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.477 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "50", 2024-03-28T05:33:39.477 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.477 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.477 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.477 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "number of past completed events to remember", 2024-03-28T05:33:39.477 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.477 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.477 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.477 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.477 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T05:33:39.477 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T05:33:39.477 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.478 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.478 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.478 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T05:33:39.478 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.478 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.478 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.478 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long the module is going to sleep", 2024-03-28T05:33:39.478 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.478 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.478 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.478 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.478 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.478 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.478 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.478 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus", 2024-03-28T05:33:39.479 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.479 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.479 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.479 INFO:teuthology.orchestra.run.smithi023.stdout: "cache": { 2024-03-28T05:33:39.479 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cache", 2024-03-28T05:33:39.479 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.479 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.479 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.479 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.479 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.479 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.479 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.479 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.479 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.479 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.479 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.480 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.480 INFO:teuthology.orchestra.run.smithi023.stdout: "exclude_perf_counters": { 2024-03-28T05:33:39.480 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "exclude_perf_counters", 2024-03-28T05:33:39.480 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.480 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.480 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.480 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.480 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.480 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.480 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.480 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T05:33:39.480 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T05:33:39.480 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.480 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.480 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.480 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.480 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.481 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.481 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.481 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.481 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.481 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.481 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.481 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.481 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.481 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.481 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.481 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.481 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.481 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.481 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.481 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.481 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.481 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.482 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.483 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.483 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.483 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.483 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.483 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.483 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.483 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.483 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.483 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.483 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.483 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.483 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.483 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.483 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.483 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.483 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.483 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.483 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.484 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.484 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.484 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.484 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.484 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.484 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.484 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.484 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.484 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.484 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.484 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.484 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.484 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.484 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.484 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.484 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.484 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools": { 2024-03-28T05:33:39.485 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools", 2024-03-28T05:33:39.485 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.485 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.485 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.485 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.485 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.485 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.485 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.485 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.485 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.485 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.485 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.485 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.485 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T05:33:39.485 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T05:33:39.485 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.485 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.485 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.486 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "300", 2024-03-28T05:33:39.486 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.486 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.486 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.486 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.486 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.486 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.486 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.486 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.486 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_interval": { 2024-03-28T05:33:39.486 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_interval", 2024-03-28T05:33:39.486 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T05:33:39.486 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.486 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.486 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15.0", 2024-03-28T05:33:39.487 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.487 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.487 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.487 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.487 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.487 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.487 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.487 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.487 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-28T05:33:39.487 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-28T05:33:39.487 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.487 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.487 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.487 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-28T05:33:39.487 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.487 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.487 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.488 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T05:33:39.488 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.488 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.488 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.488 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.488 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-28T05:33:39.488 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-28T05:33:39.488 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.488 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.488 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.488 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "9283", 2024-03-28T05:33:39.488 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.488 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.488 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.488 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T05:33:39.488 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.488 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.489 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.489 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.489 INFO:teuthology.orchestra.run.smithi023.stdout: "stale_cache_strategy": { 2024-03-28T05:33:39.489 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stale_cache_strategy", 2024-03-28T05:33:39.489 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.489 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.489 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.489 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "log", 2024-03-28T05:33:39.489 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.489 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.489 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.489 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.489 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.489 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.489 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.490 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.490 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-28T05:33:39.490 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-28T05:33:39.490 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.490 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.490 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.490 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "default", 2024-03-28T05:33:39.490 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.490 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.490 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.490 INFO:teuthology.orchestra.run.smithi023.stdout: "default", 2024-03-28T05:33:39.490 INFO:teuthology.orchestra.run.smithi023.stdout: "error" 2024-03-28T05:33:39.490 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.490 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.490 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.490 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.490 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.491 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.491 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-28T05:33:39.491 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-28T05:33:39.491 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.491 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.491 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.491 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-28T05:33:39.491 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-28T05:33:39.491 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-28T05:33:39.491 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.491 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.491 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.491 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.491 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.491 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.491 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.491 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.492 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.492 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2024-03-28T05:33:39.492 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.492 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.492 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.492 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.492 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.492 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.492 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.492 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.492 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.492 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.492 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.492 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.492 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.492 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.492 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.492 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.493 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.493 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.493 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.493 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.493 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.493 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.493 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.493 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.493 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.493 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.493 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.493 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.493 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.493 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.493 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.493 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.493 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.494 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.494 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.494 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.494 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.494 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.494 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.494 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.494 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.494 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.494 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.494 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.494 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.494 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.494 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.494 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.494 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.494 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.494 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.495 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.495 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.495 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.495 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.495 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.495 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.495 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.495 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.495 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.495 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.495 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.495 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.495 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.495 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.495 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.495 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.495 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.496 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.496 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.496 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.496 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.496 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.496 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_snap_create": { 2024-03-28T05:33:39.496 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_snap_create", 2024-03-28T05:33:39.496 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.496 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.496 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.496 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:39.496 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.496 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.496 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.496 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.496 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.496 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.496 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.497 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.497 INFO:teuthology.orchestra.run.smithi023.stdout: "mirror_snapshot_schedule": { 2024-03-28T05:33:39.497 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T05:33:39.497 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.497 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.497 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.497 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.497 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.497 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.497 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.497 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.497 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.497 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.497 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.497 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.497 INFO:teuthology.orchestra.run.smithi023.stdout: "trash_purge_schedule": { 2024-03-28T05:33:39.497 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "trash_purge_schedule", 2024-03-28T05:33:39.497 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.498 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.498 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.498 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.498 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.498 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.498 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.498 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.498 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.498 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.498 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.498 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.498 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.498 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.498 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.498 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "restful", 2024-03-28T05:33:39.498 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.499 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.499 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.499 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_auth": { 2024-03-28T05:33:39.499 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_auth", 2024-03-28T05:33:39.499 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.499 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.499 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.499 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.499 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.499 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.499 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.499 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.499 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.499 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.499 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.499 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.499 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-28T05:33:39.500 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-28T05:33:39.500 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.500 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.500 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.500 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.500 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.500 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.500 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.500 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.500 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.500 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.500 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.500 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.500 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.500 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.501 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.501 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.501 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.501 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.501 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.501 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.501 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.501 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.501 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.501 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.501 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.501 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.501 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.501 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.501 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.501 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.502 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.502 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.502 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.502 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.502 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.502 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.502 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.502 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.502 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.502 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.502 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.502 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.502 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.502 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.502 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.502 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.502 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.503 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.503 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.503 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.503 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.503 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.503 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.503 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.503 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.503 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.503 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.503 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.503 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.503 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.503 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.503 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.503 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.503 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.504 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.504 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.504 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.504 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.504 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.504 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.504 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.504 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.504 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.504 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.504 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.504 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.504 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.504 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.504 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.504 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.504 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.504 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.505 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-28T05:33:39.505 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-28T05:33:39.505 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.505 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.505 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.505 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.505 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.505 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.505 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.505 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.505 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.505 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.505 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.505 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.505 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-28T05:33:39.505 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-28T05:33:39.505 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.506 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.506 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.506 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.506 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.506 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.506 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.506 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.506 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.506 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.506 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.506 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.506 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.506 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.506 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.506 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rgw", 2024-03-28T05:33:39.506 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.507 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.507 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.507 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.507 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.507 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.507 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.507 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.507 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.507 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.507 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.507 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.507 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.507 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.507 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.507 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.507 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.507 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.508 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.508 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.508 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.508 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.508 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.508 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.508 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.508 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.508 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.508 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.508 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.508 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.508 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.508 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.508 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.508 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.508 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.508 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.509 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.509 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.509 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.509 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.509 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.509 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.509 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.509 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.509 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.509 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.509 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.509 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.509 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.509 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.509 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.509 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.509 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.509 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.510 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.510 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.510 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.510 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.510 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.510 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.510 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.510 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.510 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.510 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.510 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.510 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.510 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.510 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.510 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.510 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.510 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.511 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.511 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.511 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.511 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.511 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.511 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rook", 2024-03-28T05:33:39.511 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.511 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.511 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.511 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.511 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.511 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.511 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.511 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.511 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.511 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.511 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.511 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.512 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.512 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.512 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.512 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.512 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.512 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.512 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.512 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.512 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.512 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.512 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.512 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.512 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.512 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.512 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.512 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.513 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.513 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.513 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.513 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.513 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.513 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.513 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.513 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.513 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.513 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.513 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.513 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.513 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.513 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.513 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.513 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.513 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.513 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.514 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.514 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.514 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.514 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.514 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.514 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.514 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.514 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.514 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.514 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.514 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.514 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.514 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.514 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.514 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.514 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.514 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.515 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.515 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.515 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.515 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.515 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.515 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.515 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.515 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.515 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.515 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.515 INFO:teuthology.orchestra.run.smithi023.stdout: "storage_class": { 2024-03-28T05:33:39.515 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "storage_class", 2024-03-28T05:33:39.515 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.515 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.515 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.516 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "local", 2024-03-28T05:33:39.516 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.516 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.516 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.516 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T05:33:39.516 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.516 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.516 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.516 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.516 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.516 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.516 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.516 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "selftest", 2024-03-28T05:33:39.516 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.516 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.516 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.517 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.517 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.517 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.517 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.517 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.517 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.517 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.517 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.517 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.517 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.517 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.517 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.517 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.517 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.517 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.517 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.517 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.518 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.518 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.518 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.518 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.518 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.518 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.518 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.518 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.518 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.518 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.518 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.518 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.518 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.518 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.518 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.518 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.518 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.519 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.519 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.519 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.519 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.519 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.519 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.519 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.519 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.519 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.519 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.519 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.519 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.519 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.519 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.519 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.519 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.519 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.519 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.520 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.520 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.520 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.520 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.520 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.520 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.520 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.520 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.520 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.520 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.520 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.520 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.520 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.520 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.520 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.521 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.521 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.521 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.521 INFO:teuthology.orchestra.run.smithi023.stdout: "roption1": { 2024-03-28T05:33:39.521 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption1", 2024-03-28T05:33:39.521 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.521 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.521 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.521 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.521 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.521 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.521 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.521 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.521 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.521 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.521 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.521 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.521 INFO:teuthology.orchestra.run.smithi023.stdout: "roption2": { 2024-03-28T05:33:39.522 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption2", 2024-03-28T05:33:39.522 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.522 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.522 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.522 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "xyz", 2024-03-28T05:33:39.522 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.522 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.522 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.522 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.522 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.522 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.522 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.522 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.522 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption1": { 2024-03-28T05:33:39.522 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption1", 2024-03-28T05:33:39.522 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.523 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.523 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.523 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.523 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.523 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.523 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.523 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.523 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.523 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.523 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.523 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.523 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption2": { 2024-03-28T05:33:39.523 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption2", 2024-03-28T05:33:39.523 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.523 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.523 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.523 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.523 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.524 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.524 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.524 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.524 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.524 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.524 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.524 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.524 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption3": { 2024-03-28T05:33:39.524 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption3", 2024-03-28T05:33:39.524 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T05:33:39.524 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.524 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.524 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.524 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.524 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.524 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.525 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.525 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.525 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.525 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.525 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.525 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption4": { 2024-03-28T05:33:39.525 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption4", 2024-03-28T05:33:39.525 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.525 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.525 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.525 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.525 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.525 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.525 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.525 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.525 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.525 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.525 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.526 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.526 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption5": { 2024-03-28T05:33:39.526 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption5", 2024-03-28T05:33:39.526 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.526 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.526 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.526 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.526 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.526 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.526 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.526 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.526 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.526 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.526 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.526 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.526 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption6": { 2024-03-28T05:33:39.526 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption6", 2024-03-28T05:33:39.527 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.527 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.527 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.527 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.527 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.527 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.527 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.527 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.527 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.527 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.527 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.527 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.527 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption7": { 2024-03-28T05:33:39.527 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption7", 2024-03-28T05:33:39.527 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.527 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.527 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.528 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.528 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T05:33:39.528 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "42", 2024-03-28T05:33:39.528 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.528 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.528 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.528 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.528 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.528 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.528 INFO:teuthology.orchestra.run.smithi023.stdout: "testkey": { 2024-03-28T05:33:39.528 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testkey", 2024-03-28T05:33:39.528 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.528 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.528 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.528 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.528 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.528 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.528 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.529 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.529 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.529 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.529 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.529 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.529 INFO:teuthology.orchestra.run.smithi023.stdout: "testlkey": { 2024-03-28T05:33:39.529 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testlkey", 2024-03-28T05:33:39.529 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.529 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.529 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.529 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.529 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.529 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.529 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.529 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.529 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.529 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.529 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.530 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.530 INFO:teuthology.orchestra.run.smithi023.stdout: "testnewline": { 2024-03-28T05:33:39.530 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testnewline", 2024-03-28T05:33:39.530 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.530 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.530 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.530 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.530 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.530 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.530 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.530 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.530 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.530 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.530 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.530 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.530 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.531 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.531 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.531 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-28T05:33:39.531 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.531 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.531 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.531 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_m_granularity": { 2024-03-28T05:33:39.531 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_m_granularity", 2024-03-28T05:33:39.531 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.531 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.531 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.531 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.531 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.531 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.531 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.531 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T05:33:39.531 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.531 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.532 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.532 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.532 INFO:teuthology.orchestra.run.smithi023.stdout: "dump_on_update": { 2024-03-28T05:33:39.532 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dump_on_update", 2024-03-28T05:33:39.532 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.532 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.532 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.532 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.532 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.532 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.532 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.532 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "dump database to debug log on update", 2024-03-28T05:33:39.532 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.532 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.532 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.532 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.532 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.533 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.533 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.533 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.533 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.533 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.533 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.533 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.533 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.533 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.533 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.533 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.533 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.533 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.533 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.533 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.533 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.533 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.533 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.534 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.534 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.534 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.534 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.534 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.534 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.534 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.534 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.534 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.534 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.534 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.534 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.534 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.534 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.534 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.534 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.534 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.535 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.535 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.535 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.535 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.535 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.535 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.535 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.535 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.535 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.535 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.535 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.535 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.535 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.535 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.535 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.535 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.535 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.536 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.536 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.536 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.536 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.536 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.536 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.536 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.536 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.536 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.536 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.536 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.536 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.536 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.536 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.536 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.536 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.536 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.536 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.537 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.537 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.537 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stats", 2024-03-28T05:33:39.537 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.537 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.537 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.537 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.537 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.537 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.537 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.537 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.537 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.537 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.537 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.537 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.537 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.537 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.537 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.538 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.538 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.538 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.538 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.538 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.538 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.538 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.538 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.538 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.538 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.538 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.538 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.538 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.538 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.538 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.538 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.539 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.539 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.539 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.539 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.539 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.539 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.539 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.539 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.539 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.539 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.539 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.539 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.539 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.539 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.539 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.539 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.539 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.540 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.540 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.540 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.540 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.540 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.540 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.540 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.540 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.540 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.540 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.540 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.540 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.540 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.540 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.540 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.540 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.540 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.541 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.541 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.541 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.541 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.541 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.541 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.541 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.541 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.541 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.541 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.541 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.541 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "status", 2024-03-28T05:33:39.541 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.541 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.541 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.541 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.541 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.541 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.542 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.542 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.542 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.542 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.542 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.542 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.542 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.542 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.542 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.542 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.542 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.542 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.542 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.542 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.542 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.542 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.543 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.543 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.543 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.543 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.543 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.543 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.543 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.543 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.543 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.543 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.543 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.543 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.543 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.543 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.543 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.543 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.543 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.543 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.544 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.544 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.544 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.544 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.544 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.544 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.544 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.544 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.544 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.544 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.544 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.544 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.544 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.544 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.544 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.544 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.544 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.544 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.545 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.545 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.545 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.545 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.545 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.545 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.545 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.545 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.545 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.545 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.545 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.545 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.545 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.545 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.545 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.545 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.545 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.545 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.546 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telegraf", 2024-03-28T05:33:39.546 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.546 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.546 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.546 INFO:teuthology.orchestra.run.smithi023.stdout: "address": { 2024-03-28T05:33:39.546 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "address", 2024-03-28T05:33:39.546 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.546 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.546 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.546 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T05:33:39.546 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.546 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.546 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.546 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.546 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.546 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.547 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.547 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.547 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T05:33:39.547 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T05:33:39.547 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.547 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.547 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.547 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15", 2024-03-28T05:33:39.547 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.547 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.547 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.547 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.547 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.547 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.547 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.547 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.547 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.547 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.548 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.548 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.548 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.548 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.548 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.548 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.548 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.548 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.548 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.548 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.548 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.548 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.548 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.548 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.548 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.548 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.548 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.549 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.549 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.549 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.549 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.549 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.549 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.549 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.549 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.549 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.549 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.549 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.549 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.549 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.549 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.549 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.549 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.549 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.550 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.550 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.550 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.550 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.550 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.550 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.550 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.550 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.550 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.550 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.550 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.550 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.550 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.550 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.550 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.550 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.550 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.551 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.551 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.551 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.551 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.551 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.551 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.551 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.551 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.551 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.551 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.551 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.551 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.551 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.551 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.551 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.551 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.551 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.551 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.552 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.552 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.552 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telemetry", 2024-03-28T05:33:39.552 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.552 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.552 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.552 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_basic": { 2024-03-28T05:33:39.552 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_basic", 2024-03-28T05:33:39.552 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.552 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.552 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.552 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.552 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.552 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.552 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.552 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T05:33:39.552 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.553 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.553 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.553 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.553 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_crash": { 2024-03-28T05:33:39.553 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_crash", 2024-03-28T05:33:39.553 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.553 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.553 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.553 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.553 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.553 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.553 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.553 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T05:33:39.553 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.553 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.553 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.553 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.553 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_device": { 2024-03-28T05:33:39.554 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_device", 2024-03-28T05:33:39.554 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.554 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.554 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.554 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.554 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.554 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.554 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.554 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T05:33:39.554 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.554 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.554 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.554 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.554 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_ident": { 2024-03-28T05:33:39.554 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_ident", 2024-03-28T05:33:39.554 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.554 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.555 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.555 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.555 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.555 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.555 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.555 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T05:33:39.555 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.555 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.555 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.555 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.555 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_perf": { 2024-03-28T05:33:39.555 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_perf", 2024-03-28T05:33:39.555 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.555 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.555 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.555 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.555 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.556 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.556 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.556 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T05:33:39.556 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.556 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.556 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.556 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.556 INFO:teuthology.orchestra.run.smithi023.stdout: "contact": { 2024-03-28T05:33:39.556 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "contact", 2024-03-28T05:33:39.556 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.556 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.556 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.556 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.556 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.556 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.556 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.556 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.556 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.557 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.557 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.557 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.557 INFO:teuthology.orchestra.run.smithi023.stdout: "description": { 2024-03-28T05:33:39.557 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "description", 2024-03-28T05:33:39.557 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.557 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.557 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.557 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.557 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.557 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.557 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.557 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.557 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.557 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.557 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.557 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.558 INFO:teuthology.orchestra.run.smithi023.stdout: "device_url": { 2024-03-28T05:33:39.558 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_url", 2024-03-28T05:33:39.558 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.558 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.558 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.558 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T05:33:39.558 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.558 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.558 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.558 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.558 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.558 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.558 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.558 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.558 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-28T05:33:39.558 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-28T05:33:39.559 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.559 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.559 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.559 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.559 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.559 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.559 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.559 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.559 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.559 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.559 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.559 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.559 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T05:33:39.559 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T05:33:39.559 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.559 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.559 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.560 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "24", 2024-03-28T05:33:39.560 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "8", 2024-03-28T05:33:39.560 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.560 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.560 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.560 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.560 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.560 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.560 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.560 INFO:teuthology.orchestra.run.smithi023.stdout: "last_opt_revision": { 2024-03-28T05:33:39.560 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "last_opt_revision", 2024-03-28T05:33:39.560 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.560 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.560 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.560 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1", 2024-03-28T05:33:39.560 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.560 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.561 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.561 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.561 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.561 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.561 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.561 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.561 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard": { 2024-03-28T05:33:39.561 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard", 2024-03-28T05:33:39.561 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.561 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.561 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.561 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.561 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.561 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.561 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.561 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.562 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.562 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.562 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.562 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.562 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard_description": { 2024-03-28T05:33:39.562 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard_description", 2024-03-28T05:33:39.562 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.562 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.562 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.562 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.562 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.562 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.562 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.562 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.562 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.562 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.563 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.563 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.563 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.563 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.563 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.563 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.563 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.563 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.563 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.563 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.563 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.563 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.563 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.563 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.563 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.563 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.563 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.564 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.564 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.564 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.564 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.564 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.564 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.564 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.564 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.564 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.564 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.564 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.564 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.564 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.564 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.564 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.564 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.564 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.565 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.565 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.565 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.565 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.565 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.565 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.565 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.565 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.565 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.565 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.565 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.565 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.565 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.565 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.565 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.565 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.565 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.565 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.566 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.566 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.566 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.566 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.566 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.566 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.566 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.566 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.566 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.566 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.566 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.566 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.566 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.566 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.566 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.566 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.567 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.567 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.567 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.567 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.567 INFO:teuthology.orchestra.run.smithi023.stdout: "organization": { 2024-03-28T05:33:39.567 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "organization", 2024-03-28T05:33:39.567 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.567 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.567 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.567 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.567 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.567 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.567 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.567 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.567 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.567 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.568 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.568 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.568 INFO:teuthology.orchestra.run.smithi023.stdout: "proxy": { 2024-03-28T05:33:39.568 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "proxy", 2024-03-28T05:33:39.568 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.568 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.568 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.568 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.568 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.568 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.568 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.568 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.568 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.568 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.568 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.568 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.568 INFO:teuthology.orchestra.run.smithi023.stdout: "url": { 2024-03-28T05:33:39.568 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url", 2024-03-28T05:33:39.569 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.569 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.569 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.569 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T05:33:39.569 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.569 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.569 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.569 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.569 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.569 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.569 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.569 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.569 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.569 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.570 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.570 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "test_orchestrator", 2024-03-28T05:33:39.570 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.570 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.570 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.570 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.570 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.570 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.570 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.570 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.570 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.570 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.570 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.570 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.571 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.571 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.571 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.571 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.571 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.571 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.571 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.571 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.571 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.571 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.571 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.571 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.571 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.571 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.571 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.571 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.571 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.571 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.572 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.572 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.572 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.572 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.572 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.572 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.572 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.572 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.572 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.572 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.572 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.572 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.572 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.572 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.572 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.572 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.572 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.573 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.573 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.573 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.573 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.573 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.573 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.573 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.573 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.573 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.573 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.573 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.573 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.573 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.573 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.573 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.573 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.573 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.574 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.574 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.574 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.574 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.574 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.574 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.574 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.574 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.574 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.574 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.574 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.574 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.574 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2024-03-28T05:33:39.574 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.574 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.574 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.575 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.575 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.575 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.575 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.575 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.575 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.575 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.575 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.575 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.575 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.575 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.575 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.575 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.575 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.575 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.575 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.575 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.576 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.576 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.576 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.576 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.576 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.576 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.576 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.576 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.576 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.576 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.576 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.576 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.576 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.576 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.576 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.576 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.576 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.577 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.577 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.577 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.577 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.577 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.577 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.577 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.577 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.577 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.577 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.577 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.577 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.577 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.577 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.577 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.577 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.577 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.578 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.578 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.578 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.578 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.578 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.578 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.578 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.578 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.578 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.578 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.578 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.578 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.578 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.578 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.578 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.579 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.579 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.579 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.579 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.579 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_clones": { 2024-03-28T05:33:39.579 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_clones", 2024-03-28T05:33:39.579 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.579 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.579 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.579 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4", 2024-03-28T05:33:39.579 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.579 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.579 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.579 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T05:33:39.579 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.579 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.579 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.579 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.580 INFO:teuthology.orchestra.run.smithi023.stdout: "periodic_async_work": { 2024-03-28T05:33:39.580 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "periodic_async_work", 2024-03-28T05:33:39.580 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.580 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.580 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.580 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.580 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.580 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.580 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.580 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Periodically check for async work", 2024-03-28T05:33:39.580 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.580 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.580 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.580 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.580 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_delay": { 2024-03-28T05:33:39.581 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_delay", 2024-03-28T05:33:39.581 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.581 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.581 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.581 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T05:33:39.581 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.581 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.581 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.581 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T05:33:39.581 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.581 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.581 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.581 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.581 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_no_wait": { 2024-03-28T05:33:39.581 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T05:33:39.581 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.581 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.582 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.582 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.582 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.582 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.582 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.582 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T05:33:39.582 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.582 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.582 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.582 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.582 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.582 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.582 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.582 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix", 2024-03-28T05:33:39.582 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.582 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.582 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.583 INFO:teuthology.orchestra.run.smithi023.stdout: "discovery_interval": { 2024-03-28T05:33:39.583 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "discovery_interval", 2024-03-28T05:33:39.583 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T05:33:39.583 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.583 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.583 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "100", 2024-03-28T05:33:39.583 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.583 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.583 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.583 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.583 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.583 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.583 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.583 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.583 INFO:teuthology.orchestra.run.smithi023.stdout: "identifier": { 2024-03-28T05:33:39.583 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "identifier", 2024-03-28T05:33:39.583 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.584 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.584 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.584 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.584 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.584 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.584 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.584 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.584 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.584 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.584 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.584 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.584 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T05:33:39.584 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T05:33:39.584 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.584 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.584 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.584 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T05:33:39.584 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.585 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.585 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.585 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.585 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.585 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.585 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.585 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.585 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.585 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.585 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.585 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.585 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.585 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.585 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.585 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.585 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.586 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.586 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.586 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.586 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.586 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.586 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.586 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.586 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.586 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.586 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.586 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.586 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.586 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.586 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.586 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.586 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.586 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.587 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.587 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.587 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.587 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.587 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.587 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.587 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.587 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.587 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.587 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.587 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.587 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.587 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.587 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.587 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.587 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.587 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.587 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.588 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.588 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.588 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.588 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.588 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.588 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.588 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.588 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.588 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.588 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.588 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.588 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.588 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.588 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.588 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.588 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.588 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.588 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.589 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.589 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.589 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.589 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.589 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.589 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.589 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.589 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.589 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_host": { 2024-03-28T05:33:39.589 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_host", 2024-03-28T05:33:39.589 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.589 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.589 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.589 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.589 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.589 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.589 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.590 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.590 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.590 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.590 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.590 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.590 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_port": { 2024-03-28T05:33:39.590 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_port", 2024-03-28T05:33:39.590 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.590 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.590 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.590 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10051", 2024-03-28T05:33:39.590 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.590 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.590 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.590 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.591 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.591 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.591 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.591 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.591 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_sender": { 2024-03-28T05:33:39.591 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_sender", 2024-03-28T05:33:39.591 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.591 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.591 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.591 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T05:33:39.591 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.591 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.591 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.591 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.591 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.591 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.591 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.591 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.592 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.592 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.592 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T05:33:39.592 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.592 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.592 INFO:teuthology.orchestra.run.smithi023.stdout: "gid": 25234, 2024-03-28T05:33:39.592 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "z", 2024-03-28T05:33:39.592 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_features": 4540701547738038271, 2024-03-28T05:33:39.592 INFO:teuthology.orchestra.run.smithi023.stdout: "available_modules": [ 2024-03-28T05:33:39.592 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.592 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alerts", 2024-03-28T05:33:39.592 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.592 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.592 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.592 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T05:33:39.592 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T05:33:39.592 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.593 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.593 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.593 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T05:33:39.593 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.593 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.593 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.593 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T05:33:39.593 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.593 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.593 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.593 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.593 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.593 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.593 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.593 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.593 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.593 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.593 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.594 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.594 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.594 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.594 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.594 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.594 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.594 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.594 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.594 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.594 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.594 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.594 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.594 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.594 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.594 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.594 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.594 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.595 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.595 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.595 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.595 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.595 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.595 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.595 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.595 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.595 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.595 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.595 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.595 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.595 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.595 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.595 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.595 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.595 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.596 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.596 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.596 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.596 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.596 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.596 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.596 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.596 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.596 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.596 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.596 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.596 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.596 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.596 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.596 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.596 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.596 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.597 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.597 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.597 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.597 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.597 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.597 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.597 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.597 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.597 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.597 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.597 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.597 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.597 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_destination": { 2024-03-28T05:33:39.597 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_destination", 2024-03-28T05:33:39.597 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.597 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.597 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.597 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.598 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.598 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.598 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.598 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email address to send alerts to", 2024-03-28T05:33:39.598 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.598 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.598 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.598 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.598 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_from_name": { 2024-03-28T05:33:39.598 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_from_name", 2024-03-28T05:33:39.598 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.598 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.598 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.598 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "Ceph", 2024-03-28T05:33:39.598 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.598 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.598 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.599 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email From: name", 2024-03-28T05:33:39.599 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.599 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.599 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.599 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.599 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_host": { 2024-03-28T05:33:39.599 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_host", 2024-03-28T05:33:39.599 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.599 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.599 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.599 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.599 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.599 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.599 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.599 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP server", 2024-03-28T05:33:39.599 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.600 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.600 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.600 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.600 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_password": { 2024-03-28T05:33:39.600 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_password", 2024-03-28T05:33:39.600 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.600 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.600 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.600 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.600 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.600 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.600 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.600 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Password to authenticate with", 2024-03-28T05:33:39.600 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.600 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.600 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.600 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.600 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_port": { 2024-03-28T05:33:39.600 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_port", 2024-03-28T05:33:39.601 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.601 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.601 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.601 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "465", 2024-03-28T05:33:39.601 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.601 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.601 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.601 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP port", 2024-03-28T05:33:39.601 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.601 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.601 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.601 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.601 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_sender": { 2024-03-28T05:33:39.601 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_sender", 2024-03-28T05:33:39.601 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.601 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.601 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.602 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.602 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.602 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.602 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.602 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP envelope sender", 2024-03-28T05:33:39.602 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.602 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.602 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.602 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.602 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_ssl": { 2024-03-28T05:33:39.602 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_ssl", 2024-03-28T05:33:39.602 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.602 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.602 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.602 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.602 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.602 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.602 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.603 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T05:33:39.603 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.603 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.603 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.603 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.603 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_user": { 2024-03-28T05:33:39.603 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_user", 2024-03-28T05:33:39.603 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.603 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.603 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.603 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.603 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.603 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.603 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.603 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "User to authenticate as", 2024-03-28T05:33:39.603 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.603 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.604 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.604 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.604 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.604 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.604 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.604 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "balancer", 2024-03-28T05:33:39.604 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.604 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.604 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.604 INFO:teuthology.orchestra.run.smithi023.stdout: "active": { 2024-03-28T05:33:39.604 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "active", 2024-03-28T05:33:39.604 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.604 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.604 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.604 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.604 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.604 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.605 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.605 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T05:33:39.605 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.605 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.605 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.605 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.605 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_time": { 2024-03-28T05:33:39.605 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_time", 2024-03-28T05:33:39.605 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.605 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.605 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.605 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0000", 2024-03-28T05:33:39.605 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.605 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.605 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.605 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T05:33:39.605 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T05:33:39.606 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.606 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.606 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.606 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_weekday": { 2024-03-28T05:33:39.606 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_weekday", 2024-03-28T05:33:39.606 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T05:33:39.606 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.606 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.606 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T05:33:39.606 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-28T05:33:39.606 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-28T05:33:39.606 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.606 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T05:33:39.606 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T05:33:39.606 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.607 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.607 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.607 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_max_iterations": { 2024-03-28T05:33:39.607 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_max_iterations", 2024-03-28T05:33:39.607 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T05:33:39.607 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.607 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.607 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "25", 2024-03-28T05:33:39.607 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T05:33:39.607 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "250", 2024-03-28T05:33:39.607 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.607 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T05:33:39.607 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.607 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.607 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.607 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.607 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_metrics": { 2024-03-28T05:33:39.608 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_metrics", 2024-03-28T05:33:39.608 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.608 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.608 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.608 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T05:33:39.608 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.608 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.608 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.608 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T05:33:39.608 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T05:33:39.608 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.608 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.608 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.608 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_step": { 2024-03-28T05:33:39.608 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_step", 2024-03-28T05:33:39.608 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T05:33:39.608 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.608 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.609 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.5", 2024-03-28T05:33:39.609 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0.001", 2024-03-28T05:33:39.609 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "0.999", 2024-03-28T05:33:39.609 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.609 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "aggressiveness of optimization", 2024-03-28T05:33:39.609 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T05:33:39.609 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.609 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.609 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.609 INFO:teuthology.orchestra.run.smithi023.stdout: "end_time": { 2024-03-28T05:33:39.609 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_time", 2024-03-28T05:33:39.609 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.609 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.609 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.609 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2359", 2024-03-28T05:33:39.609 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.609 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.610 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.610 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T05:33:39.610 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T05:33:39.610 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.610 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.610 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.610 INFO:teuthology.orchestra.run.smithi023.stdout: "end_weekday": { 2024-03-28T05:33:39.610 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_weekday", 2024-03-28T05:33:39.610 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T05:33:39.610 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.610 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.610 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T05:33:39.610 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-28T05:33:39.610 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-28T05:33:39.610 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.610 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T05:33:39.610 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T05:33:39.611 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.611 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.611 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.611 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.611 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.611 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.611 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.611 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.611 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.611 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.611 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.611 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.611 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.611 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.611 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.611 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.611 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.612 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.612 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.612 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.612 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.612 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.612 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.612 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.612 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.612 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.612 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.612 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.612 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.612 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.612 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.612 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.612 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.612 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.612 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.613 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.613 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.613 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.613 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.613 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.613 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.613 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.613 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.613 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.613 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.613 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.613 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.613 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.613 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.613 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.613 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.613 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.614 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.614 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.614 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.614 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.614 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.614 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.614 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.614 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.614 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.614 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.614 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.614 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.614 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.614 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.614 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.614 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.614 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.615 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.615 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.615 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.615 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.615 INFO:teuthology.orchestra.run.smithi023.stdout: "min_score": { 2024-03-28T05:33:39.615 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_score", 2024-03-28T05:33:39.615 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T05:33:39.615 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.615 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.615 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T05:33:39.615 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.615 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.615 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.615 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T05:33:39.615 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.615 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.615 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.616 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.616 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-28T05:33:39.616 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-28T05:33:39.616 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.616 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.616 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.616 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "upmap", 2024-03-28T05:33:39.616 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.616 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.616 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.616 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-compat", 2024-03-28T05:33:39.616 INFO:teuthology.orchestra.run.smithi023.stdout: "none", 2024-03-28T05:33:39.616 INFO:teuthology.orchestra.run.smithi023.stdout: "read", 2024-03-28T05:33:39.616 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap", 2024-03-28T05:33:39.616 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap-read" 2024-03-28T05:33:39.617 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.617 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Balancer mode", 2024-03-28T05:33:39.617 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.617 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.617 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.617 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.617 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_ids": { 2024-03-28T05:33:39.617 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_ids", 2024-03-28T05:33:39.617 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.617 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.617 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.617 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.617 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.617 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.617 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.617 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T05:33:39.617 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.618 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.618 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.618 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.618 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T05:33:39.618 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T05:33:39.618 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.618 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.618 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.618 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T05:33:39.618 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.618 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.621 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.621 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T05:33:39.621 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.621 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.621 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.621 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.621 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_deviation": { 2024-03-28T05:33:39.621 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_deviation", 2024-03-28T05:33:39.621 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.621 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.621 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.621 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T05:33:39.621 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T05:33:39.621 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.622 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.622 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T05:33:39.622 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T05:33:39.622 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.622 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.622 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.622 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_optimizations": { 2024-03-28T05:33:39.622 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_optimizations", 2024-03-28T05:33:39.622 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T05:33:39.622 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.622 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.622 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:39.622 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.622 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.622 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.622 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T05:33:39.622 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.623 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.623 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.623 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.623 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.623 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.623 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.623 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm", 2024-03-28T05:33:39.623 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.623 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.623 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.623 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_down_multiplier": { 2024-03-28T05:33:39.623 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_down_multiplier", 2024-03-28T05:33:39.623 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T05:33:39.623 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.623 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.623 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-28T05:33:39.623 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.623 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.624 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.624 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T05:33:39.624 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.624 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.624 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.624 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.624 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_refresh_rate": { 2024-03-28T05:33:39.624 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_refresh_rate", 2024-03-28T05:33:39.624 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.624 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.624 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.624 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "20", 2024-03-28T05:33:39.624 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.624 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.624 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.624 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T05:33:39.624 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.624 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.625 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.625 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.625 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_starting_port": { 2024-03-28T05:33:39.625 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_starting_port", 2024-03-28T05:33:39.625 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.625 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.625 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.625 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4721", 2024-03-28T05:33:39.625 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.625 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.625 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.625 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T05:33:39.625 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.625 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.625 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.625 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.625 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_ptrace": { 2024-03-28T05:33:39.626 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_ptrace", 2024-03-28T05:33:39.626 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.626 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.626 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.626 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.626 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.626 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.626 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.626 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T05:33:39.626 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T05:33:39.626 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.626 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.626 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.626 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_interval": { 2024-03-28T05:33:39.626 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_interval", 2024-03-28T05:33:39.626 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.627 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.627 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.627 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T05:33:39.627 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.627 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.627 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.627 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T05:33:39.627 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.627 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.627 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.627 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.627 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_memory_target_ratio": { 2024-03-28T05:33:39.627 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T05:33:39.627 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T05:33:39.627 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.627 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.627 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.7", 2024-03-28T05:33:39.627 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.628 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.628 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.628 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T05:33:39.628 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.628 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.628 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.628 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.628 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm_log_destination": { 2024-03-28T05:33:39.628 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm_log_destination", 2024-03-28T05:33:39.628 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.628 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.628 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.628 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.628 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.628 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.628 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.628 INFO:teuthology.orchestra.run.smithi023.stdout: "file", 2024-03-28T05:33:39.629 INFO:teuthology.orchestra.run.smithi023.stdout: "file,syslog", 2024-03-28T05:33:39.629 INFO:teuthology.orchestra.run.smithi023.stdout: "syslog" 2024-03-28T05:33:39.629 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.629 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T05:33:39.629 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.629 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.629 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.629 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.629 INFO:teuthology.orchestra.run.smithi023.stdout: "cgroups_split": { 2024-03-28T05:33:39.629 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cgroups_split", 2024-03-28T05:33:39.629 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.629 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.629 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.629 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.629 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.629 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.629 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.630 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T05:33:39.630 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.630 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.630 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.630 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.630 INFO:teuthology.orchestra.run.smithi023.stdout: "config_checks_enabled": { 2024-03-28T05:33:39.630 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_checks_enabled", 2024-03-28T05:33:39.630 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.630 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.630 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.630 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.630 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.630 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.630 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.631 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T05:33:39.631 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.631 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.631 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.631 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.631 INFO:teuthology.orchestra.run.smithi023.stdout: "config_dashboard": { 2024-03-28T05:33:39.631 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_dashboard", 2024-03-28T05:33:39.631 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.631 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.631 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.631 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.631 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.631 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.631 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.631 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T05:33:39.631 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.631 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.631 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.632 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.632 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_alertmanager": { 2024-03-28T05:33:39.632 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_alertmanager", 2024-03-28T05:33:39.632 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.632 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.632 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.632 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T05:33:39.632 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.632 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.632 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.632 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T05:33:39.632 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.632 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.632 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.632 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.632 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_base": { 2024-03-28T05:33:39.632 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_base", 2024-03-28T05:33:39.633 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.633 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.633 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.633 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T05:33:39.633 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.633 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.633 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.633 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Container image name, without the tag", 2024-03-28T05:33:39.633 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.633 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.633 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.633 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.633 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_elasticsearch": { 2024-03-28T05:33:39.633 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_elasticsearch", 2024-03-28T05:33:39.633 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.633 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.633 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.633 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T05:33:39.634 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.634 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.634 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.634 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "elasticsearch container image", 2024-03-28T05:33:39.634 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.634 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.634 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.634 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.634 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_grafana": { 2024-03-28T05:33:39.634 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_grafana", 2024-03-28T05:33:39.634 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.634 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.634 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.634 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T05:33:39.634 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.634 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.634 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.635 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T05:33:39.635 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.635 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.635 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.635 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.635 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_haproxy": { 2024-03-28T05:33:39.635 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_haproxy", 2024-03-28T05:33:39.635 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.635 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.635 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.635 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T05:33:39.635 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.635 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.635 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.635 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "HAproxy container image", 2024-03-28T05:33:39.635 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.635 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.635 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.636 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.636 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_agent": { 2024-03-28T05:33:39.636 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_agent", 2024-03-28T05:33:39.636 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.636 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.636 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.636 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T05:33:39.636 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.636 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.636 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.636 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger agent container image", 2024-03-28T05:33:39.636 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.636 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.636 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.636 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.636 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_collector": { 2024-03-28T05:33:39.636 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_collector", 2024-03-28T05:33:39.637 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.637 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.637 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.637 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T05:33:39.637 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.637 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.637 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.637 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger collector container image", 2024-03-28T05:33:39.637 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.637 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.637 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.637 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.637 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_query": { 2024-03-28T05:33:39.637 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_query", 2024-03-28T05:33:39.637 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.637 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.637 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.638 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T05:33:39.638 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.638 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.638 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.638 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger query container image", 2024-03-28T05:33:39.638 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.638 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.638 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.638 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.638 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_keepalived": { 2024-03-28T05:33:39.638 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_keepalived", 2024-03-28T05:33:39.638 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.638 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.638 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.638 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T05:33:39.638 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.639 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.639 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.639 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Keepalived container image", 2024-03-28T05:33:39.639 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.639 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.639 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.639 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.639 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_loki": { 2024-03-28T05:33:39.639 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_loki", 2024-03-28T05:33:39.639 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.639 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.639 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.639 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T05:33:39.639 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.639 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.639 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.639 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Loki container image", 2024-03-28T05:33:39.640 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.640 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.640 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.640 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.640 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_node_exporter": { 2024-03-28T05:33:39.640 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_node_exporter", 2024-03-28T05:33:39.640 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.640 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.640 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.640 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T05:33:39.640 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.640 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.640 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.640 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T05:33:39.640 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.641 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.641 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.641 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.641 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_nvmeof": { 2024-03-28T05:33:39.641 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_nvmeof", 2024-03-28T05:33:39.641 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.641 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.641 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.641 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-28T05:33:39.641 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.641 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.641 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.641 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Nvme-of container image", 2024-03-28T05:33:39.641 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.641 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.641 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.641 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.641 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_prometheus": { 2024-03-28T05:33:39.642 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_prometheus", 2024-03-28T05:33:39.642 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.642 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.642 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.642 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T05:33:39.642 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.642 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.642 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.642 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T05:33:39.642 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.642 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.642 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.642 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.642 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_promtail": { 2024-03-28T05:33:39.642 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_promtail", 2024-03-28T05:33:39.642 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.642 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.643 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.643 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T05:33:39.643 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.643 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.643 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.643 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Promtail container image", 2024-03-28T05:33:39.643 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.643 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.643 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.643 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.643 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_snmp_gateway": { 2024-03-28T05:33:39.643 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_snmp_gateway", 2024-03-28T05:33:39.643 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.643 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.643 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.643 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T05:33:39.644 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.644 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.644 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.644 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SNMP Gateway container image", 2024-03-28T05:33:39.644 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.644 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.644 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.644 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.644 INFO:teuthology.orchestra.run.smithi023.stdout: "container_init": { 2024-03-28T05:33:39.644 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_init", 2024-03-28T05:33:39.644 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.644 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.644 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.644 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.644 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.644 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.644 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.645 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T05:33:39.645 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.645 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.645 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.645 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.645 INFO:teuthology.orchestra.run.smithi023.stdout: "daemon_cache_timeout": { 2024-03-28T05:33:39.645 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "daemon_cache_timeout", 2024-03-28T05:33:39.645 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.645 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.645 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.645 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T05:33:39.645 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.645 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.645 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.645 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T05:33:39.645 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.645 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.645 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.646 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.646 INFO:teuthology.orchestra.run.smithi023.stdout: "default_cephadm_command_timeout": { 2024-03-28T05:33:39.646 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T05:33:39.646 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.646 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.646 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.646 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "900", 2024-03-28T05:33:39.646 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.646 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.646 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.646 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T05:33:39.646 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.646 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.646 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.646 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.646 INFO:teuthology.orchestra.run.smithi023.stdout: "default_registry": { 2024-03-28T05:33:39.647 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_registry", 2024-03-28T05:33:39.647 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.647 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.647 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.647 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io", 2024-03-28T05:33:39.647 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.647 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.647 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.647 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T05:33:39.647 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.647 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.647 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.647 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.647 INFO:teuthology.orchestra.run.smithi023.stdout: "device_cache_timeout": { 2024-03-28T05:33:39.647 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_cache_timeout", 2024-03-28T05:33:39.647 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.647 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.648 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.648 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1800", 2024-03-28T05:33:39.648 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.648 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.648 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.648 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache device inventory", 2024-03-28T05:33:39.648 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.648 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.648 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.648 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.648 INFO:teuthology.orchestra.run.smithi023.stdout: "device_enhanced_scan": { 2024-03-28T05:33:39.648 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_enhanced_scan", 2024-03-28T05:33:39.648 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.648 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.648 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.648 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.648 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.649 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.649 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.649 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T05:33:39.649 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.649 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.649 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.649 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.649 INFO:teuthology.orchestra.run.smithi023.stdout: "facts_cache_timeout": { 2024-03-28T05:33:39.649 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "facts_cache_timeout", 2024-03-28T05:33:39.649 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.649 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.649 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.649 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T05:33:39.649 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.649 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.649 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.649 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache host facts data", 2024-03-28T05:33:39.650 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.650 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.650 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.650 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.650 INFO:teuthology.orchestra.run.smithi023.stdout: "host_check_interval": { 2024-03-28T05:33:39.650 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "host_check_interval", 2024-03-28T05:33:39.650 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.650 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.650 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.650 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T05:33:39.650 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.650 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.650 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.650 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to perform a host check", 2024-03-28T05:33:39.650 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.650 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.651 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.651 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.651 INFO:teuthology.orchestra.run.smithi023.stdout: "hw_monitoring": { 2024-03-28T05:33:39.651 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hw_monitoring", 2024-03-28T05:33:39.651 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.651 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.651 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.651 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.651 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.651 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.651 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.651 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T05:33:39.651 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.651 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.651 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.651 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.651 INFO:teuthology.orchestra.run.smithi023.stdout: "inventory_list_all": { 2024-03-28T05:33:39.651 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "inventory_list_all", 2024-03-28T05:33:39.652 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.652 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.652 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.652 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.652 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.652 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.652 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.652 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T05:33:39.652 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.652 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.652 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.652 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.652 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.652 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.652 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.652 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.652 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.653 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.653 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.653 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.653 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.653 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.653 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.653 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.653 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.653 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.653 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.653 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.653 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.653 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.653 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.653 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.653 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.653 INFO:teuthology.orchestra.run.smithi023.stdout: "log_refresh_metadata": { 2024-03-28T05:33:39.654 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_refresh_metadata", 2024-03-28T05:33:39.654 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.654 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.654 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.654 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.654 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.654 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.654 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.654 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T05:33:39.654 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.654 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.654 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.654 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.654 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.654 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.654 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.654 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.655 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.655 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.655 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.655 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.655 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.655 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T05:33:39.655 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.655 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.655 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.655 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.655 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.655 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.655 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.655 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.655 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.655 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.655 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.655 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.656 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.656 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.656 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.656 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.656 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.656 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.656 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.656 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.656 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.656 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.656 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.656 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.656 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.656 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.656 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.657 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.657 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.657 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.657 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.657 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.657 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.657 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.657 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.657 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.657 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.657 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.657 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.657 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T05:33:39.657 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T05:33:39.657 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.657 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.657 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.657 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.658 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.658 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.658 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.658 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T05:33:39.658 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.658 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.658 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.658 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.658 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T05:33:39.658 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T05:33:39.658 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.658 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.658 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.658 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "*", 2024-03-28T05:33:39.658 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.658 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.658 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.659 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T05:33:39.659 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.659 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.659 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.659 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.659 INFO:teuthology.orchestra.run.smithi023.stdout: "max_count_per_host": { 2024-03-28T05:33:39.659 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_count_per_host", 2024-03-28T05:33:39.659 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.659 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.659 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.659 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:39.659 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.659 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.659 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.659 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of daemons per service per host", 2024-03-28T05:33:39.660 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.660 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.660 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.660 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.660 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd_draining_count": { 2024-03-28T05:33:39.660 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_osd_draining_count", 2024-03-28T05:33:39.660 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.660 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.660 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.660 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:39.660 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.660 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.660 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.660 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T05:33:39.660 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.660 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.660 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.661 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.661 INFO:teuthology.orchestra.run.smithi023.stdout: "migration_current": { 2024-03-28T05:33:39.661 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "migration_current", 2024-03-28T05:33:39.661 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.661 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.661 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.661 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.661 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.661 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.661 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.661 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "internal - do not modify", 2024-03-28T05:33:39.661 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.661 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.661 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.661 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.661 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-28T05:33:39.662 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-28T05:33:39.662 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.662 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.662 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.662 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "root", 2024-03-28T05:33:39.662 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.662 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.662 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.662 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm-package", 2024-03-28T05:33:39.662 INFO:teuthology.orchestra.run.smithi023.stdout: "root" 2024-03-28T05:33:39.662 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.662 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T05:33:39.662 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.662 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.662 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.662 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.663 INFO:teuthology.orchestra.run.smithi023.stdout: "oob_default_addr": { 2024-03-28T05:33:39.663 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "oob_default_addr", 2024-03-28T05:33:39.663 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.663 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.663 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.663 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "169.254.1.1", 2024-03-28T05:33:39.663 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.663 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.663 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.663 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T05:33:39.663 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.663 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.663 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.663 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.663 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_alerts_path": { 2024-03-28T05:33:39.663 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_alerts_path", 2024-03-28T05:33:39.664 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.664 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.664 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.664 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T05:33:39.664 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.664 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.664 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.664 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T05:33:39.664 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.664 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.664 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.664 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.664 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_insecure": { 2024-03-28T05:33:39.664 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_insecure", 2024-03-28T05:33:39.664 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.664 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.664 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.665 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.665 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.665 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.665 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.665 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T05:33:39.665 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.665 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.665 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.665 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.665 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_password": { 2024-03-28T05:33:39.665 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_password", 2024-03-28T05:33:39.665 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.665 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.665 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.665 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.665 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.665 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.665 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.666 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T05:33:39.666 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.666 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.666 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.666 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.666 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_url": { 2024-03-28T05:33:39.666 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_url", 2024-03-28T05:33:39.666 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.666 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.666 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.666 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.666 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.666 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.666 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.666 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T05:33:39.666 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.667 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.667 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.667 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.667 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_username": { 2024-03-28T05:33:39.667 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_username", 2024-03-28T05:33:39.667 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.667 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.667 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.667 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.667 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.667 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.667 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.667 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T05:33:39.667 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.667 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.667 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.668 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.668 INFO:teuthology.orchestra.run.smithi023.stdout: "secure_monitoring_stack": { 2024-03-28T05:33:39.668 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "secure_monitoring_stack", 2024-03-28T05:33:39.668 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.668 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.668 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.668 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.668 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.668 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.668 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.668 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T05:33:39.668 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.668 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.668 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.668 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.668 INFO:teuthology.orchestra.run.smithi023.stdout: "service_discovery_port": { 2024-03-28T05:33:39.668 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "service_discovery_port", 2024-03-28T05:33:39.668 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.669 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.669 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.669 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8765", 2024-03-28T05:33:39.669 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.669 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.669 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.669 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "cephadm service discovery port", 2024-03-28T05:33:39.669 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.669 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.669 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.669 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.669 INFO:teuthology.orchestra.run.smithi023.stdout: "ssh_config_file": { 2024-03-28T05:33:39.669 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssh_config_file", 2024-03-28T05:33:39.669 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.669 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.669 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.670 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.670 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.670 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.670 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.670 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T05:33:39.670 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.670 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.670 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.670 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.670 INFO:teuthology.orchestra.run.smithi023.stdout: "use_agent": { 2024-03-28T05:33:39.670 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_agent", 2024-03-28T05:33:39.670 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.670 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.670 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.670 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.670 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.671 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.671 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.671 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T05:33:39.671 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.671 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.671 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.671 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.671 INFO:teuthology.orchestra.run.smithi023.stdout: "use_repo_digest": { 2024-03-28T05:33:39.671 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_repo_digest", 2024-03-28T05:33:39.671 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.671 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.671 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.671 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.671 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.671 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.671 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.671 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T05:33:39.672 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.672 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.672 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.672 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.672 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_failed_host_check": { 2024-03-28T05:33:39.672 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_failed_host_check", 2024-03-28T05:33:39.672 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.672 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.672 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.672 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.672 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.672 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.672 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.672 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T05:33:39.672 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.672 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.672 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.672 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.673 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_daemons": { 2024-03-28T05:33:39.673 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_daemons", 2024-03-28T05:33:39.673 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.673 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.673 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.673 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.673 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.673 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.673 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.673 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T05:33:39.673 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.673 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.673 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.673 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.673 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_hosts": { 2024-03-28T05:33:39.673 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_hosts", 2024-03-28T05:33:39.673 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.673 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.674 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.674 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.674 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.674 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.674 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.674 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T05:33:39.674 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.674 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.674 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.674 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.674 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.674 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.674 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.674 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crash", 2024-03-28T05:33:39.674 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.674 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.674 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.675 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.675 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.675 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.675 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.675 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.675 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.675 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.675 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.675 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.675 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.675 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.675 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.675 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.675 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.675 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.675 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.675 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.675 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.676 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.676 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.676 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.676 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.676 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.676 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.676 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.676 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.676 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.676 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.676 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.676 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.676 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.676 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.676 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.676 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.676 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.677 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.677 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.677 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.677 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.677 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.677 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.677 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.677 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.677 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.677 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.677 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.677 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.677 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.677 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.677 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.677 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.678 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.678 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.678 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.678 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.678 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.678 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.678 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.678 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.678 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.678 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.678 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.678 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.678 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.678 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.678 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.678 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.679 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.679 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.679 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.679 INFO:teuthology.orchestra.run.smithi023.stdout: "retain_interval": { 2024-03-28T05:33:39.679 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retain_interval", 2024-03-28T05:33:39.679 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.679 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.679 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.679 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "31536000", 2024-03-28T05:33:39.679 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.679 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.679 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.679 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T05:33:39.679 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.679 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.679 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.680 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.680 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_recent_interval": { 2024-03-28T05:33:39.680 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_recent_interval", 2024-03-28T05:33:39.680 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.680 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.680 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.680 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1209600", 2024-03-28T05:33:39.680 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.680 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.680 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.680 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T05:33:39.680 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.680 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.680 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.680 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.681 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.681 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.681 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.681 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dashboard", 2024-03-28T05:33:39.681 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.681 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.681 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.681 INFO:teuthology.orchestra.run.smithi023.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T05:33:39.681 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T05:33:39.681 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.681 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.681 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.681 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:39.681 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.681 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.681 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.681 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.681 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.682 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.682 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.682 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.682 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T05:33:39.682 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T05:33:39.682 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.682 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.682 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.682 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.682 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.682 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.682 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.682 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.682 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.682 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.682 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.682 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.683 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T05:33:39.683 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T05:33:39.683 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.683 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.683 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.683 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.683 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.683 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.683 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.683 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.683 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.683 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.683 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.683 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.683 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_ENABLED": { 2024-03-28T05:33:39.683 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T05:33:39.683 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.684 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.684 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.684 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.684 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.684 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.684 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.684 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.684 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.684 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.684 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.684 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.684 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T05:33:39.684 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T05:33:39.684 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.684 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.684 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.684 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.685 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.685 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.685 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.685 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.685 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.685 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.685 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.685 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.685 INFO:teuthology.orchestra.run.smithi023.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T05:33:39.685 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T05:33:39.685 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.685 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.685 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.685 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.686 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.686 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.686 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.686 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.686 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.686 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.686 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.686 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.686 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T05:33:39.686 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T05:33:39.686 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.686 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.686 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.686 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.686 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.686 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.687 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.687 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.687 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.687 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.687 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.687 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.687 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T05:33:39.687 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T05:33:39.687 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.687 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.687 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.687 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.687 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.687 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.687 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.687 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.687 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.687 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.688 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.688 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.688 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T05:33:39.688 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T05:33:39.688 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.688 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.688 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.688 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.688 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.688 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.688 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.688 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.688 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.688 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.688 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.688 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.688 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T05:33:39.689 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T05:33:39.689 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.689 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.689 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.689 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.689 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.689 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.689 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.689 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.689 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.689 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.689 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.689 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.689 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T05:33:39.689 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T05:33:39.689 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.689 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.690 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.690 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.690 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.690 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.690 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.690 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.690 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.690 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.690 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.690 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.690 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T05:33:39.690 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T05:33:39.690 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.690 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.690 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.690 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.691 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.691 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.691 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.691 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.691 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.691 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.691 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.691 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.691 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T05:33:39.691 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T05:33:39.691 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.691 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.691 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.691 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.691 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.691 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.691 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.691 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.692 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.692 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.692 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.692 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.692 INFO:teuthology.orchestra.run.smithi023.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T05:33:39.692 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T05:33:39.692 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.692 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.692 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.692 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.692 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.692 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.692 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.692 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.692 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.693 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.693 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.693 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.693 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T05:33:39.693 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T05:33:39.693 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.693 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.693 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.693 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-28T05:33:39.693 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.693 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.693 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.693 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.693 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.693 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.693 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.693 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.694 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T05:33:39.694 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T05:33:39.694 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.694 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.694 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.694 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.694 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.694 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.694 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.694 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.694 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.694 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.694 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.694 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.694 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_URL": { 2024-03-28T05:33:39.694 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_URL", 2024-03-28T05:33:39.695 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.695 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.695 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.695 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.695 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.695 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.695 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.695 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.695 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.695 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.695 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.695 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.695 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T05:33:39.695 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T05:33:39.695 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.695 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.695 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.695 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-28T05:33:39.696 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.696 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.696 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.696 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.696 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.696 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.696 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.696 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.696 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T05:33:39.696 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T05:33:39.696 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.696 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.696 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.696 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.696 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.696 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.696 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.697 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.697 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.697 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.697 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.697 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.697 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T05:33:39.697 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T05:33:39.697 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.697 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.697 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.697 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.697 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.697 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.697 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.697 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.697 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.697 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.697 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.698 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.698 INFO:teuthology.orchestra.run.smithi023.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T05:33:39.698 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T05:33:39.698 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.698 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.698 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.698 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.698 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.698 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.698 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.698 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.698 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.698 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.698 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.698 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.698 INFO:teuthology.orchestra.run.smithi023.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T05:33:39.699 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T05:33:39.699 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.699 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.699 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.699 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.699 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.699 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.699 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.699 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.699 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.699 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.699 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.699 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.699 INFO:teuthology.orchestra.run.smithi023.stdout: "MULTICLUSTER_CONFIG": { 2024-03-28T05:33:39.699 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-28T05:33:39.699 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.700 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.700 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.700 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "{}", 2024-03-28T05:33:39.700 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.700 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.700 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.700 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.700 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.700 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.700 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.700 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.700 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T05:33:39.700 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T05:33:39.700 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.700 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.700 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.701 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.701 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.701 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.701 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.701 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.701 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.701 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.701 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.701 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.701 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T05:33:39.701 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T05:33:39.701 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.701 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.701 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.701 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.701 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.702 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.702 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.702 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.702 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.702 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.702 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.702 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.702 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T05:33:39.702 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T05:33:39.702 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.702 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.702 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.702 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.702 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.703 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.703 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.703 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.703 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.703 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.703 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.703 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.703 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T05:33:39.703 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T05:33:39.703 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.703 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.703 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.703 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.703 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.703 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.704 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.704 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.704 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.704 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.704 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.704 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.704 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T05:33:39.704 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T05:33:39.704 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.704 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.704 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.704 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.704 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.704 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.704 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.705 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.705 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.705 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.705 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.705 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.705 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T05:33:39.705 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T05:33:39.705 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.705 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.705 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.705 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.705 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.705 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.705 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.705 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.705 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.706 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.706 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.706 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.706 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T05:33:39.706 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T05:33:39.706 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.706 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.706 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.706 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.706 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.706 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.706 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.706 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.706 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.707 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.707 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.707 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.707 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T05:33:39.707 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T05:33:39.707 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.707 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.707 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.707 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.707 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.707 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.707 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.707 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.707 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.707 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.707 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.707 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.708 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T05:33:39.708 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T05:33:39.708 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.708 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.708 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.708 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.708 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.708 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.708 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.708 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.708 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.708 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.708 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.708 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.708 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T05:33:39.708 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T05:33:39.709 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.709 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.709 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.709 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.709 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.709 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.709 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.709 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.709 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.709 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.709 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.709 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.709 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T05:33:39.709 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T05:33:39.709 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.709 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.710 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.710 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T05:33:39.710 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.710 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.710 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.710 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.710 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.710 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.710 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.710 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.710 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T05:33:39.710 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T05:33:39.710 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.710 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.710 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.711 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:39.711 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.711 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.711 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.711 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.711 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.711 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.711 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.711 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.711 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T05:33:39.711 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T05:33:39.711 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.711 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.711 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.711 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8", 2024-03-28T05:33:39.711 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.711 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.712 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.712 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.712 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.712 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.712 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.712 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.712 INFO:teuthology.orchestra.run.smithi023.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T05:33:39.712 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T05:33:39.712 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.712 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.712 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.712 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "45", 2024-03-28T05:33:39.712 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.712 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.712 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.712 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.712 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.713 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.713 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.713 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.713 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T05:33:39.713 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T05:33:39.713 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.713 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.713 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.713 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.713 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.713 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.713 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.713 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.713 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.713 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.713 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.713 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.713 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T05:33:39.714 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T05:33:39.714 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.714 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.714 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.714 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-28T05:33:39.714 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.714 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.714 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.714 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.714 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.714 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.714 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.714 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.714 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T05:33:39.714 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T05:33:39.714 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.715 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.715 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.715 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.715 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.715 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.715 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.715 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.715 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.715 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.715 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.715 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.715 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T05:33:39.715 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T05:33:39.715 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.715 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.715 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.715 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.716 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.716 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.716 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.716 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.716 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.716 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.716 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.716 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.716 INFO:teuthology.orchestra.run.smithi023.stdout: "UNSAFE_TLS_v1_2": { 2024-03-28T05:33:39.716 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-28T05:33:39.716 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.716 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.716 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.716 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.716 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.716 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.717 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.717 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.717 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.717 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.717 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.717 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.717 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T05:33:39.717 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T05:33:39.717 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.717 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.717 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.717 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T05:33:39.717 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.717 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.717 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.717 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.717 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.718 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.718 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.718 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.718 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T05:33:39.718 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T05:33:39.718 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.718 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.718 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.718 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:39.718 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.718 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.718 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.718 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.718 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.718 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.718 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.719 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.719 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T05:33:39.719 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T05:33:39.719 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.719 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.719 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.719 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T05:33:39.719 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.719 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.719 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.719 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.719 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.719 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.719 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.719 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.719 INFO:teuthology.orchestra.run.smithi023.stdout: "cross_origin_url": { 2024-03-28T05:33:39.719 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cross_origin_url", 2024-03-28T05:33:39.720 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.720 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.720 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.720 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.720 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.720 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.720 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.720 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.720 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.720 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.720 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.720 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.720 INFO:teuthology.orchestra.run.smithi023.stdout: "crt_file": { 2024-03-28T05:33:39.720 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crt_file", 2024-03-28T05:33:39.720 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.720 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.720 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.720 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.721 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.721 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.721 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.721 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.721 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.721 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.721 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.721 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.721 INFO:teuthology.orchestra.run.smithi023.stdout: "debug": { 2024-03-28T05:33:39.721 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "debug", 2024-03-28T05:33:39.721 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.721 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.721 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.721 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.721 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.721 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.721 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.721 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable/disable debug options", 2024-03-28T05:33:39.722 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.722 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.722 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.722 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.722 INFO:teuthology.orchestra.run.smithi023.stdout: "jwt_token_ttl": { 2024-03-28T05:33:39.722 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "jwt_token_ttl", 2024-03-28T05:33:39.722 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.722 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.722 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.722 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "28800", 2024-03-28T05:33:39.722 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.722 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.722 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.722 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.722 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.722 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.723 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.723 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.723 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-28T05:33:39.723 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-28T05:33:39.723 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.723 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.723 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.723 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.723 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.723 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.723 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.723 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.723 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.723 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.723 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.723 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.723 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.724 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.724 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.724 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.724 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.724 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.724 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.724 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.724 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.724 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.724 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.724 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.724 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.724 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.724 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.724 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.724 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.724 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.724 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.725 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.725 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.725 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.725 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.725 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.725 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.725 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.725 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.725 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.725 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.725 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.725 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.725 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.725 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.725 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.725 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.725 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.726 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.726 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.726 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.726 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.726 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.726 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.726 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.726 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.726 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.726 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.726 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.726 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.726 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.726 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.726 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.726 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.726 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.727 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.727 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.727 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.727 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.727 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.727 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.727 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.727 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.727 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.727 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.727 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.727 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.727 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.727 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.727 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.727 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.727 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.727 INFO:teuthology.orchestra.run.smithi023.stdout: "motd": { 2024-03-28T05:33:39.728 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "motd", 2024-03-28T05:33:39.728 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.728 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.728 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.728 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.728 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.728 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.728 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.728 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "The message of the day", 2024-03-28T05:33:39.728 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.728 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.728 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.728 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.728 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect_resolve_ip_addr": { 2024-03-28T05:33:39.728 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T05:33:39.728 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.728 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.729 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.729 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.729 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.729 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.729 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.729 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.729 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.729 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.729 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.729 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.729 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-28T05:33:39.729 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-28T05:33:39.729 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.729 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.729 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.729 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-28T05:33:39.729 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.730 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.730 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.730 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.730 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.730 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.730 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.730 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.730 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-28T05:33:39.730 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-28T05:33:39.730 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.730 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.730 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.730 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8080", 2024-03-28T05:33:39.730 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.730 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.730 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.730 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.731 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.731 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.731 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.731 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.731 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-28T05:33:39.731 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-28T05:33:39.731 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.731 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.731 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.731 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.731 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.731 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.731 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.731 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.731 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.731 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.731 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.732 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.732 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl_server_port": { 2024-03-28T05:33:39.732 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl_server_port", 2024-03-28T05:33:39.732 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.732 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.732 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.732 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8443", 2024-03-28T05:33:39.732 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.732 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.732 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.732 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.732 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.732 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.732 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.732 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.732 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-28T05:33:39.732 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-28T05:33:39.732 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.733 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.733 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.733 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "redirect", 2024-03-28T05:33:39.733 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.733 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.733 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.733 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.733 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect" 2024-03-28T05:33:39.733 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.733 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.733 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.733 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.733 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.733 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.733 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-28T05:33:39.733 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-28T05:33:39.733 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.734 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.734 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.734 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-28T05:33:39.734 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-28T05:33:39.734 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-28T05:33:39.734 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.734 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.734 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.734 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.734 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.734 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.734 INFO:teuthology.orchestra.run.smithi023.stdout: "url_prefix": { 2024-03-28T05:33:39.734 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url_prefix", 2024-03-28T05:33:39.734 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.734 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.734 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.734 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.735 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.735 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.735 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.735 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.735 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.735 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.735 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.735 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.735 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.735 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.735 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.735 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2024-03-28T05:33:39.735 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.735 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.735 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.735 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_monitoring": { 2024-03-28T05:33:39.735 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_monitoring", 2024-03-28T05:33:39.735 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.736 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.736 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.736 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.736 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.736 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.736 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.736 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "monitor device health metrics", 2024-03-28T05:33:39.736 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.736 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.736 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.736 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.736 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.736 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.736 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.736 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.736 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.737 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.737 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.737 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.737 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.737 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.737 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.737 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.737 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.737 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.737 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.737 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.737 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.737 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.737 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.737 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.738 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.738 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.738 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.738 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.738 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.738 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.738 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.738 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.738 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.738 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.738 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.738 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.738 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.738 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.738 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.738 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.738 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.739 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.739 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.739 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.739 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.739 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.739 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.739 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.739 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.739 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.739 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.739 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.739 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.739 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.739 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.739 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.739 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.739 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.740 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.740 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.740 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.740 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.740 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.740 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.740 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.740 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.740 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.740 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.740 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.740 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.740 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.740 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.740 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.740 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.740 INFO:teuthology.orchestra.run.smithi023.stdout: "mark_out_threshold": { 2024-03-28T05:33:39.741 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mark_out_threshold", 2024-03-28T05:33:39.741 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.741 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.741 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.741 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2419200", 2024-03-28T05:33:39.741 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.741 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.741 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.741 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T05:33:39.741 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.741 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.741 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.741 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.741 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": { 2024-03-28T05:33:39.741 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_name", 2024-03-28T05:33:39.741 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.741 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.741 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.742 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "device_health_metrics", 2024-03-28T05:33:39.742 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.742 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.742 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.742 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T05:33:39.742 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.742 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.742 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.742 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.742 INFO:teuthology.orchestra.run.smithi023.stdout: "retention_period": { 2024-03-28T05:33:39.742 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retention_period", 2024-03-28T05:33:39.742 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.742 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.742 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.742 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15552000", 2024-03-28T05:33:39.742 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.742 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.743 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.743 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain device health metrics", 2024-03-28T05:33:39.743 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.743 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.743 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.743 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.743 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_frequency": { 2024-03-28T05:33:39.743 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_frequency", 2024-03-28T05:33:39.743 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.743 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.743 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.743 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-28T05:33:39.743 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.743 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.743 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.743 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T05:33:39.743 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.743 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.744 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.744 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.744 INFO:teuthology.orchestra.run.smithi023.stdout: "self_heal": { 2024-03-28T05:33:39.744 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "self_heal", 2024-03-28T05:33:39.744 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.744 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.744 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.744 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.744 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.744 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.744 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.744 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T05:33:39.744 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.744 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.744 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.744 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.744 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T05:33:39.745 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T05:33:39.745 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.745 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.745 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.745 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T05:33:39.745 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.745 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.745 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.745 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T05:33:39.745 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.745 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.745 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.745 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.745 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_threshold": { 2024-03-28T05:33:39.745 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_threshold", 2024-03-28T05:33:39.745 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.745 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.746 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.746 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7257600", 2024-03-28T05:33:39.746 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.746 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.746 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.746 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T05:33:39.746 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.746 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.746 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.746 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.746 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.746 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.746 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.746 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "diskprediction_local", 2024-03-28T05:33:39.746 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.746 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.746 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.747 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.747 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.747 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.747 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.747 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.747 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.747 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.747 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.747 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.747 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.747 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.747 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.747 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.747 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.747 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.747 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.747 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.748 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.748 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.748 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.748 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.748 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.748 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.748 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.748 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.748 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.748 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.748 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.748 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.748 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.748 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.748 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.748 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.748 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.748 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.749 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.749 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.749 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.749 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.749 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.749 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.749 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.749 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.749 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.749 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.749 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.749 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.749 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.749 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.749 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.749 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.750 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.750 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.750 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.750 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.750 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.750 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.750 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.750 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.750 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.750 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.750 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.750 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.750 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.750 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.750 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.750 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.750 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.751 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.751 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.751 INFO:teuthology.orchestra.run.smithi023.stdout: "predict_interval": { 2024-03-28T05:33:39.751 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predict_interval", 2024-03-28T05:33:39.751 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.751 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.751 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.751 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-28T05:33:39.751 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.751 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.751 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.751 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.751 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.751 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.751 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.751 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.751 INFO:teuthology.orchestra.run.smithi023.stdout: "predictor_model": { 2024-03-28T05:33:39.751 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predictor_model", 2024-03-28T05:33:39.752 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.752 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.752 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.752 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "prophetstor", 2024-03-28T05:33:39.752 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.752 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.752 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.752 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.752 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.752 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.752 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.752 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.752 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T05:33:39.752 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T05:33:39.752 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.752 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.752 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.752 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T05:33:39.753 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.753 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.753 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.753 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.753 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.753 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.753 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.753 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.753 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.753 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.753 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.753 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "influx", 2024-03-28T05:33:39.753 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": false, 2024-03-28T05:33:39.753 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "influxdb python module not found", 2024-03-28T05:33:39.753 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.753 INFO:teuthology.orchestra.run.smithi023.stdout: "batch_size": { 2024-03-28T05:33:39.753 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "batch_size", 2024-03-28T05:33:39.754 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.754 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.754 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.754 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5000", 2024-03-28T05:33:39.754 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.754 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.754 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.754 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T05:33:39.754 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.754 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.754 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.754 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.754 INFO:teuthology.orchestra.run.smithi023.stdout: "database": { 2024-03-28T05:33:39.754 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "database", 2024-03-28T05:33:39.754 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.754 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.754 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.755 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "ceph", 2024-03-28T05:33:39.755 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.755 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.755 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.755 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T05:33:39.755 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.755 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.755 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.755 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.755 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": { 2024-03-28T05:33:39.755 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hostname", 2024-03-28T05:33:39.755 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.755 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.755 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.755 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.755 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.756 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.756 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.756 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server hostname", 2024-03-28T05:33:39.756 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.756 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.756 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.756 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.756 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T05:33:39.756 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T05:33:39.756 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.756 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.756 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.756 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "30", 2024-03-28T05:33:39.756 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "5", 2024-03-28T05:33:39.756 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.756 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.757 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T05:33:39.757 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.757 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.757 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.757 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.757 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.757 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.757 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.757 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.757 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.757 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.757 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.757 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.757 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.757 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.757 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.757 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.757 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.758 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.758 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.758 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.758 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.758 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.758 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.758 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.758 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.758 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.758 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.758 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.758 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.758 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.758 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.758 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.758 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.759 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.759 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.759 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.759 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.759 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.759 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.759 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.759 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.759 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.759 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.759 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.759 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.759 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.759 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.759 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.759 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.759 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.760 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.760 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.760 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.760 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.760 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.760 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.760 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.760 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.760 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.760 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.760 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.760 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.760 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.760 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.760 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.761 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.761 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.761 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.761 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.761 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.761 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.761 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.761 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.761 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.761 INFO:teuthology.orchestra.run.smithi023.stdout: "password": { 2024-03-28T05:33:39.761 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "password", 2024-03-28T05:33:39.761 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.761 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.761 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.761 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.761 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.761 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.762 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.762 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "password of InfluxDB server user", 2024-03-28T05:33:39.762 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.762 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.762 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.762 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.762 INFO:teuthology.orchestra.run.smithi023.stdout: "port": { 2024-03-28T05:33:39.762 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "port", 2024-03-28T05:33:39.762 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.762 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.762 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.762 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8086", 2024-03-28T05:33:39.762 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.762 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.762 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.763 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server port", 2024-03-28T05:33:39.763 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.763 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.763 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.763 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.763 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-28T05:33:39.763 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-28T05:33:39.763 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.763 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.763 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.763 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "false", 2024-03-28T05:33:39.763 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.763 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.763 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.763 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T05:33:39.763 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.763 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.763 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.764 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.764 INFO:teuthology.orchestra.run.smithi023.stdout: "threads": { 2024-03-28T05:33:39.764 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threads", 2024-03-28T05:33:39.764 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.764 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.764 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.764 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T05:33:39.764 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T05:33:39.764 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "32", 2024-03-28T05:33:39.764 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.764 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T05:33:39.764 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.764 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.764 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.764 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.764 INFO:teuthology.orchestra.run.smithi023.stdout: "username": { 2024-03-28T05:33:39.765 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "username", 2024-03-28T05:33:39.765 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.765 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.765 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.765 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.765 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.765 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.765 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.765 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "username of InfluxDB server user", 2024-03-28T05:33:39.765 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.765 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.765 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.765 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.765 INFO:teuthology.orchestra.run.smithi023.stdout: "verify_ssl": { 2024-03-28T05:33:39.765 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "verify_ssl", 2024-03-28T05:33:39.765 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.765 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.765 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.766 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "true", 2024-03-28T05:33:39.766 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.766 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.766 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.766 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T05:33:39.766 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.766 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.766 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.766 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.766 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.766 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.766 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.766 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "insights", 2024-03-28T05:33:39.766 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.766 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.766 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.766 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.767 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.767 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.767 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.767 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.767 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.767 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.767 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.767 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.767 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.767 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.767 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.767 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.767 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.767 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.767 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.767 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.768 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.768 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.768 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.768 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.768 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.768 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.768 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.768 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.768 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.768 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.768 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.768 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.768 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.768 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.768 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.768 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.768 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.768 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.769 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.769 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.769 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.769 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.769 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.769 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.769 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.769 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.769 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.769 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.769 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.769 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.769 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.769 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.769 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.769 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.769 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.770 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.770 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.770 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.770 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.770 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.770 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.770 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.770 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.770 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.770 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.770 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.770 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.770 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.770 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.770 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.770 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.771 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.771 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.771 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.771 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.771 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.771 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "iostat", 2024-03-28T05:33:39.771 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.771 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.771 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.771 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.771 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.771 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.771 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.771 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.771 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.771 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.771 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.772 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.772 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.772 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.772 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.772 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.772 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.772 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.772 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.772 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.772 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.772 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.772 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.772 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.772 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.772 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.772 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.772 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.773 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.773 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.773 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.773 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.773 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.773 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.773 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.773 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.773 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.773 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.773 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.773 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.773 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.773 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.773 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.773 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.773 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.773 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.774 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.774 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.774 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.774 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.774 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.774 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.774 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.774 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.774 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.774 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.774 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.774 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.774 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.774 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.774 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.774 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.774 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.774 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.775 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.775 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.775 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.775 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.775 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.775 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.775 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.775 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.775 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.775 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.775 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.775 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.775 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "k8sevents", 2024-03-28T05:33:39.775 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.775 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.775 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.775 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_event_retention_days": { 2024-03-28T05:33:39.776 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ceph_event_retention_days", 2024-03-28T05:33:39.776 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.776 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.776 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.776 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7", 2024-03-28T05:33:39.776 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.776 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.776 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.776 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T05:33:39.776 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.776 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.776 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.776 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.776 INFO:teuthology.orchestra.run.smithi023.stdout: "config_check_secs": { 2024-03-28T05:33:39.776 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_check_secs", 2024-03-28T05:33:39.776 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.776 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.776 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.777 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:39.777 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "10", 2024-03-28T05:33:39.777 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.777 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.777 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T05:33:39.777 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.777 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.777 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.777 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.777 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.777 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.777 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.777 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.777 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.777 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.777 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.777 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.777 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.778 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.778 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.778 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.778 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.778 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.778 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.778 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.778 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.778 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.778 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.778 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.778 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.778 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.778 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.778 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.778 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.779 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.779 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.779 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.779 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.779 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.779 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.779 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.779 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.779 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.779 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.779 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.779 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.779 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.779 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.779 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.779 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.780 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.780 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.780 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.780 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.780 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.780 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.780 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.780 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.780 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.780 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.780 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.780 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.780 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.780 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.780 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.780 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.781 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.781 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.781 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.781 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.781 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.781 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.781 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.781 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.781 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.781 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.781 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.781 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.781 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.781 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.781 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.781 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.781 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "localpool", 2024-03-28T05:33:39.781 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.782 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.782 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.782 INFO:teuthology.orchestra.run.smithi023.stdout: "failure_domain": { 2024-03-28T05:33:39.782 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "failure_domain", 2024-03-28T05:33:39.782 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.782 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.782 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.782 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "host", 2024-03-28T05:33:39.782 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.782 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.782 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.782 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "failure domain for any created local pool", 2024-03-28T05:33:39.782 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T05:33:39.782 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.782 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.782 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.782 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.782 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.783 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.783 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.783 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.783 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.783 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.783 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.783 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.783 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.783 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.783 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.783 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.783 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.783 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.783 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.783 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.783 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.783 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.783 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.784 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.784 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.784 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.784 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.784 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.784 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.784 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.784 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.784 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.784 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.784 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.784 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.784 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.784 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.784 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.784 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.784 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.785 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.785 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.785 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.785 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.785 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.785 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.785 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.785 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.785 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.785 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.785 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.785 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.785 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.785 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.785 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.785 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.785 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.785 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.786 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.786 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.786 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.786 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.786 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.786 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.786 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.786 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.786 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.786 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.786 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.786 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.786 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.786 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.786 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.786 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": { 2024-03-28T05:33:39.786 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_size", 2024-03-28T05:33:39.786 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.787 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.787 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.787 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.787 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.787 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.787 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.787 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default min_size for any created local pool", 2024-03-28T05:33:39.787 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T05:33:39.787 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.787 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.787 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.787 INFO:teuthology.orchestra.run.smithi023.stdout: "num_rep": { 2024-03-28T05:33:39.787 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "num_rep", 2024-03-28T05:33:39.787 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.787 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.787 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.787 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3", 2024-03-28T05:33:39.788 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.788 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.788 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.788 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default replica count for any created local pool", 2024-03-28T05:33:39.788 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.788 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.788 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.788 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.788 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": { 2024-03-28T05:33:39.788 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_num", 2024-03-28T05:33:39.788 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.788 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.788 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.788 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "128", 2024-03-28T05:33:39.788 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.788 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.788 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.788 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T05:33:39.789 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.789 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.789 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.789 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.789 INFO:teuthology.orchestra.run.smithi023.stdout: "prefix": { 2024-03-28T05:33:39.789 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prefix", 2024-03-28T05:33:39.789 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.789 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.789 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.789 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.789 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.789 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.789 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.790 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name prefix for any created local pool", 2024-03-28T05:33:39.790 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.790 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.790 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.790 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.790 INFO:teuthology.orchestra.run.smithi023.stdout: "subtree": { 2024-03-28T05:33:39.790 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "subtree", 2024-03-28T05:33:39.790 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.790 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.790 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.790 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "rack", 2024-03-28T05:33:39.790 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.790 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.790 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.790 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T05:33:39.791 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T05:33:39.791 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.791 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.791 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.791 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.791 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.791 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.791 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mds_autoscaler", 2024-03-28T05:33:39.791 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.791 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.791 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.791 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.791 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.791 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.791 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.791 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.791 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.791 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.792 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.792 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.792 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.792 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.792 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.792 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.792 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.792 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.792 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.792 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.792 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.792 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.792 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.792 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.792 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.792 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.792 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.793 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.793 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.793 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.793 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.793 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.793 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.793 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.793 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.793 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.793 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.793 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.793 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.793 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.793 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.793 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.793 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.793 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.794 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.794 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.794 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.794 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.794 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.794 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.794 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.794 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.794 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.794 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.794 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.794 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.794 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.794 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.794 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.794 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.794 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.795 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.795 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.795 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.795 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.795 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.795 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.795 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.795 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.795 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.795 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.795 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.795 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.795 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.795 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.795 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.795 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirroring", 2024-03-28T05:33:39.795 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.795 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.796 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.796 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.796 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.796 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.796 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.796 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.796 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.796 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.796 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.796 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.796 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.796 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.796 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.796 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.796 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.797 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.797 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.797 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.797 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.797 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.797 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.797 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.797 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.797 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.797 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.797 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.797 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.797 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.797 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.797 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.797 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.797 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.797 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.798 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.798 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.798 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.798 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.798 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.798 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.798 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.798 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.798 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.798 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.798 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.798 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.798 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.798 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.798 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.798 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.798 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.798 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.799 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.799 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.799 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.799 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.799 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.799 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.799 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.799 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.799 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.799 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.799 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.799 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.799 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.799 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.799 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.799 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.799 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.800 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.800 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.800 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.800 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.800 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.800 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.800 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "nfs", 2024-03-28T05:33:39.800 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.800 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.800 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.800 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.800 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.800 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.800 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.800 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.800 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.800 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.801 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.801 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.801 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.801 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.801 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.801 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.801 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.801 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.801 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.801 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.801 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.801 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.801 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.801 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.801 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.801 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.801 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.802 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.802 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.802 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.802 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.802 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.802 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.802 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.802 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.802 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.802 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.802 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.802 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.802 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.802 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.802 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.802 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.802 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.803 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.803 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.803 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.803 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.803 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.803 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.803 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.803 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.803 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.803 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.803 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.803 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.803 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.803 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.803 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.803 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.803 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.804 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.804 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.804 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.804 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.804 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.804 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.804 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.804 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.804 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.804 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.804 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.804 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.804 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.804 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.804 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.804 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-28T05:33:39.804 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.804 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.805 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.805 INFO:teuthology.orchestra.run.smithi023.stdout: "fail_fs": { 2024-03-28T05:33:39.805 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "fail_fs", 2024-03-28T05:33:39.805 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.805 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.805 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.805 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.805 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.805 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.805 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.805 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T05:33:39.805 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.805 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.805 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.805 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.805 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.805 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.806 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.807 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.808 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.809 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator": { 2024-03-28T05:33:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-28T05:33:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2024-03-28T05:33:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: "rook", 2024-03-28T05:33:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: "test_orchestrator" 2024-03-28T05:33:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Orchestrator backend", 2024-03-28T05:33:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.810 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_perf_query", 2024-03-28T05:33:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.811 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.812 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.813 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.814 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_support", 2024-03-28T05:33:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.815 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.816 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.817 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.817 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.817 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.817 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.817 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.817 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.817 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.817 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.817 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.817 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.817 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.817 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.817 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.817 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.817 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.817 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.818 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.818 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.818 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.818 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.818 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.818 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.818 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.818 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.818 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.818 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.818 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.818 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.818 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.818 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.818 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.818 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.818 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.819 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.819 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.819 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.819 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.819 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.819 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.819 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.819 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.819 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.819 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.819 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.819 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.819 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.819 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.819 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.819 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.819 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.820 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.820 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.820 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.820 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.820 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.820 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.820 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_autoscaler", 2024-03-28T05:33:39.820 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.820 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.820 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.820 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.820 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.820 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.820 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.820 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.820 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.820 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.820 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.821 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.821 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.821 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.821 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.821 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.821 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.821 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.821 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.821 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.821 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.821 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.821 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.821 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.821 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.821 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.821 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.821 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.822 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.822 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.822 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.822 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.822 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.822 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.822 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.822 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.822 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.822 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.822 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.822 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.822 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.822 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.822 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.822 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.823 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.823 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.823 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.823 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.823 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.823 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.823 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.823 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.823 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.823 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.823 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.823 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.823 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.823 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.823 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.823 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.823 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.824 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.824 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.824 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.824 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.824 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.824 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.824 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.824 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.824 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.824 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.824 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.824 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.824 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T05:33:39.824 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T05:33:39.824 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.824 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.825 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.825 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T05:33:39.825 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.825 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.825 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.825 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.825 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.825 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.825 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.825 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.825 INFO:teuthology.orchestra.run.smithi023.stdout: "threshold": { 2024-03-28T05:33:39.825 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threshold", 2024-03-28T05:33:39.825 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T05:33:39.825 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.825 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.825 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-28T05:33:39.825 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1.0", 2024-03-28T05:33:39.826 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.826 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.826 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "scaling threshold", 2024-03-28T05:33:39.826 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T05:33:39.826 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.826 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.826 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.826 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.826 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.826 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.826 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "progress", 2024-03-28T05:33:39.826 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.826 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.826 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.827 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_pg_recovery_event": { 2024-03-28T05:33:39.827 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_pg_recovery_event", 2024-03-28T05:33:39.827 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.827 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.827 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.827 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.827 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.827 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.827 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.827 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T05:33:39.827 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.827 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.827 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.827 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.827 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-28T05:33:39.827 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-28T05:33:39.828 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.828 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.828 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.828 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.828 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.828 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.828 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.828 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.828 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.828 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.828 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.828 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.828 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.828 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.828 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.829 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.829 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.829 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.829 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.829 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.829 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.829 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.829 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.829 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.829 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.829 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.829 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.829 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.829 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.829 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.829 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.829 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.830 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.830 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.830 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.830 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.830 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.830 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.830 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.830 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.830 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.830 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.830 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.830 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.830 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.830 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.830 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.831 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.831 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.831 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.831 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.831 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.831 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.831 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.831 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.831 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.831 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.831 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.831 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.831 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.831 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.831 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.831 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.832 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: "max_completed_events": { 2024-03-28T05:33:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_completed_events", 2024-03-28T05:33:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "50", 2024-03-28T05:33:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "number of past completed events to remember", 2024-03-28T05:33:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T05:33:39.833 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T05:33:39.834 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.834 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.834 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.834 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T05:33:39.834 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.834 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.834 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.834 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long the module is going to sleep", 2024-03-28T05:33:39.834 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.834 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.834 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.834 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.834 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.834 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.834 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.834 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus", 2024-03-28T05:33:39.835 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.835 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.835 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.835 INFO:teuthology.orchestra.run.smithi023.stdout: "cache": { 2024-03-28T05:33:39.835 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cache", 2024-03-28T05:33:39.835 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.835 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.835 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.835 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.835 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.835 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.835 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.835 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.835 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.835 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.835 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.835 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.836 INFO:teuthology.orchestra.run.smithi023.stdout: "exclude_perf_counters": { 2024-03-28T05:33:39.836 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "exclude_perf_counters", 2024-03-28T05:33:39.836 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.836 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.836 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.836 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.836 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.836 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.836 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.836 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T05:33:39.836 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T05:33:39.836 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.836 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.836 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.836 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.836 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.836 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.837 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.837 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.837 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.837 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.837 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.837 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.837 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.837 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.837 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.837 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.837 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.837 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.837 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.837 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.838 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.838 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.838 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.838 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.838 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.838 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.838 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.838 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.838 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.838 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.838 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.838 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.838 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.838 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.838 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.839 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.839 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.839 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.839 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.839 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.839 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.839 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.839 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.839 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.839 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.839 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.839 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.839 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.839 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.839 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.839 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.840 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.840 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.840 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.840 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.840 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.840 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.840 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.840 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.840 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.840 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.840 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.840 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.840 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.841 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.841 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.841 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.841 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.841 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.841 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.841 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.841 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.841 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.841 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools": { 2024-03-28T05:33:39.841 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools", 2024-03-28T05:33:39.841 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.841 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.841 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.841 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.841 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.841 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.841 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.842 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.842 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.842 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.842 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.842 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.842 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T05:33:39.842 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T05:33:39.842 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.842 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.842 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.842 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "300", 2024-03-28T05:33:39.842 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.842 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.842 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.842 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.843 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.843 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.843 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.843 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.843 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_interval": { 2024-03-28T05:33:39.843 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_interval", 2024-03-28T05:33:39.843 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T05:33:39.843 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.843 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.843 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15.0", 2024-03-28T05:33:39.843 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.843 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.843 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.843 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.843 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.843 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.843 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.843 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.844 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-28T05:33:39.844 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-28T05:33:39.844 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.844 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.844 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.844 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-28T05:33:39.844 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.844 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.844 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.844 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T05:33:39.844 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.844 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.844 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.844 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.844 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-28T05:33:39.844 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-28T05:33:39.845 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.845 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.845 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.845 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "9283", 2024-03-28T05:33:39.845 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.845 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.845 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.845 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T05:33:39.845 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.845 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.845 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.845 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.845 INFO:teuthology.orchestra.run.smithi023.stdout: "stale_cache_strategy": { 2024-03-28T05:33:39.845 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stale_cache_strategy", 2024-03-28T05:33:39.845 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.845 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.845 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.846 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "log", 2024-03-28T05:33:39.846 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.846 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.846 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.846 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.846 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.846 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.846 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.846 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.846 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-28T05:33:39.846 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-28T05:33:39.846 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.846 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.846 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.846 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "default", 2024-03-28T05:33:39.847 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.847 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.847 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.847 INFO:teuthology.orchestra.run.smithi023.stdout: "default", 2024-03-28T05:33:39.847 INFO:teuthology.orchestra.run.smithi023.stdout: "error" 2024-03-28T05:33:39.847 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.847 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.847 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.847 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.847 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.847 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.847 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-28T05:33:39.847 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-28T05:33:39.847 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.847 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.847 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.848 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-28T05:33:39.848 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-28T05:33:39.848 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-28T05:33:39.848 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.848 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.848 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.848 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.848 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.848 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.848 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.848 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.848 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.848 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2024-03-28T05:33:39.848 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.848 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.848 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.848 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.849 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.849 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.849 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.849 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.849 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.849 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.849 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.849 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.849 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.849 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.849 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.849 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.849 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.849 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.849 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.850 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.850 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.850 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.850 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.850 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.850 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.850 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.850 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.850 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.850 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.850 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.850 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.850 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.850 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.850 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.851 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.851 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.851 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.851 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.851 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.851 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.851 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.851 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.851 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.851 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.851 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.851 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.851 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.851 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.851 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.852 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.852 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.852 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.852 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.852 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.852 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.852 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.852 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.852 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.852 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.852 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.852 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.852 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.852 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.852 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.852 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.852 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.853 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.853 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.853 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.853 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.853 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.853 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.853 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.853 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_snap_create": { 2024-03-28T05:33:39.853 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_snap_create", 2024-03-28T05:33:39.853 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.853 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.853 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.853 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:39.853 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.853 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.853 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.853 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.854 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.854 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.854 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.854 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.854 INFO:teuthology.orchestra.run.smithi023.stdout: "mirror_snapshot_schedule": { 2024-03-28T05:33:39.854 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T05:33:39.854 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.854 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.854 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.854 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.854 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.854 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.854 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.854 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.854 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.854 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.855 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.855 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.855 INFO:teuthology.orchestra.run.smithi023.stdout: "trash_purge_schedule": { 2024-03-28T05:33:39.855 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "trash_purge_schedule", 2024-03-28T05:33:39.855 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.855 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.855 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.855 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.855 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.855 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.855 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.855 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.855 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.855 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.855 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.855 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.855 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.856 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.856 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.856 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "restful", 2024-03-28T05:33:39.856 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.856 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.856 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.856 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_auth": { 2024-03-28T05:33:39.856 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_auth", 2024-03-28T05:33:39.856 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.856 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.856 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.856 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.856 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.856 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.856 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.856 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.856 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.857 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.857 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.857 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.857 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-28T05:33:39.857 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-28T05:33:39.857 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.857 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.857 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.857 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.857 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.857 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.857 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.857 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.857 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.857 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.857 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.858 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.858 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.858 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.858 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.858 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.858 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.858 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.858 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.858 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.858 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.858 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.858 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.858 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.858 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.858 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.858 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.858 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.859 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.859 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.859 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.859 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.859 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.859 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.859 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.859 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.859 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.859 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.859 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.859 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.859 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.859 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.859 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.859 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.860 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.860 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.860 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.860 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.860 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.860 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.860 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.860 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.860 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.860 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.860 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.860 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.860 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.860 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.860 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.860 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.861 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.861 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.861 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.861 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.861 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.861 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.861 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.861 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.861 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.861 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.861 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.861 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.861 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.861 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.861 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.862 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.862 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.862 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.862 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.862 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.862 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.862 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.862 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-28T05:33:39.862 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-28T05:33:39.862 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.862 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.862 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.862 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.862 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.862 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.862 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.863 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.863 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.863 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.863 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.863 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.863 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-28T05:33:39.863 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-28T05:33:39.863 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.863 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.863 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.863 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.863 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.863 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.863 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.863 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.863 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.863 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rgw", 2024-03-28T05:33:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.864 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.865 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.865 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.865 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.865 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.865 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.865 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.865 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.865 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.865 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.865 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.865 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.865 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.865 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.865 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.865 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.865 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.865 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.866 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.866 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.866 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.866 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.866 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.866 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.866 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.866 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.866 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.866 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.866 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.866 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.866 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.866 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.866 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.866 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.867 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.867 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.867 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.867 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.867 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.867 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.867 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.867 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.867 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.867 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.867 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.867 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.867 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.867 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.867 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.867 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.867 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.868 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.868 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.868 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.868 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.868 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.868 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.868 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.868 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.868 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.868 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.868 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.868 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.868 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.868 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.868 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.868 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rook", 2024-03-28T05:33:39.868 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.869 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.869 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.869 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.869 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.869 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.869 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.869 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.869 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.869 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.869 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.869 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.869 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.869 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.869 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.869 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.869 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.869 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.869 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.870 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.870 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.870 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.870 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.870 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.870 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.870 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.870 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.870 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.870 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.870 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.870 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.870 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.870 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.870 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.870 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.871 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.871 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.871 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.871 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.871 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.871 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.871 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.871 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.871 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.871 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.871 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.871 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.871 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.871 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.871 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.871 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.872 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.872 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.872 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.872 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.872 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.872 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.872 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.872 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.872 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.872 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.872 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.872 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.872 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.872 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.872 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.872 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.873 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.873 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.873 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.873 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.873 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.873 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.873 INFO:teuthology.orchestra.run.smithi023.stdout: "storage_class": { 2024-03-28T05:33:39.873 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "storage_class", 2024-03-28T05:33:39.873 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.873 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.873 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.873 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "local", 2024-03-28T05:33:39.873 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.873 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.873 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.873 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T05:33:39.873 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.874 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.874 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.874 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.874 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.874 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.874 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.874 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "selftest", 2024-03-28T05:33:39.874 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.874 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.874 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.874 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.874 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.874 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.874 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.874 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.874 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.875 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.875 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.875 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.875 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.875 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.875 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.875 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.875 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.875 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.875 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.875 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.875 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.875 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.875 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.875 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.875 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.875 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.876 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.876 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.876 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.876 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.876 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.876 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.876 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.876 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.876 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.876 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.876 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.876 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.876 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.876 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.876 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.876 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.876 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.876 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.877 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.877 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.877 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.877 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.877 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.877 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.877 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.877 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.877 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.877 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.877 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.877 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.877 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.877 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.877 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.877 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.877 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.878 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.878 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.878 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.878 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.878 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.878 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.878 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.878 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.878 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.878 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.878 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.878 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.878 INFO:teuthology.orchestra.run.smithi023.stdout: "roption1": { 2024-03-28T05:33:39.878 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption1", 2024-03-28T05:33:39.878 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.878 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.878 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.879 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.879 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.879 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.879 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.879 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.879 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.879 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.879 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.879 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.879 INFO:teuthology.orchestra.run.smithi023.stdout: "roption2": { 2024-03-28T05:33:39.879 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption2", 2024-03-28T05:33:39.879 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.879 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.879 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.879 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "xyz", 2024-03-28T05:33:39.879 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.880 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.880 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.880 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.880 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.880 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.880 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.880 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.880 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption1": { 2024-03-28T05:33:39.880 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption1", 2024-03-28T05:33:39.880 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.880 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.880 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.880 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.880 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.880 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.880 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.880 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.881 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.881 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.881 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.881 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.881 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption2": { 2024-03-28T05:33:39.881 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption2", 2024-03-28T05:33:39.881 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.881 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.881 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.881 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.881 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.881 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.881 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.881 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.881 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.881 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.881 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.881 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.882 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption3": { 2024-03-28T05:33:39.882 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption3", 2024-03-28T05:33:39.882 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T05:33:39.882 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.882 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.882 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.882 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.882 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.882 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.882 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.882 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.882 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.882 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.882 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.882 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption4": { 2024-03-28T05:33:39.882 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption4", 2024-03-28T05:33:39.883 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.883 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.883 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.883 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.883 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.883 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.883 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.883 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.883 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.883 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.883 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.883 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.883 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption5": { 2024-03-28T05:33:39.883 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption5", 2024-03-28T05:33:39.883 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.883 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.883 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.884 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.884 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.884 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.884 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.884 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.884 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.884 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.884 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.884 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.884 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption6": { 2024-03-28T05:33:39.884 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption6", 2024-03-28T05:33:39.884 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.884 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.884 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.884 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.884 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.884 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.884 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.885 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.885 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.885 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.885 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.885 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.885 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption7": { 2024-03-28T05:33:39.885 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption7", 2024-03-28T05:33:39.885 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.885 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.885 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.885 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.885 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T05:33:39.885 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "42", 2024-03-28T05:33:39.885 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.885 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.885 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.885 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.885 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.886 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.886 INFO:teuthology.orchestra.run.smithi023.stdout: "testkey": { 2024-03-28T05:33:39.886 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testkey", 2024-03-28T05:33:39.886 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.886 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.886 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.886 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.886 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.886 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.886 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.886 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.886 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.886 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.886 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.886 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.886 INFO:teuthology.orchestra.run.smithi023.stdout: "testlkey": { 2024-03-28T05:33:39.887 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testlkey", 2024-03-28T05:33:39.887 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.887 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.887 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.887 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.887 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.887 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.887 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.887 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.887 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.887 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.887 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.887 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.887 INFO:teuthology.orchestra.run.smithi023.stdout: "testnewline": { 2024-03-28T05:33:39.887 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testnewline", 2024-03-28T05:33:39.887 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.887 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.887 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.888 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.888 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.888 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.888 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.888 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.888 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.888 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.888 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.888 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.888 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.888 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.888 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.888 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-28T05:33:39.888 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.888 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.888 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.888 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_m_granularity": { 2024-03-28T05:33:39.889 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_m_granularity", 2024-03-28T05:33:39.889 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.889 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.889 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.889 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.889 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.889 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.889 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.889 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T05:33:39.889 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.889 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.889 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.889 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.889 INFO:teuthology.orchestra.run.smithi023.stdout: "dump_on_update": { 2024-03-28T05:33:39.889 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dump_on_update", 2024-03-28T05:33:39.889 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.889 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.889 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.890 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.890 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.890 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.890 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.890 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "dump database to debug log on update", 2024-03-28T05:33:39.890 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.890 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.890 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.890 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.890 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.890 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.890 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.890 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.890 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.890 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.890 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.891 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.891 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.891 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.891 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.891 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.891 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.891 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.891 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.891 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.891 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.891 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.891 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.891 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.891 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.891 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.891 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.891 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.892 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.892 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.892 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.892 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.892 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.892 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.892 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.892 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.892 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.892 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.892 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.892 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.892 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.892 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.892 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.892 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.892 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.893 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.893 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.893 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.893 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.893 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.893 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.893 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.893 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.893 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.893 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.893 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.893 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.893 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.893 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.893 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.893 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.893 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.894 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.894 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.894 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.894 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.894 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.894 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.894 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.894 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.894 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.894 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.894 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.894 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.894 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.894 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.894 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.894 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stats", 2024-03-28T05:33:39.894 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.895 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.895 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.895 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.895 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.895 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.895 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.895 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.895 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.895 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.895 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.895 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.895 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.895 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.895 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.895 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.895 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.895 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.896 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.896 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.896 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.896 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.896 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.896 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.896 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.896 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.896 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.896 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.896 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.896 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.896 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.896 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.896 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.896 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.896 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.896 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.897 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.897 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.897 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.897 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.897 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.897 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.897 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.897 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.897 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.897 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.897 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.897 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.897 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.897 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.897 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.897 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.897 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.898 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.898 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.898 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.898 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.898 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.898 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.898 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.898 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.898 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.898 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.898 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.898 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.898 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.898 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.898 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.898 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.899 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.899 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.899 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.899 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.899 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.899 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.899 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.899 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "status", 2024-03-28T05:33:39.899 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.899 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.899 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.899 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.899 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.899 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.899 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.899 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.899 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.900 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.900 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.900 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.900 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.900 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.900 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.900 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.900 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.900 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.900 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.900 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.900 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.900 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.900 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.900 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.900 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.900 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.901 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.901 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.901 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.901 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.901 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.901 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.901 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.901 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.901 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.901 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.901 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.901 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.901 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.901 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.901 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.901 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.901 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.902 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.902 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.902 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.902 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.902 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.902 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.902 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.902 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.902 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.902 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.902 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.902 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.902 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.902 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.902 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.902 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.902 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.903 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.903 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.903 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.903 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.903 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.903 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.903 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.903 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.903 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.903 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.903 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.903 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.903 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.903 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.903 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.903 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.903 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telegraf", 2024-03-28T05:33:39.903 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.904 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.904 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.904 INFO:teuthology.orchestra.run.smithi023.stdout: "address": { 2024-03-28T05:33:39.904 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "address", 2024-03-28T05:33:39.904 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.904 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.904 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.904 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T05:33:39.904 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.904 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.904 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.904 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.904 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.904 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.904 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.904 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.904 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T05:33:39.904 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T05:33:39.905 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.905 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.905 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.905 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15", 2024-03-28T05:33:39.905 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.905 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.905 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.905 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.905 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.905 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.905 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.905 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.905 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.905 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.905 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.905 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.906 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.906 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.906 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.906 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.906 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.906 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.906 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.906 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.906 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.906 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.906 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.906 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.906 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.906 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.906 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.906 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.906 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.907 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.907 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.907 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.907 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.907 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.907 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.907 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.907 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.907 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.907 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.907 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.907 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.907 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.907 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.907 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.907 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.907 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telemetry", 2024-03-28T05:33:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_basic": { 2024-03-28T05:33:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_basic", 2024-03-28T05:33:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T05:33:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_crash": { 2024-03-28T05:33:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_crash", 2024-03-28T05:33:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T05:33:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_device": { 2024-03-28T05:33:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_device", 2024-03-28T05:33:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T05:33:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_ident": { 2024-03-28T05:33:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_ident", 2024-03-28T05:33:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T05:33:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_perf": { 2024-03-28T05:33:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_perf", 2024-03-28T05:33:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T05:33:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "contact": { 2024-03-28T05:33:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "contact", 2024-03-28T05:33:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: "description": { 2024-03-28T05:33:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "description", 2024-03-28T05:33:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: "device_url": { 2024-03-28T05:33:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_url", 2024-03-28T05:33:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T05:33:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-28T05:33:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-28T05:33:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T05:33:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T05:33:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "24", 2024-03-28T05:33:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "8", 2024-03-28T05:33:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "last_opt_revision": { 2024-03-28T05:33:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "last_opt_revision", 2024-03-28T05:33:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1", 2024-03-28T05:33:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard": { 2024-03-28T05:33:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard", 2024-03-28T05:33:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard_description": { 2024-03-28T05:33:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard_description", 2024-03-28T05:33:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "organization": { 2024-03-28T05:33:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "organization", 2024-03-28T05:33:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "proxy": { 2024-03-28T05:33:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "proxy", 2024-03-28T05:33:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.926 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.926 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.926 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.926 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.926 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.926 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.926 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.926 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.926 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.926 INFO:teuthology.orchestra.run.smithi023.stdout: "url": { 2024-03-28T05:33:39.926 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url", 2024-03-28T05:33:39.926 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.926 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.926 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.926 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T05:33:39.926 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.926 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.926 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.927 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.927 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.927 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.927 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.927 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.927 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.927 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.927 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.927 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "test_orchestrator", 2024-03-28T05:33:39.927 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.927 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.927 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.927 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.927 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.927 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.927 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.928 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.928 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.928 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.928 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.928 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.928 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.928 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.928 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.928 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.928 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.928 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.928 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.928 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.928 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.928 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.928 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.928 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.929 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.929 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.929 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.929 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.929 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.929 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.929 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.929 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.929 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.929 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.929 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.929 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.929 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.929 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.929 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.929 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.929 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.929 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.930 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.930 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.930 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.930 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.930 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.930 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.930 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.930 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.930 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.930 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.930 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.930 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.930 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.930 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.930 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.931 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.931 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.931 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.931 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.931 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.931 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.931 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.931 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.931 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.931 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.931 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.931 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.931 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.931 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.931 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.931 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.931 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.932 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.932 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.932 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2024-03-28T05:33:39.932 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.932 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.932 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.932 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.932 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.932 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.932 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.932 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.932 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.932 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.932 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.932 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.932 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.933 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.933 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.933 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.933 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.933 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.933 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.933 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.933 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.933 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.933 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.933 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.933 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.933 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.933 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.933 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.933 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.933 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.934 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.934 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.934 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.934 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.934 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.934 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.934 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.934 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.934 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.934 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.934 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.934 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.934 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.934 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.934 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.934 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.934 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.935 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.935 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.935 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.935 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.935 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.935 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.935 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.935 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.935 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.935 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.935 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.935 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.935 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.935 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.935 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.935 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.935 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.936 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.936 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.936 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.936 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.936 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.936 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.936 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.936 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.936 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.936 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_clones": { 2024-03-28T05:33:39.936 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_clones", 2024-03-28T05:33:39.936 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.936 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.936 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.936 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4", 2024-03-28T05:33:39.936 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.937 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.937 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.937 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T05:33:39.937 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.937 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.937 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.937 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.937 INFO:teuthology.orchestra.run.smithi023.stdout: "periodic_async_work": { 2024-03-28T05:33:39.937 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "periodic_async_work", 2024-03-28T05:33:39.937 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.937 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.937 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.937 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.937 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.937 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.937 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.937 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Periodically check for async work", 2024-03-28T05:33:39.937 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.938 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.938 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.938 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.938 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_delay": { 2024-03-28T05:33:39.938 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_delay", 2024-03-28T05:33:39.938 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.938 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.938 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.938 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T05:33:39.938 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.938 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.938 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.938 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T05:33:39.938 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.938 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.938 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.939 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.939 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_no_wait": { 2024-03-28T05:33:39.939 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T05:33:39.939 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.939 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.939 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.939 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.939 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.939 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.939 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.939 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T05:33:39.939 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.939 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.939 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.939 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.940 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.940 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.940 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.940 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix", 2024-03-28T05:33:39.940 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.940 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.940 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.940 INFO:teuthology.orchestra.run.smithi023.stdout: "discovery_interval": { 2024-03-28T05:33:39.940 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "discovery_interval", 2024-03-28T05:33:39.940 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T05:33:39.940 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.940 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.940 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "100", 2024-03-28T05:33:39.940 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.940 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.940 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.940 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.940 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.941 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.941 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.941 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.941 INFO:teuthology.orchestra.run.smithi023.stdout: "identifier": { 2024-03-28T05:33:39.941 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "identifier", 2024-03-28T05:33:39.941 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.941 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.941 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.941 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.941 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.941 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.941 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.941 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.941 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.941 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.941 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.941 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.942 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T05:33:39.942 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T05:33:39.942 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.942 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.942 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.942 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T05:33:39.942 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.942 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.942 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.942 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.942 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.942 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.942 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.942 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.942 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.942 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.942 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.942 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.943 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.943 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.943 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.943 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.943 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.943 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.943 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.943 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.943 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.943 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.943 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.943 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.943 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.943 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.943 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.943 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.943 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.944 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.944 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.944 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.944 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.944 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.944 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.944 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.944 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.944 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.944 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.944 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.944 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.944 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.944 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.944 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.944 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.944 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.944 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.945 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.945 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.945 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.945 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.945 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.945 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.945 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.945 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.945 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.945 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.945 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.945 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.945 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.945 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.945 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.945 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.945 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.946 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.946 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.946 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.946 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.946 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.946 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.946 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.946 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.946 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.946 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.946 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.946 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.946 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.946 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.946 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_host": { 2024-03-28T05:33:39.946 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_host", 2024-03-28T05:33:39.946 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.947 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.947 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.947 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.947 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.947 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.947 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.947 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.947 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.947 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.947 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.947 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.947 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_port": { 2024-03-28T05:33:39.947 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_port", 2024-03-28T05:33:39.947 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.947 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.947 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.947 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10051", 2024-03-28T05:33:39.947 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.948 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.948 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.948 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.948 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.948 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.948 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.948 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.948 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_sender": { 2024-03-28T05:33:39.948 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_sender", 2024-03-28T05:33:39.948 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.948 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.948 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.948 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T05:33:39.948 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.948 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.948 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.948 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.948 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.949 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.949 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.949 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.949 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.949 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.949 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T05:33:39.949 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.949 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.949 INFO:teuthology.orchestra.run.smithi023.stdout: "modules": [ 2024-03-28T05:33:39.949 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2024-03-28T05:33:39.949 INFO:teuthology.orchestra.run.smithi023.stdout: "dashboard", 2024-03-28T05:33:39.949 INFO:teuthology.orchestra.run.smithi023.stdout: "iostat", 2024-03-28T05:33:39.949 INFO:teuthology.orchestra.run.smithi023.stdout: "nfs", 2024-03-28T05:33:39.949 INFO:teuthology.orchestra.run.smithi023.stdout: "restful", 2024-03-28T05:33:39.949 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_schedule" 2024-03-28T05:33:39.949 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.949 INFO:teuthology.orchestra.run.smithi023.stdout: "available_modules": [ 2024-03-28T05:33:39.949 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.950 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alerts", 2024-03-28T05:33:39.950 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.950 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.950 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.950 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T05:33:39.950 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T05:33:39.950 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.950 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.950 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.950 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T05:33:39.950 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.950 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.950 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.950 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T05:33:39.950 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.950 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.951 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.951 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.951 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.951 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.951 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.951 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.951 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.951 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.951 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.951 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.951 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.951 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.951 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.951 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.951 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.951 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.951 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.951 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.952 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.952 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.952 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.952 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.952 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.952 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.952 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.952 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.952 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.952 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.952 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.952 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.952 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.952 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.952 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.952 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.952 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.953 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.953 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.953 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.953 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.953 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.953 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.953 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.953 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.953 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.953 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.953 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.953 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.953 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.953 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.953 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.953 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.953 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.954 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.954 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.954 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.954 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.954 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.954 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.954 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.954 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.954 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.954 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.954 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.954 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.954 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.954 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.954 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.955 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.955 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.955 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.955 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.955 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.955 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_destination": { 2024-03-28T05:33:39.955 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_destination", 2024-03-28T05:33:39.955 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.955 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.955 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.955 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.955 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.955 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.955 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.955 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email address to send alerts to", 2024-03-28T05:33:39.955 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.956 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.956 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.956 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.956 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_from_name": { 2024-03-28T05:33:39.956 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_from_name", 2024-03-28T05:33:39.956 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.956 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.956 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.956 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "Ceph", 2024-03-28T05:33:39.956 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.956 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.956 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.956 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email From: name", 2024-03-28T05:33:39.956 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.956 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.956 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.956 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.957 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_host": { 2024-03-28T05:33:39.957 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_host", 2024-03-28T05:33:39.957 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.957 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.957 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.957 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.957 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.957 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.957 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.957 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP server", 2024-03-28T05:33:39.957 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.957 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.957 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.957 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.957 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_password": { 2024-03-28T05:33:39.957 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_password", 2024-03-28T05:33:39.957 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.957 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.958 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.958 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.958 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.958 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.958 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.958 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Password to authenticate with", 2024-03-28T05:33:39.958 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.958 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.958 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.958 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.958 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_port": { 2024-03-28T05:33:39.958 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_port", 2024-03-28T05:33:39.958 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.958 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.958 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.958 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "465", 2024-03-28T05:33:39.958 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.959 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.959 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.959 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP port", 2024-03-28T05:33:39.959 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.959 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.959 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.959 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.959 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_sender": { 2024-03-28T05:33:39.959 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_sender", 2024-03-28T05:33:39.959 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.959 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.959 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.959 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.959 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.959 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.959 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.959 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP envelope sender", 2024-03-28T05:33:39.959 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.960 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.960 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.960 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.960 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_ssl": { 2024-03-28T05:33:39.960 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_ssl", 2024-03-28T05:33:39.960 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.960 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.960 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.960 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.960 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.960 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.960 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.960 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T05:33:39.960 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.960 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.960 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.960 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.960 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_user": { 2024-03-28T05:33:39.961 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_user", 2024-03-28T05:33:39.961 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.961 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.961 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.961 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.961 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.961 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.961 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.961 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "User to authenticate as", 2024-03-28T05:33:39.961 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.961 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.961 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.961 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.961 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.961 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.962 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.962 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "balancer", 2024-03-28T05:33:39.962 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.962 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.962 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.962 INFO:teuthology.orchestra.run.smithi023.stdout: "active": { 2024-03-28T05:33:39.962 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "active", 2024-03-28T05:33:39.962 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.962 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.962 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.962 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.962 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.962 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.962 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.962 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T05:33:39.962 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.962 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.963 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.963 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.963 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_time": { 2024-03-28T05:33:39.963 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_time", 2024-03-28T05:33:39.963 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.963 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.963 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.963 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0000", 2024-03-28T05:33:39.963 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.963 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.963 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.963 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T05:33:39.963 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T05:33:39.963 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.963 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.963 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.963 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_weekday": { 2024-03-28T05:33:39.963 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_weekday", 2024-03-28T05:33:39.964 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T05:33:39.964 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.964 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.964 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T05:33:39.964 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-28T05:33:39.964 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-28T05:33:39.964 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.964 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T05:33:39.964 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T05:33:39.964 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.964 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.964 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.964 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_max_iterations": { 2024-03-28T05:33:39.964 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_max_iterations", 2024-03-28T05:33:39.964 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T05:33:39.964 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.964 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.965 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "25", 2024-03-28T05:33:39.965 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T05:33:39.965 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "250", 2024-03-28T05:33:39.965 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.965 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T05:33:39.965 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.965 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.965 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.965 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.965 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_metrics": { 2024-03-28T05:33:39.965 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_metrics", 2024-03-28T05:33:39.965 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.965 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.965 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.965 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T05:33:39.965 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.966 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.966 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.966 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T05:33:39.966 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T05:33:39.966 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.966 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.966 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.966 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_step": { 2024-03-28T05:33:39.966 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_step", 2024-03-28T05:33:39.966 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T05:33:39.966 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.966 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.966 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.5", 2024-03-28T05:33:39.966 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0.001", 2024-03-28T05:33:39.966 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "0.999", 2024-03-28T05:33:39.966 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.967 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "aggressiveness of optimization", 2024-03-28T05:33:39.967 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T05:33:39.967 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.967 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.967 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.967 INFO:teuthology.orchestra.run.smithi023.stdout: "end_time": { 2024-03-28T05:33:39.967 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_time", 2024-03-28T05:33:39.967 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.967 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.967 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.967 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2359", 2024-03-28T05:33:39.967 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.967 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.967 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.967 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T05:33:39.968 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T05:33:39.968 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.968 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.968 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.968 INFO:teuthology.orchestra.run.smithi023.stdout: "end_weekday": { 2024-03-28T05:33:39.968 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_weekday", 2024-03-28T05:33:39.968 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T05:33:39.968 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.968 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.968 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T05:33:39.968 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-28T05:33:39.968 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-28T05:33:39.968 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.968 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T05:33:39.968 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T05:33:39.969 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.969 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.969 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.969 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:39.969 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:39.969 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.969 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.969 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.969 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.969 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.969 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.969 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.969 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.969 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.969 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.969 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.970 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.970 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.970 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.970 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.970 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.970 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.970 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.970 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.970 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:39.970 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:39.970 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.970 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.970 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.970 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.970 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.970 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.971 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.971 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.971 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.971 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.971 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.971 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.971 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:39.971 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:39.971 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.971 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.971 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.971 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:39.971 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.971 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.971 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.972 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:39.972 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:39.972 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:39.972 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:39.972 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:39.972 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:39.972 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.972 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.972 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.972 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.972 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.972 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.972 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:39.972 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:39.972 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.972 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.973 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.973 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.973 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.973 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.973 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.973 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:39.973 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.973 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.973 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.973 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.973 INFO:teuthology.orchestra.run.smithi023.stdout: "min_score": { 2024-03-28T05:33:39.973 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_score", 2024-03-28T05:33:39.973 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T05:33:39.973 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.974 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.974 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T05:33:39.974 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.974 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.974 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.974 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T05:33:39.974 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.974 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.974 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.974 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.974 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-28T05:33:39.974 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-28T05:33:39.974 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.974 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.974 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.974 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "upmap", 2024-03-28T05:33:39.975 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.975 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.975 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.975 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-compat", 2024-03-28T05:33:39.975 INFO:teuthology.orchestra.run.smithi023.stdout: "none", 2024-03-28T05:33:39.975 INFO:teuthology.orchestra.run.smithi023.stdout: "read", 2024-03-28T05:33:39.975 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap", 2024-03-28T05:33:39.975 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap-read" 2024-03-28T05:33:39.975 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.975 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Balancer mode", 2024-03-28T05:33:39.975 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.975 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.975 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.975 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.975 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_ids": { 2024-03-28T05:33:39.975 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_ids", 2024-03-28T05:33:39.976 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.976 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.976 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.976 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.976 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.976 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.976 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.976 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T05:33:39.976 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.976 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.976 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.976 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.976 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T05:33:39.976 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T05:33:39.976 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.976 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.976 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.977 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T05:33:39.977 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.977 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.977 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.977 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T05:33:39.977 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.977 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.977 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.977 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.977 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_deviation": { 2024-03-28T05:33:39.977 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_deviation", 2024-03-28T05:33:39.977 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.977 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.977 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.977 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T05:33:39.977 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T05:33:39.978 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.978 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.978 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T05:33:39.978 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T05:33:39.978 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.978 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.978 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.978 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_optimizations": { 2024-03-28T05:33:39.978 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_optimizations", 2024-03-28T05:33:39.978 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T05:33:39.978 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.978 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.978 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:39.978 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T05:33:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm", 2024-03-28T05:33:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_down_multiplier": { 2024-03-28T05:33:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_down_multiplier", 2024-03-28T05:33:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T05:33:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-28T05:33:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T05:33:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_refresh_rate": { 2024-03-28T05:33:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_refresh_rate", 2024-03-28T05:33:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "20", 2024-03-28T05:33:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T05:33:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_starting_port": { 2024-03-28T05:33:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_starting_port", 2024-03-28T05:33:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4721", 2024-03-28T05:33:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T05:33:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_ptrace": { 2024-03-28T05:33:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_ptrace", 2024-03-28T05:33:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T05:33:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T05:33:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_interval": { 2024-03-28T05:33:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_interval", 2024-03-28T05:33:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T05:33:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T05:33:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_memory_target_ratio": { 2024-03-28T05:33:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T05:33:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T05:33:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.7", 2024-03-28T05:33:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T05:33:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm_log_destination": { 2024-03-28T05:33:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm_log_destination", 2024-03-28T05:33:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: "file", 2024-03-28T05:33:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: "file,syslog", 2024-03-28T05:33:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: "syslog" 2024-03-28T05:33:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T05:33:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: "cgroups_split": { 2024-03-28T05:33:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cgroups_split", 2024-03-28T05:33:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T05:33:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "config_checks_enabled": { 2024-03-28T05:33:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_checks_enabled", 2024-03-28T05:33:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T05:33:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "config_dashboard": { 2024-03-28T05:33:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_dashboard", 2024-03-28T05:33:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T05:33:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_alertmanager": { 2024-03-28T05:33:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_alertmanager", 2024-03-28T05:33:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T05:33:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T05:33:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_base": { 2024-03-28T05:33:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_base", 2024-03-28T05:33:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T05:33:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Container image name, without the tag", 2024-03-28T05:33:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_elasticsearch": { 2024-03-28T05:33:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_elasticsearch", 2024-03-28T05:33:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T05:33:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "elasticsearch container image", 2024-03-28T05:33:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_grafana": { 2024-03-28T05:33:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_grafana", 2024-03-28T05:33:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T05:33:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T05:33:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_haproxy": { 2024-03-28T05:33:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_haproxy", 2024-03-28T05:33:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T05:33:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "HAproxy container image", 2024-03-28T05:33:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_agent": { 2024-03-28T05:33:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_agent", 2024-03-28T05:33:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T05:33:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger agent container image", 2024-03-28T05:33:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_collector": { 2024-03-28T05:33:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_collector", 2024-03-28T05:33:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T05:33:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger collector container image", 2024-03-28T05:33:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_query": { 2024-03-28T05:33:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_query", 2024-03-28T05:33:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T05:33:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger query container image", 2024-03-28T05:33:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_keepalived": { 2024-03-28T05:33:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_keepalived", 2024-03-28T05:33:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.995 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.995 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.995 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T05:33:39.995 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.995 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.995 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.995 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Keepalived container image", 2024-03-28T05:33:39.995 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.995 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.995 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.995 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.995 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_loki": { 2024-03-28T05:33:39.995 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_loki", 2024-03-28T05:33:39.995 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.995 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.995 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.995 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T05:33:39.996 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.996 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.996 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.996 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Loki container image", 2024-03-28T05:33:39.996 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.996 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.996 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.996 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.996 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_node_exporter": { 2024-03-28T05:33:39.996 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_node_exporter", 2024-03-28T05:33:39.996 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.996 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.996 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.996 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T05:33:39.996 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.996 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.997 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.997 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T05:33:39.997 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.997 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.997 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.997 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.997 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_nvmeof": { 2024-03-28T05:33:39.997 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_nvmeof", 2024-03-28T05:33:39.997 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.997 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.997 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.997 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-28T05:33:39.997 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.997 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.997 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.998 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Nvme-of container image", 2024-03-28T05:33:39.998 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.998 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.998 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.998 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.998 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_prometheus": { 2024-03-28T05:33:39.998 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_prometheus", 2024-03-28T05:33:39.998 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:39.999 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:39.999 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:39.999 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T05:33:39.999 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:39.999 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:39.999 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:39.999 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T05:33:39.999 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:39.999 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:39.999 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:39.999 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:39.999 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_promtail": { 2024-03-28T05:33:39.999 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_promtail", 2024-03-28T05:33:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T05:33:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Promtail container image", 2024-03-28T05:33:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_snmp_gateway": { 2024-03-28T05:33:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_snmp_gateway", 2024-03-28T05:33:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T05:33:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SNMP Gateway container image", 2024-03-28T05:33:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "container_init": { 2024-03-28T05:33:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_init", 2024-03-28T05:33:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T05:33:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: "daemon_cache_timeout": { 2024-03-28T05:33:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "daemon_cache_timeout", 2024-03-28T05:33:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T05:33:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T05:33:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "default_cephadm_command_timeout": { 2024-03-28T05:33:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T05:33:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "900", 2024-03-28T05:33:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T05:33:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "default_registry": { 2024-03-28T05:33:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_registry", 2024-03-28T05:33:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io", 2024-03-28T05:33:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T05:33:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "device_cache_timeout": { 2024-03-28T05:33:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_cache_timeout", 2024-03-28T05:33:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1800", 2024-03-28T05:33:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache device inventory", 2024-03-28T05:33:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "device_enhanced_scan": { 2024-03-28T05:33:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_enhanced_scan", 2024-03-28T05:33:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T05:33:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "facts_cache_timeout": { 2024-03-28T05:33:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "facts_cache_timeout", 2024-03-28T05:33:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T05:33:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache host facts data", 2024-03-28T05:33:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: "host_check_interval": { 2024-03-28T05:33:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "host_check_interval", 2024-03-28T05:33:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T05:33:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to perform a host check", 2024-03-28T05:33:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "hw_monitoring": { 2024-03-28T05:33:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hw_monitoring", 2024-03-28T05:33:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T05:33:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "inventory_list_all": { 2024-03-28T05:33:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "inventory_list_all", 2024-03-28T05:33:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T05:33:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: "log_refresh_metadata": { 2024-03-28T05:33:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_refresh_metadata", 2024-03-28T05:33:40.011 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.011 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.011 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.011 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.011 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.011 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.011 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.011 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T05:33:40.011 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.011 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.011 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.011 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.011 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.011 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.011 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.011 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.011 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.011 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T05:33:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T05:33:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T05:33:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T05:33:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T05:33:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T05:33:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "*", 2024-03-28T05:33:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T05:33:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: "max_count_per_host": { 2024-03-28T05:33:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_count_per_host", 2024-03-28T05:33:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of daemons per service per host", 2024-03-28T05:33:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd_draining_count": { 2024-03-28T05:33:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_osd_draining_count", 2024-03-28T05:33:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T05:33:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: "migration_current": { 2024-03-28T05:33:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "migration_current", 2024-03-28T05:33:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "internal - do not modify", 2024-03-28T05:33:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-28T05:33:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-28T05:33:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "root", 2024-03-28T05:33:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm-package", 2024-03-28T05:33:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: "root" 2024-03-28T05:33:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T05:33:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: "oob_default_addr": { 2024-03-28T05:33:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "oob_default_addr", 2024-03-28T05:33:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "169.254.1.1", 2024-03-28T05:33:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T05:33:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_alerts_path": { 2024-03-28T05:33:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_alerts_path", 2024-03-28T05:33:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T05:33:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T05:33:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_insecure": { 2024-03-28T05:33:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_insecure", 2024-03-28T05:33:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T05:33:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_password": { 2024-03-28T05:33:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_password", 2024-03-28T05:33:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T05:33:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_url": { 2024-03-28T05:33:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_url", 2024-03-28T05:33:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T05:33:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_username": { 2024-03-28T05:33:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_username", 2024-03-28T05:33:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T05:33:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "secure_monitoring_stack": { 2024-03-28T05:33:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "secure_monitoring_stack", 2024-03-28T05:33:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T05:33:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "service_discovery_port": { 2024-03-28T05:33:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "service_discovery_port", 2024-03-28T05:33:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8765", 2024-03-28T05:33:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "cephadm service discovery port", 2024-03-28T05:33:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "ssh_config_file": { 2024-03-28T05:33:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssh_config_file", 2024-03-28T05:33:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T05:33:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "use_agent": { 2024-03-28T05:33:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_agent", 2024-03-28T05:33:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T05:33:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "use_repo_digest": { 2024-03-28T05:33:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_repo_digest", 2024-03-28T05:33:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T05:33:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_failed_host_check": { 2024-03-28T05:33:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_failed_host_check", 2024-03-28T05:33:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T05:33:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_daemons": { 2024-03-28T05:33:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_daemons", 2024-03-28T05:33:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T05:33:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_hosts": { 2024-03-28T05:33:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_hosts", 2024-03-28T05:33:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T05:33:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crash", 2024-03-28T05:33:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "retain_interval": { 2024-03-28T05:33:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retain_interval", 2024-03-28T05:33:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "31536000", 2024-03-28T05:33:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T05:33:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_recent_interval": { 2024-03-28T05:33:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_recent_interval", 2024-03-28T05:33:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1209600", 2024-03-28T05:33:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T05:33:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dashboard", 2024-03-28T05:33:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T05:33:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T05:33:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T05:33:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T05:33:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T05:33:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T05:33:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_ENABLED": { 2024-03-28T05:33:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T05:33:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T05:33:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T05:33:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T05:33:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T05:33:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T05:33:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T05:33:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T05:33:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T05:33:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T05:33:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T05:33:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T05:33:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T05:33:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T05:33:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T05:33:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.050 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T05:33:40.050 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T05:33:40.050 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.050 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.050 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.050 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.050 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.050 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.050 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.050 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.050 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.050 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.050 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.050 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.050 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T05:33:40.050 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T05:33:40.050 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.051 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.051 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.051 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.051 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.051 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.051 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.051 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.051 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.051 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.051 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.051 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.051 INFO:teuthology.orchestra.run.smithi023.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T05:33:40.051 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T05:33:40.051 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.051 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.051 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.051 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.051 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.052 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.052 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.052 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.052 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.052 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.052 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.052 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.052 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T05:33:40.052 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T05:33:40.052 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.052 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.052 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.052 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-28T05:33:40.052 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.052 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.052 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.053 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.053 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.053 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.053 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.053 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.053 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T05:33:40.053 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T05:33:40.053 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.053 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.053 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.053 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.053 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.053 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.053 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.053 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.053 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.053 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.054 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.054 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.054 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_URL": { 2024-03-28T05:33:40.054 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_URL", 2024-03-28T05:33:40.054 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.054 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.054 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.054 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.054 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.054 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.054 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.054 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.054 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.054 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.054 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.054 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.055 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T05:33:40.055 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T05:33:40.055 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.055 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.055 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.055 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-28T05:33:40.055 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.055 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.055 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.055 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.055 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.055 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.055 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.055 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.055 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T05:33:40.055 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T05:33:40.055 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.055 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.056 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.056 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.056 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.056 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.056 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.056 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.056 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.056 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.056 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.056 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.056 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T05:33:40.056 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T05:33:40.056 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.056 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.056 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.056 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.056 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.057 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.057 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.057 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.057 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.057 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.057 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.057 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.057 INFO:teuthology.orchestra.run.smithi023.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T05:33:40.057 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T05:33:40.057 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.057 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.057 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.057 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.057 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.057 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.057 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.057 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.057 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.058 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.058 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.058 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.058 INFO:teuthology.orchestra.run.smithi023.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T05:33:40.058 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T05:33:40.058 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.058 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.058 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.058 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.058 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.058 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.058 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.058 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.058 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.058 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.059 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.059 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.059 INFO:teuthology.orchestra.run.smithi023.stdout: "MULTICLUSTER_CONFIG": { 2024-03-28T05:33:40.059 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-28T05:33:40.059 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.059 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.059 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.059 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "{}", 2024-03-28T05:33:40.059 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.059 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.059 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.059 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.059 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.059 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.059 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.059 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.059 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T05:33:40.059 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T05:33:40.060 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.060 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.060 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.060 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.060 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.060 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.060 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.060 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.060 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.060 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.060 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.060 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.060 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T05:33:40.060 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T05:33:40.060 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.060 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.060 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.061 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.061 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.061 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.061 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.061 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.061 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.061 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.061 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.061 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.061 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T05:33:40.061 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T05:33:40.061 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.061 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.061 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.061 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.062 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.062 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.062 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.062 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.062 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.062 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.062 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.062 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.062 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T05:33:40.062 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T05:33:40.062 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.062 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.062 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.062 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.062 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.062 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.063 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.063 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.063 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.063 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.063 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.063 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.063 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T05:33:40.063 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T05:33:40.063 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.063 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.063 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.063 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.063 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.063 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.063 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.063 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.063 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.063 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.064 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.064 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.064 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T05:33:40.064 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T05:33:40.064 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.064 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.064 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.064 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.064 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.064 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.064 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.064 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.064 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.064 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.064 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.064 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.064 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T05:33:40.064 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T05:33:40.065 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.065 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.065 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.065 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.065 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.065 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.065 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.065 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.065 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.065 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.065 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.065 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.065 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T05:33:40.065 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T05:33:40.065 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.065 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.065 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.065 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.066 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.066 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.066 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.066 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.066 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.066 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.066 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.066 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.066 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T05:33:40.066 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T05:33:40.066 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.066 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.066 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.066 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.066 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.067 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.067 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.067 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.067 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.067 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.067 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.067 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.067 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T05:33:40.067 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T05:33:40.067 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.067 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.067 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.067 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.067 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.067 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.067 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.067 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.068 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.068 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.068 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.068 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.068 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T05:33:40.068 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T05:33:40.068 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.068 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.068 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.068 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T05:33:40.068 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.068 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.068 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.068 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.068 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.068 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.068 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.069 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.069 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T05:33:40.069 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T05:33:40.069 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.069 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.069 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.069 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:40.069 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.069 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.069 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.069 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.069 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.069 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.069 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.069 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.069 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T05:33:40.070 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T05:33:40.070 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.070 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.070 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.070 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8", 2024-03-28T05:33:40.070 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.070 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.070 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.070 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.070 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.070 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.070 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.070 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.070 INFO:teuthology.orchestra.run.smithi023.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T05:33:40.070 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T05:33:40.070 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.070 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.071 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.071 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "45", 2024-03-28T05:33:40.071 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.071 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.071 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.071 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.071 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.071 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.071 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.071 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.071 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T05:33:40.071 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T05:33:40.071 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.071 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.071 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.071 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.071 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.071 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.072 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.072 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.072 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.072 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.072 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.072 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.072 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T05:33:40.072 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T05:33:40.072 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.072 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.072 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.072 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-28T05:33:40.072 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.072 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.072 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.072 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.072 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.073 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.073 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.073 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T05:33:40.073 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T05:33:40.073 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.073 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.073 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.073 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.073 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.073 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.073 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.073 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.073 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.073 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.073 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.074 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T05:33:40.074 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T05:33:40.074 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.074 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.074 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.074 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.074 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.074 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.074 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.074 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.074 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.074 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.074 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.074 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.074 INFO:teuthology.orchestra.run.smithi023.stdout: "UNSAFE_TLS_v1_2": { 2024-03-28T05:33:40.074 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-28T05:33:40.074 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.075 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.075 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.075 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.075 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.075 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.075 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.075 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.075 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.075 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.075 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.075 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.075 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T05:33:40.075 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T05:33:40.075 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.075 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.075 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.075 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T05:33:40.076 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.076 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.076 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.076 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.076 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.076 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.076 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.076 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.076 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T05:33:40.076 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T05:33:40.076 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.076 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.076 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.076 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:40.076 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.076 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.076 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.076 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.077 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.077 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.077 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.077 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.077 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T05:33:40.077 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T05:33:40.077 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.077 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.077 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.077 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T05:33:40.077 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.077 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.077 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.077 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.077 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.077 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.077 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.077 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.078 INFO:teuthology.orchestra.run.smithi023.stdout: "cross_origin_url": { 2024-03-28T05:33:40.078 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cross_origin_url", 2024-03-28T05:33:40.078 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.078 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.078 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.078 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.078 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.078 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.078 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.078 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.078 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.078 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.078 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.078 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.078 INFO:teuthology.orchestra.run.smithi023.stdout: "crt_file": { 2024-03-28T05:33:40.078 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crt_file", 2024-03-28T05:33:40.079 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.079 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.079 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.079 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.079 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.079 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.079 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.079 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.079 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.079 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.079 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.079 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.079 INFO:teuthology.orchestra.run.smithi023.stdout: "debug": { 2024-03-28T05:33:40.079 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "debug", 2024-03-28T05:33:40.079 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.079 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.079 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.080 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.080 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.080 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.080 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.080 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable/disable debug options", 2024-03-28T05:33:40.080 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.080 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.080 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.080 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.080 INFO:teuthology.orchestra.run.smithi023.stdout: "jwt_token_ttl": { 2024-03-28T05:33:40.080 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "jwt_token_ttl", 2024-03-28T05:33:40.080 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.080 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.080 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.080 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "28800", 2024-03-28T05:33:40.080 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.081 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.081 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.081 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.081 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.081 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.081 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.081 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.081 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-28T05:33:40.081 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-28T05:33:40.081 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.081 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.081 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.081 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.081 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.081 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.081 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.081 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.081 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.082 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.082 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.082 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.082 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.082 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.082 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.082 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.082 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.082 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.082 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.082 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.082 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.082 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.082 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.082 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.082 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.082 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.083 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.083 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.083 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.083 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.083 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.083 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.083 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.083 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.083 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.083 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.083 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.083 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.083 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.083 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.083 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.083 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.083 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.084 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.084 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.084 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.084 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.084 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.084 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.084 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.084 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.084 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.084 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.084 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.084 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.084 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.084 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.084 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.084 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.084 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.084 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.085 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.085 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.085 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.085 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.085 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.085 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.085 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.085 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.085 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.085 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.085 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.085 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.085 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.085 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.085 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.085 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.085 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.085 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.086 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.086 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.086 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.086 INFO:teuthology.orchestra.run.smithi023.stdout: "motd": { 2024-03-28T05:33:40.086 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "motd", 2024-03-28T05:33:40.086 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.086 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.086 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.086 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.086 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.086 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.086 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.086 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "The message of the day", 2024-03-28T05:33:40.086 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.086 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.086 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.086 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.087 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect_resolve_ip_addr": { 2024-03-28T05:33:40.087 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T05:33:40.087 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.087 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.087 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.087 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.087 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.087 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.087 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.087 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.087 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.087 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.087 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.087 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.087 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-28T05:33:40.087 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-28T05:33:40.087 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.088 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.088 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.088 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-28T05:33:40.088 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.088 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.088 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.088 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.088 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.088 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.088 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.088 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.088 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-28T05:33:40.088 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-28T05:33:40.088 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.088 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.088 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.088 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8080", 2024-03-28T05:33:40.089 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.089 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.089 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.089 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.089 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.089 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.089 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.089 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.089 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-28T05:33:40.089 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-28T05:33:40.089 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.089 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.089 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.089 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.089 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.089 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.089 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.090 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.090 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.090 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.090 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.090 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.090 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl_server_port": { 2024-03-28T05:33:40.090 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl_server_port", 2024-03-28T05:33:40.090 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.090 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.090 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.090 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8443", 2024-03-28T05:33:40.090 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.090 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.090 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.090 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.090 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.090 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.091 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.091 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.091 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-28T05:33:40.091 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-28T05:33:40.091 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.091 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.091 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.091 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "redirect", 2024-03-28T05:33:40.091 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.091 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.091 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.091 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.091 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect" 2024-03-28T05:33:40.091 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.091 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.091 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.092 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.092 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.092 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.092 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-28T05:33:40.092 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-28T05:33:40.092 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.092 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.092 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.092 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-28T05:33:40.092 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-28T05:33:40.092 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-28T05:33:40.092 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.092 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.092 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.092 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.093 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.093 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.093 INFO:teuthology.orchestra.run.smithi023.stdout: "url_prefix": { 2024-03-28T05:33:40.093 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url_prefix", 2024-03-28T05:33:40.093 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.093 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.093 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.093 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.093 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.093 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.093 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.093 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.093 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.093 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.093 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.093 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.093 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.094 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.094 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.094 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2024-03-28T05:33:40.094 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.094 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.094 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.094 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_monitoring": { 2024-03-28T05:33:40.094 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_monitoring", 2024-03-28T05:33:40.094 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.094 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.094 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.094 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.094 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.094 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.094 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.094 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "monitor device health metrics", 2024-03-28T05:33:40.095 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.095 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.095 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.095 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.095 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.095 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.095 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.095 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.095 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.095 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.095 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.095 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.095 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.095 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.095 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.095 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.095 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.095 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.096 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.096 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.096 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.096 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.096 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.096 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.096 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.096 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.096 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.096 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.096 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.096 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.096 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.096 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.096 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.096 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.096 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.097 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.097 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.097 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.097 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.097 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.097 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.097 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.097 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.097 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.097 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.097 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.097 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.097 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.097 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.097 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.097 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.097 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.098 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.098 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.098 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.098 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.098 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.098 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.098 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.098 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.098 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.098 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.098 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.098 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.098 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.098 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.098 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.098 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.099 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.099 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.099 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.099 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.099 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.099 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.099 INFO:teuthology.orchestra.run.smithi023.stdout: "mark_out_threshold": { 2024-03-28T05:33:40.099 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mark_out_threshold", 2024-03-28T05:33:40.099 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:40.099 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.099 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.099 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2419200", 2024-03-28T05:33:40.099 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.099 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.099 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.099 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T05:33:40.099 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.100 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.100 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.100 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.100 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": { 2024-03-28T05:33:40.100 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_name", 2024-03-28T05:33:40.100 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.100 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.100 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.100 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "device_health_metrics", 2024-03-28T05:33:40.100 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.100 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.100 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.100 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T05:33:40.100 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.100 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.100 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.100 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.101 INFO:teuthology.orchestra.run.smithi023.stdout: "retention_period": { 2024-03-28T05:33:40.101 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retention_period", 2024-03-28T05:33:40.101 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:40.101 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.101 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.101 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15552000", 2024-03-28T05:33:40.101 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.101 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.101 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.101 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain device health metrics", 2024-03-28T05:33:40.101 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.101 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.101 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.101 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.101 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_frequency": { 2024-03-28T05:33:40.101 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_frequency", 2024-03-28T05:33:40.101 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:40.102 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.102 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.102 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-28T05:33:40.102 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.102 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.102 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.102 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T05:33:40.102 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.102 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.102 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.102 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.102 INFO:teuthology.orchestra.run.smithi023.stdout: "self_heal": { 2024-03-28T05:33:40.102 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "self_heal", 2024-03-28T05:33:40.102 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.102 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.102 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.102 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.103 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.103 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.103 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.103 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T05:33:40.103 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.103 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.103 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.103 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.103 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T05:33:40.103 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T05:33:40.103 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:40.103 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.103 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.103 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T05:33:40.103 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.103 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.103 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.103 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T05:33:40.104 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.104 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.104 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.104 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.104 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_threshold": { 2024-03-28T05:33:40.104 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_threshold", 2024-03-28T05:33:40.104 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:40.104 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.104 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.104 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7257600", 2024-03-28T05:33:40.104 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.104 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.104 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.104 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T05:33:40.104 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.104 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.104 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.105 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.105 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.105 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.105 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.105 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "diskprediction_local", 2024-03-28T05:33:40.105 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.105 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.105 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.105 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.105 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.105 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.105 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.105 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.105 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.105 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.105 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.105 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.106 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.106 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.106 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.106 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.106 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.106 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.106 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.106 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.106 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.106 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.106 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.106 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.106 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.106 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.106 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.106 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.107 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.107 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.107 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.107 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.107 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.107 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.107 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.107 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.107 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.107 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.107 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.107 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.107 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.107 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.107 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.107 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.107 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.107 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.108 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.108 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.108 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.108 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.108 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.108 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.108 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.108 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.108 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.108 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.108 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.108 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.108 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.108 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.108 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.108 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.108 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.109 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.109 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.109 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.109 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.109 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.109 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.109 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.109 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.109 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.109 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.109 INFO:teuthology.orchestra.run.smithi023.stdout: "predict_interval": { 2024-03-28T05:33:40.109 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predict_interval", 2024-03-28T05:33:40.109 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.109 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.109 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.109 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-28T05:33:40.109 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.110 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.110 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.110 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.110 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.110 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.110 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.110 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.110 INFO:teuthology.orchestra.run.smithi023.stdout: "predictor_model": { 2024-03-28T05:33:40.110 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predictor_model", 2024-03-28T05:33:40.110 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.110 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.110 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.110 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "prophetstor", 2024-03-28T05:33:40.111 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.111 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.111 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.111 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.111 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.111 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.111 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.111 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.111 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T05:33:40.111 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T05:33:40.111 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.111 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.111 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.111 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T05:33:40.111 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.111 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.111 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.112 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.112 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.112 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.112 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.112 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.112 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.112 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.112 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.112 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "influx", 2024-03-28T05:33:40.112 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": false, 2024-03-28T05:33:40.112 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "influxdb python module not found", 2024-03-28T05:33:40.112 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.112 INFO:teuthology.orchestra.run.smithi023.stdout: "batch_size": { 2024-03-28T05:33:40.112 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "batch_size", 2024-03-28T05:33:40.112 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.112 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.112 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.112 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5000", 2024-03-28T05:33:40.113 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.113 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.113 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.113 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T05:33:40.113 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.113 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.113 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.113 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.113 INFO:teuthology.orchestra.run.smithi023.stdout: "database": { 2024-03-28T05:33:40.113 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "database", 2024-03-28T05:33:40.113 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.113 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.113 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.113 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "ceph", 2024-03-28T05:33:40.113 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.113 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.113 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.114 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T05:33:40.114 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.114 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.114 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.114 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.114 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": { 2024-03-28T05:33:40.114 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hostname", 2024-03-28T05:33:40.114 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.114 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.114 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.114 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.114 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.114 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.114 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.114 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server hostname", 2024-03-28T05:33:40.114 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.114 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.115 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.115 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.115 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T05:33:40.115 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T05:33:40.115 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:40.115 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.115 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.115 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "30", 2024-03-28T05:33:40.115 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "5", 2024-03-28T05:33:40.115 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.115 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.115 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T05:33:40.115 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.115 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.115 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.115 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.115 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.116 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.116 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.116 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.116 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.116 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.116 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.116 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.116 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.116 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.116 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.116 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.116 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.116 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.116 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.116 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.116 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.116 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.117 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.117 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.117 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.117 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.117 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.117 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.117 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.117 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.117 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.117 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.117 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.117 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.117 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.117 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.117 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.117 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.117 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.118 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.118 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.118 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.118 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.118 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.118 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.118 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.118 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.118 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.118 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.118 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.118 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.118 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.118 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.118 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.118 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.119 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.119 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.119 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.119 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.119 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.119 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.119 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.119 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.119 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.119 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.119 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.119 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.119 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.119 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.119 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.119 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.120 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.120 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.120 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.120 INFO:teuthology.orchestra.run.smithi023.stdout: "password": { 2024-03-28T05:33:40.120 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "password", 2024-03-28T05:33:40.120 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.120 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.120 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.120 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.120 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.120 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.120 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.120 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "password of InfluxDB server user", 2024-03-28T05:33:40.120 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.120 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.120 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.120 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.120 INFO:teuthology.orchestra.run.smithi023.stdout: "port": { 2024-03-28T05:33:40.121 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "port", 2024-03-28T05:33:40.121 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.121 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.121 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.121 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8086", 2024-03-28T05:33:40.121 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.121 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.121 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.121 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server port", 2024-03-28T05:33:40.121 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.121 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.121 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.121 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.121 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-28T05:33:40.121 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-28T05:33:40.121 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.121 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.121 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.122 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "false", 2024-03-28T05:33:40.122 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.122 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.122 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.122 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T05:33:40.122 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.122 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.122 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.122 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.122 INFO:teuthology.orchestra.run.smithi023.stdout: "threads": { 2024-03-28T05:33:40.122 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threads", 2024-03-28T05:33:40.122 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.122 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.122 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.122 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T05:33:40.122 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T05:33:40.122 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "32", 2024-03-28T05:33:40.123 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.123 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T05:33:40.123 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.123 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.123 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.123 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.123 INFO:teuthology.orchestra.run.smithi023.stdout: "username": { 2024-03-28T05:33:40.123 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "username", 2024-03-28T05:33:40.123 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.123 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.123 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.123 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.123 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.123 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.123 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.123 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "username of InfluxDB server user", 2024-03-28T05:33:40.123 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.123 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.123 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.124 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.124 INFO:teuthology.orchestra.run.smithi023.stdout: "verify_ssl": { 2024-03-28T05:33:40.124 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "verify_ssl", 2024-03-28T05:33:40.124 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.124 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.124 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.124 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "true", 2024-03-28T05:33:40.124 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.124 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.124 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.124 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T05:33:40.124 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.124 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.124 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.124 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.124 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.124 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.125 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.125 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "insights", 2024-03-28T05:33:40.125 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.125 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.125 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.125 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.125 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.125 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.125 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.125 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.125 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.125 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.125 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.125 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.125 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.125 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.125 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.126 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.126 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.126 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.126 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.126 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.126 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.126 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.126 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.126 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.126 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.126 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.126 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.126 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.126 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.126 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.126 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.126 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.127 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.127 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.127 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.127 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.127 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.127 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.127 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.127 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.127 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.127 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.127 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.127 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.127 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.127 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.127 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.127 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.127 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.127 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.128 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.128 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.128 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.128 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.128 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.128 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.128 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.128 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.128 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.128 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.128 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.128 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.128 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.128 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.128 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.128 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.129 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.129 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.129 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.129 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.129 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.129 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.129 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.129 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.129 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.129 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.129 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "iostat", 2024-03-28T05:33:40.129 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.129 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.129 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.129 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.129 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.129 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.130 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.130 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.130 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.130 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.130 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.130 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.130 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.130 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.130 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.130 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.130 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.130 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.130 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.130 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.130 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.130 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.131 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.131 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.131 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.131 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.131 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.131 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.131 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.131 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.131 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.131 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.131 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.131 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.131 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.131 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.131 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.131 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.131 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.132 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.132 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.132 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.132 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.132 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.132 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.132 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.132 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.132 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.132 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.132 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.132 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.132 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.132 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.132 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.132 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.132 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.132 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.133 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.133 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.133 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.133 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.133 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.133 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.133 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.133 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.133 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.133 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.133 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.133 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.133 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.133 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.133 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.133 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.133 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.134 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.134 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.134 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "k8sevents", 2024-03-28T05:33:40.134 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.134 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.134 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.134 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_event_retention_days": { 2024-03-28T05:33:40.134 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ceph_event_retention_days", 2024-03-28T05:33:40.134 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.134 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.134 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.134 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7", 2024-03-28T05:33:40.134 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.134 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.134 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.134 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T05:33:40.134 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.135 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.135 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.135 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.135 INFO:teuthology.orchestra.run.smithi023.stdout: "config_check_secs": { 2024-03-28T05:33:40.135 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_check_secs", 2024-03-28T05:33:40.135 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.135 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.135 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.135 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:40.135 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "10", 2024-03-28T05:33:40.135 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.135 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.135 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T05:33:40.135 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.135 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.135 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.135 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.136 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.136 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.136 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.136 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.136 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.136 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.136 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.136 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.136 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.136 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.136 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.136 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.136 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.136 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.136 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.136 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.136 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.136 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.136 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.137 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.137 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.137 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.137 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.137 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.137 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.137 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.137 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.137 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.137 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.137 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.137 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.137 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.137 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.137 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.137 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.137 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.138 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.138 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.138 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.138 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.138 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.138 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.138 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.138 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.138 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.138 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.138 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.138 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.138 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.138 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.138 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.138 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.138 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.139 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.139 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.139 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.139 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.139 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.139 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.139 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.139 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.139 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.139 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.139 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.139 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.139 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.139 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.139 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.139 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.140 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.140 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.140 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.140 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.140 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "localpool", 2024-03-28T05:33:40.140 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.140 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.140 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.140 INFO:teuthology.orchestra.run.smithi023.stdout: "failure_domain": { 2024-03-28T05:33:40.140 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "failure_domain", 2024-03-28T05:33:40.140 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.140 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.140 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.140 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "host", 2024-03-28T05:33:40.140 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.140 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.140 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.141 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "failure domain for any created local pool", 2024-03-28T05:33:40.141 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T05:33:40.141 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.141 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.141 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.141 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.141 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.141 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.141 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.141 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.141 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.141 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.141 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.141 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.141 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.141 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.141 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.141 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.142 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.142 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.142 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.142 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.142 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.142 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.142 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.142 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.142 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.142 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.142 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.142 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.142 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.142 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.142 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.142 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.143 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.143 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.143 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.143 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.143 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.143 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.143 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.143 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.143 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.143 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.143 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.143 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.143 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.143 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.143 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.143 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.143 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.143 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.144 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.144 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.144 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.144 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.144 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.144 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.144 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.144 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.144 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.144 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.144 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.144 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.144 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.144 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.144 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.144 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.144 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.145 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.145 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.145 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.145 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.145 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.145 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.145 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": { 2024-03-28T05:33:40.145 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_size", 2024-03-28T05:33:40.145 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.145 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.145 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.145 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.145 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.145 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.145 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.145 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default min_size for any created local pool", 2024-03-28T05:33:40.145 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T05:33:40.145 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.146 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.146 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.146 INFO:teuthology.orchestra.run.smithi023.stdout: "num_rep": { 2024-03-28T05:33:40.146 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "num_rep", 2024-03-28T05:33:40.146 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.146 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.146 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.146 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3", 2024-03-28T05:33:40.146 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.146 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.146 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.146 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default replica count for any created local pool", 2024-03-28T05:33:40.146 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.146 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.146 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.146 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.146 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": { 2024-03-28T05:33:40.147 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_num", 2024-03-28T05:33:40.147 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.147 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.147 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.147 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "128", 2024-03-28T05:33:40.147 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.147 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.147 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.147 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T05:33:40.147 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.147 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.147 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.147 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.147 INFO:teuthology.orchestra.run.smithi023.stdout: "prefix": { 2024-03-28T05:33:40.147 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prefix", 2024-03-28T05:33:40.147 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.147 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.147 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.148 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.148 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.148 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.148 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.148 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name prefix for any created local pool", 2024-03-28T05:33:40.148 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.148 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.148 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.148 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.148 INFO:teuthology.orchestra.run.smithi023.stdout: "subtree": { 2024-03-28T05:33:40.148 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "subtree", 2024-03-28T05:33:40.148 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.148 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.148 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.148 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "rack", 2024-03-28T05:33:40.148 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.148 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.148 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.149 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T05:33:40.149 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T05:33:40.149 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.149 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.149 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.149 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.149 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.149 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.149 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mds_autoscaler", 2024-03-28T05:33:40.149 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.149 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.149 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.149 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.149 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.149 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.149 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.149 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.150 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.150 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.150 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.150 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.150 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.150 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.150 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.150 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.150 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.150 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.150 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.150 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.150 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.150 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.150 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.150 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.150 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.151 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.151 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.151 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.151 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.151 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.151 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.151 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.151 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.151 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.151 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.151 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.151 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.151 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.151 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.151 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.151 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.151 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirroring", 2024-03-28T05:33:40.154 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.154 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.154 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.154 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.154 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.154 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.154 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.154 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.154 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.154 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.154 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.154 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.154 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.154 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.154 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.154 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.154 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.155 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.155 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.155 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.155 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.155 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.155 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.155 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.155 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.155 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.155 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.155 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.155 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.155 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.155 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.155 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.155 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.156 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.156 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.156 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.156 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.156 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.156 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.156 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.156 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.156 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.156 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.156 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.156 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.156 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.156 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.156 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.156 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.156 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.157 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.157 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.157 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.157 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.157 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.157 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.157 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.157 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.157 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.157 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.157 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.157 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.157 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.157 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.157 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.157 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.158 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.158 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.158 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.158 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.158 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.158 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.158 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.158 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.158 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.158 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.158 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "nfs", 2024-03-28T05:33:40.158 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.158 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.158 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.158 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.158 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.158 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.159 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.159 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.159 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.159 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.159 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.159 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.159 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.159 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.159 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.159 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.159 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.159 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.159 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.159 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.159 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.159 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.160 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.160 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.160 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.160 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.160 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.160 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.160 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.160 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.160 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.160 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.160 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.160 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.160 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.160 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.160 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.160 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.160 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.161 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.161 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.161 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.161 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.161 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.161 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.161 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.161 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.161 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.161 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.161 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.161 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.161 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.161 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.161 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.161 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.161 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.162 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.162 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.162 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.162 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.162 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.162 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.162 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.162 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.162 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.162 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.162 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.162 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.162 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.162 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.162 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.162 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-28T05:33:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: "fail_fs": { 2024-03-28T05:33:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "fail_fs", 2024-03-28T05:33:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T05:33:40.164 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.164 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.164 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.164 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.164 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.164 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.164 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.164 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.164 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.164 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.164 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.164 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.164 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.164 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.164 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.164 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.164 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.165 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.165 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.165 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.165 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.165 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.165 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.165 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.165 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.165 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.165 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.166 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.166 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.166 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.166 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.166 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.166 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.166 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.166 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.166 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.166 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.166 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.166 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.166 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.166 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.167 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.167 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.167 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.167 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.167 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.167 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.167 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.167 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.167 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.167 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.167 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.168 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.168 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.168 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.168 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.168 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.168 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.168 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.168 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.168 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.168 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.168 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.168 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.168 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.168 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.168 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.169 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.169 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.169 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.169 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.169 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.169 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.169 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.169 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator": { 2024-03-28T05:33:40.169 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-28T05:33:40.169 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.169 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.169 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.169 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.169 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.169 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.169 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.170 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2024-03-28T05:33:40.170 INFO:teuthology.orchestra.run.smithi023.stdout: "rook", 2024-03-28T05:33:40.170 INFO:teuthology.orchestra.run.smithi023.stdout: "test_orchestrator" 2024-03-28T05:33:40.170 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.170 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Orchestrator backend", 2024-03-28T05:33:40.170 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.170 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.170 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.170 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.170 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.170 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.170 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.170 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_perf_query", 2024-03-28T05:33:40.170 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.170 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.171 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.171 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.171 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.171 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.171 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.171 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.171 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.171 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.171 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.171 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.171 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.171 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.171 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.171 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.171 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.171 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.171 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.171 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.172 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.172 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.172 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.172 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.172 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.172 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.172 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.172 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.172 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.172 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.172 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.172 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.172 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.172 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.172 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.172 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.172 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.173 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.173 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.173 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.173 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.173 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.173 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.173 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.173 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.173 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.173 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.173 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.173 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.173 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.173 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.173 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.173 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.173 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.174 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.174 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.174 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.174 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.174 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.174 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.174 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.174 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.174 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.174 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.174 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.174 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.174 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.174 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.174 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.174 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.175 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.175 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.175 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.175 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.175 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.175 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.175 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_support", 2024-03-28T05:33:40.175 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.175 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.175 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.175 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.175 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.175 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.175 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.175 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.177 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.177 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.177 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.177 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.177 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.177 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.177 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.177 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.177 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.177 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.177 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.177 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.177 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.177 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.177 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.177 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.177 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.178 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.178 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.178 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.178 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.178 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.178 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.178 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.178 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.178 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.178 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.178 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.178 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.178 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.178 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.178 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.178 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.178 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.179 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.179 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.179 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.179 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.179 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.179 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.179 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.179 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.179 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.179 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.179 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.179 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.179 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.179 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.179 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.179 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.179 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.180 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_autoscaler", 2024-03-28T05:33:40.180 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.180 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.180 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.180 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.180 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.180 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.180 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.180 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.180 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.180 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.180 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.180 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.180 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.180 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.180 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.181 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.181 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.181 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.181 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.181 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.181 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.181 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.181 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.181 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.181 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.181 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.181 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.181 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.181 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.181 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.181 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.181 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.182 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.182 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.182 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.182 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.182 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.182 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.182 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.182 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.182 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.182 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.182 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.182 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.182 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.182 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.182 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.182 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.182 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.183 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.183 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.183 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.183 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.183 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.183 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.183 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.183 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.183 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.183 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.183 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.183 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.183 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.183 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.183 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.183 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.183 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T05:33:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T05:33:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T05:33:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.185 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.185 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.185 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.185 INFO:teuthology.orchestra.run.smithi023.stdout: "threshold": { 2024-03-28T05:33:40.185 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threshold", 2024-03-28T05:33:40.185 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T05:33:40.185 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.185 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.185 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-28T05:33:40.185 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1.0", 2024-03-28T05:33:40.185 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.185 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.185 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "scaling threshold", 2024-03-28T05:33:40.185 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T05:33:40.185 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.185 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.185 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.186 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.186 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.186 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.186 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "progress", 2024-03-28T05:33:40.186 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.186 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.186 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.186 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_pg_recovery_event": { 2024-03-28T05:33:40.186 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_pg_recovery_event", 2024-03-28T05:33:40.186 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.186 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.186 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.186 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.186 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.186 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.186 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.186 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T05:33:40.187 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.187 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.187 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.187 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.187 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-28T05:33:40.187 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-28T05:33:40.187 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.187 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.187 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.187 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.187 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.187 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.187 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.187 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.187 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.187 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.187 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.188 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.188 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.188 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.188 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.188 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.188 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.188 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.188 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.188 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.188 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.188 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.188 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.188 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.188 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.188 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.188 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.188 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.188 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.189 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.189 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.189 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.189 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.189 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.189 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.189 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.189 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.189 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.189 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.189 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.189 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.189 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.189 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.189 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.189 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.189 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.189 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.190 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.190 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.190 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.190 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.190 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.190 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.190 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.190 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.190 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.190 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.190 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.190 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.190 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.190 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.190 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.190 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.191 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.191 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.191 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.191 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.191 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.191 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.191 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.191 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.191 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.191 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.191 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.191 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.191 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.191 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.191 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.191 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.191 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.192 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.192 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.192 INFO:teuthology.orchestra.run.smithi023.stdout: "max_completed_events": { 2024-03-28T05:33:40.192 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_completed_events", 2024-03-28T05:33:40.192 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.192 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.192 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.192 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "50", 2024-03-28T05:33:40.192 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.192 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.192 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.192 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "number of past completed events to remember", 2024-03-28T05:33:40.192 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.192 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.192 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.192 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.192 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T05:33:40.192 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T05:33:40.193 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:40.193 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.193 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.193 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T05:33:40.193 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.193 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.193 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.193 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long the module is going to sleep", 2024-03-28T05:33:40.193 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.193 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.193 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.193 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.193 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.193 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.193 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.193 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus", 2024-03-28T05:33:40.193 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.194 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.194 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.194 INFO:teuthology.orchestra.run.smithi023.stdout: "cache": { 2024-03-28T05:33:40.194 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cache", 2024-03-28T05:33:40.194 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.194 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.194 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.194 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.194 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.194 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.194 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.194 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.194 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.194 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.194 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.194 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.194 INFO:teuthology.orchestra.run.smithi023.stdout: "exclude_perf_counters": { 2024-03-28T05:33:40.194 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "exclude_perf_counters", 2024-03-28T05:33:40.195 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.195 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.195 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.195 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.195 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.195 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.195 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.195 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T05:33:40.195 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T05:33:40.195 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.195 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.195 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.195 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.195 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.195 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.195 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.195 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.196 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.196 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.196 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.196 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.196 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.196 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.196 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.196 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.196 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.196 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.196 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.196 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.196 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.196 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.196 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.196 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.196 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.196 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.197 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.197 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.197 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.197 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.197 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.197 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.197 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.197 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.197 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.197 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.197 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.197 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.197 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.197 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.197 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.197 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.198 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.198 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.198 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.198 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.198 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.198 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.198 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.198 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.198 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.198 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.198 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.198 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.198 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.198 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.198 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.198 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.198 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.198 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.199 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.199 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.199 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.199 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.199 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.199 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.199 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.199 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.199 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.199 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.199 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.199 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.199 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.199 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools": { 2024-03-28T05:33:40.199 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools", 2024-03-28T05:33:40.200 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.200 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.200 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.200 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.200 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.200 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.200 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.200 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.200 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.200 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.200 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.200 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.200 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T05:33:40.200 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T05:33:40.200 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.201 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.201 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.201 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "300", 2024-03-28T05:33:40.201 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.201 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.201 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.201 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.201 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.201 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.201 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.201 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.201 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_interval": { 2024-03-28T05:33:40.201 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_interval", 2024-03-28T05:33:40.201 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T05:33:40.201 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.201 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.201 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15.0", 2024-03-28T05:33:40.202 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.202 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.202 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.202 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.202 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.202 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.202 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.202 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.202 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-28T05:33:40.202 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-28T05:33:40.202 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.202 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.202 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.202 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-28T05:33:40.202 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.203 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.203 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.203 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T05:33:40.203 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.203 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.203 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.203 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.203 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-28T05:33:40.203 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-28T05:33:40.203 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.203 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.203 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.203 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "9283", 2024-03-28T05:33:40.203 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.203 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.203 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.204 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T05:33:40.204 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.204 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.204 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.204 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.204 INFO:teuthology.orchestra.run.smithi023.stdout: "stale_cache_strategy": { 2024-03-28T05:33:40.204 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stale_cache_strategy", 2024-03-28T05:33:40.204 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.204 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.204 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.204 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "log", 2024-03-28T05:33:40.204 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.204 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.204 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.204 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.204 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.205 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.205 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.205 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.205 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-28T05:33:40.205 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-28T05:33:40.205 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.205 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.205 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.205 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "default", 2024-03-28T05:33:40.205 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.205 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.205 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.205 INFO:teuthology.orchestra.run.smithi023.stdout: "default", 2024-03-28T05:33:40.205 INFO:teuthology.orchestra.run.smithi023.stdout: "error" 2024-03-28T05:33:40.205 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.205 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.205 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.206 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.206 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.206 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.206 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-28T05:33:40.206 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-28T05:33:40.206 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.206 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.206 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.206 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-28T05:33:40.206 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-28T05:33:40.206 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-28T05:33:40.206 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.206 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.206 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.206 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.206 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.206 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.206 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.207 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.207 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.207 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2024-03-28T05:33:40.207 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.207 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.207 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.207 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.207 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.207 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.207 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.207 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.207 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.207 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.207 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.207 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.207 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.207 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.208 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.208 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.208 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.208 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.208 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.208 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.208 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.208 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.208 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.208 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.208 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.208 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.208 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.208 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.208 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.208 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.208 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.209 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.209 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.209 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.209 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.209 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.209 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.209 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.209 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.209 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.209 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.209 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.209 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.209 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.209 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.209 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.209 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.210 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.210 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.210 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.210 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.210 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.210 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.210 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.210 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.210 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.210 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.210 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.210 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.210 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.210 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.210 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.210 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.211 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.211 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.211 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.211 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.211 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.211 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.211 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.211 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.211 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.211 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.211 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_snap_create": { 2024-03-28T05:33:40.211 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_snap_create", 2024-03-28T05:33:40.211 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.211 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.211 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.211 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T05:33:40.211 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.211 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.212 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.212 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.212 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.212 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.212 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.212 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.212 INFO:teuthology.orchestra.run.smithi023.stdout: "mirror_snapshot_schedule": { 2024-03-28T05:33:40.212 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T05:33:40.212 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.212 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.212 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.212 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.212 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.212 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.212 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.212 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.212 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.213 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.213 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.213 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.213 INFO:teuthology.orchestra.run.smithi023.stdout: "trash_purge_schedule": { 2024-03-28T05:33:40.213 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "trash_purge_schedule", 2024-03-28T05:33:40.213 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.213 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.213 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.213 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.213 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.213 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.213 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.213 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.213 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.213 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.213 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.213 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.213 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.214 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.214 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.214 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "restful", 2024-03-28T05:33:40.214 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.214 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.214 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.214 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_auth": { 2024-03-28T05:33:40.214 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_auth", 2024-03-28T05:33:40.214 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.214 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.214 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.214 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.214 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.214 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.214 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.214 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.215 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.215 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.215 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.215 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.215 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-28T05:33:40.215 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-28T05:33:40.215 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.215 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.215 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.215 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.215 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.215 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.215 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.215 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.215 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.215 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.215 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.215 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.216 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.216 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.216 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.216 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.216 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.216 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.216 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.216 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.216 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.216 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.216 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.216 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.216 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.216 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.216 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.216 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.216 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.217 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.217 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.217 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.217 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.217 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.217 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.217 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.217 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.217 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.217 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.217 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.217 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.217 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.217 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.217 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.217 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.217 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.217 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.218 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.218 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.218 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.218 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.218 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.218 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.218 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.218 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.218 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.218 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.218 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.218 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.218 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.218 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.218 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.219 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.219 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.219 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.219 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.219 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.219 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.219 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.219 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.219 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.219 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.219 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.219 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.219 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.219 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.219 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.219 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.220 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.220 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.220 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.220 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.220 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-28T05:33:40.220 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-28T05:33:40.220 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.220 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.220 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.220 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.220 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.220 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.220 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.220 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.220 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.220 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.220 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.220 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.221 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-28T05:33:40.221 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-28T05:33:40.221 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.221 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.221 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.221 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.221 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.221 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.221 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.221 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.221 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.221 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.221 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.221 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.221 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.221 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.221 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.221 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rgw", 2024-03-28T05:33:40.222 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.222 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.222 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.222 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.222 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.222 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.222 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.222 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.222 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.222 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.222 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.222 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.222 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.222 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.222 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.222 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.223 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.223 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.223 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.223 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.223 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.223 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.223 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.223 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.223 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.223 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.223 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.223 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.223 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.223 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.223 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.223 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.223 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.223 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.224 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.224 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.224 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.224 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.224 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.224 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.224 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.224 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.224 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.224 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.224 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.224 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.224 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.224 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.224 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.224 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.224 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.225 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.225 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.225 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.225 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.225 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.225 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.225 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.225 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.225 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.225 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.225 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.225 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.225 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.225 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.225 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.225 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.225 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.226 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.226 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.226 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.226 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.226 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.226 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.226 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.226 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.226 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rook", 2024-03-28T05:33:40.226 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.226 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.226 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.226 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.226 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.226 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.226 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.226 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.227 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.227 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.227 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.227 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.227 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.227 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.227 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.227 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.227 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.227 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.227 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.227 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.227 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.227 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.227 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.227 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.227 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.228 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.228 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.228 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.228 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.228 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.228 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.228 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.228 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.228 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.228 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.228 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.228 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.228 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.228 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.228 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.228 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.229 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.229 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.229 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.229 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.229 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.229 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.229 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.229 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.229 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.229 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.229 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.229 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.229 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.229 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.229 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.229 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.229 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.230 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.230 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.230 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.230 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.230 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.230 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.230 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.230 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.230 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.230 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.230 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.230 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.230 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.230 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.230 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.230 INFO:teuthology.orchestra.run.smithi023.stdout: "storage_class": { 2024-03-28T05:33:40.231 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "storage_class", 2024-03-28T05:33:40.231 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.231 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.231 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.231 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "local", 2024-03-28T05:33:40.231 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.231 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.233 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.233 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T05:33:40.233 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.233 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.233 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.233 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.233 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.234 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.234 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.234 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "selftest", 2024-03-28T05:33:40.234 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.234 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.234 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.234 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.234 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.234 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.234 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.234 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.234 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.234 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.234 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.234 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.235 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.235 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.235 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.235 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.235 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.235 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.235 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.235 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.235 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.235 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.235 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.235 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.235 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.235 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.235 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.235 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.235 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.236 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.236 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.236 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.236 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.236 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.236 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.236 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.236 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.236 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.236 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.236 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.236 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.236 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.236 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.236 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.236 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.236 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.236 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.236 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.237 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.237 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.237 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.237 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.237 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.237 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.237 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.237 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.237 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.237 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.237 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.237 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.237 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.237 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.237 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.237 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.237 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.237 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.238 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.238 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.238 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.238 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.238 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.238 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.238 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.238 INFO:teuthology.orchestra.run.smithi023.stdout: "roption1": { 2024-03-28T05:33:40.238 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption1", 2024-03-28T05:33:40.238 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.238 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.238 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.238 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.238 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.238 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.238 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.239 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.239 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.239 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.239 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.239 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.239 INFO:teuthology.orchestra.run.smithi023.stdout: "roption2": { 2024-03-28T05:33:40.239 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption2", 2024-03-28T05:33:40.239 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.239 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.239 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.239 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "xyz", 2024-03-28T05:33:40.239 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.239 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.239 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.239 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.239 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.239 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.240 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.240 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.240 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption1": { 2024-03-28T05:33:40.240 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption1", 2024-03-28T05:33:40.240 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.240 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.240 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.240 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.240 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.240 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.240 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.240 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.240 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.240 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.240 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.240 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.240 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption2": { 2024-03-28T05:33:40.241 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption2", 2024-03-28T05:33:40.241 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.241 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.241 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.241 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.241 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.241 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.241 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.241 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.241 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.241 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.241 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.241 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.241 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption3": { 2024-03-28T05:33:40.241 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption3", 2024-03-28T05:33:40.241 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T05:33:40.241 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.241 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.242 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.242 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.242 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.242 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.242 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.242 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.242 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.242 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.242 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.242 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption4": { 2024-03-28T05:33:40.242 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption4", 2024-03-28T05:33:40.242 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.242 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.242 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.242 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.242 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.242 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.243 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.243 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.243 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.243 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.243 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.243 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.243 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption5": { 2024-03-28T05:33:40.243 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption5", 2024-03-28T05:33:40.243 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.243 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.243 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.243 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.243 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.243 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.243 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.243 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.243 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.244 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.244 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.244 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption6": { 2024-03-28T05:33:40.244 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption6", 2024-03-28T05:33:40.244 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.244 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.244 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.244 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.244 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.244 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.244 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.244 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.244 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.244 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.244 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.245 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption7": { 2024-03-28T05:33:40.245 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption7", 2024-03-28T05:33:40.245 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.245 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.245 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.245 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.245 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T05:33:40.245 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "42", 2024-03-28T05:33:40.245 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.245 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.245 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.245 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.245 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.245 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.245 INFO:teuthology.orchestra.run.smithi023.stdout: "testkey": { 2024-03-28T05:33:40.245 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testkey", 2024-03-28T05:33:40.245 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.246 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.246 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.246 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.246 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.246 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.246 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.246 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.246 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.246 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.246 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.246 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.246 INFO:teuthology.orchestra.run.smithi023.stdout: "testlkey": { 2024-03-28T05:33:40.246 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testlkey", 2024-03-28T05:33:40.246 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.246 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.246 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.247 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.247 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.247 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.247 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.247 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.247 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.247 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.247 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.247 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.247 INFO:teuthology.orchestra.run.smithi023.stdout: "testnewline": { 2024-03-28T05:33:40.247 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testnewline", 2024-03-28T05:33:40.247 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.247 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.247 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.247 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.247 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.247 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.247 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.248 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.248 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.248 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.248 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.248 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.248 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.248 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.248 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.248 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-28T05:33:40.248 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.248 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.248 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.248 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_m_granularity": { 2024-03-28T05:33:40.248 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_m_granularity", 2024-03-28T05:33:40.248 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.248 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.248 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.249 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.249 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.249 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.249 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.249 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T05:33:40.249 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.249 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.249 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.249 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.249 INFO:teuthology.orchestra.run.smithi023.stdout: "dump_on_update": { 2024-03-28T05:33:40.249 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dump_on_update", 2024-03-28T05:33:40.249 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.249 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.249 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.249 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.250 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.250 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.250 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.250 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "dump database to debug log on update", 2024-03-28T05:33:40.250 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.250 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.250 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.250 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.250 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.250 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.250 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.250 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.250 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.250 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.250 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.250 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.250 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.251 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.251 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.251 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.251 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.251 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.251 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.251 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.251 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.251 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.251 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.251 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.251 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.251 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.251 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.251 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.251 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.251 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.252 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.252 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.252 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.252 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.252 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.252 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.252 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.252 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.252 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.252 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.252 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.252 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.252 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.252 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.252 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.252 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.252 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.252 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.252 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.253 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.253 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.253 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.253 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.253 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.253 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.253 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.253 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.253 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.253 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.253 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.253 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.253 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.253 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.253 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.253 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.253 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.254 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.254 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.254 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.254 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.254 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.254 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.254 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.254 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.254 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.254 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.254 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.254 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stats", 2024-03-28T05:33:40.254 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.254 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.255 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.255 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.255 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.255 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.255 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.255 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.255 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.255 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.255 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.255 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.255 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.255 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.255 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.255 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.255 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.255 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.255 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.255 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.256 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.256 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.256 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.256 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.256 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.256 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.256 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.256 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.256 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.256 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.256 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.256 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.256 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.256 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.256 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.256 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.256 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.256 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.257 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.257 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.257 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.257 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.257 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.257 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.257 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.257 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.257 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.257 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.257 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.257 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.257 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.257 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.257 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.257 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.257 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.258 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.258 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.258 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.258 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.258 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.258 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.258 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.258 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.258 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.258 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.258 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.258 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.258 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.258 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.258 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.259 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.259 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.259 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.259 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.259 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.259 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.259 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "status", 2024-03-28T05:33:40.259 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.259 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.259 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.259 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.259 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.259 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.259 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.259 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.259 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.260 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.260 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.260 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.260 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.260 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.260 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.260 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.260 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.260 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.260 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.260 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.260 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.260 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.260 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.260 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.261 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.261 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.261 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.261 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.261 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.261 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.261 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.261 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.261 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.261 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.261 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.261 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.261 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.261 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.261 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.261 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.261 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.262 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.262 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.262 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.262 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.262 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.262 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.262 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.262 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.262 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.262 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.262 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.262 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.262 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.262 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.262 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.263 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.263 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.263 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.263 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.263 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.263 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.263 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.263 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.263 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.263 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.263 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.263 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.263 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.263 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.263 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.263 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.264 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.264 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.264 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.264 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.264 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telegraf", 2024-03-28T05:33:40.264 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.264 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.264 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.264 INFO:teuthology.orchestra.run.smithi023.stdout: "address": { 2024-03-28T05:33:40.264 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "address", 2024-03-28T05:33:40.264 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.264 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.264 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.264 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T05:33:40.264 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.264 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.264 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.265 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.265 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.265 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.265 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.265 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.265 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T05:33:40.265 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T05:33:40.265 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:40.265 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.265 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.265 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15", 2024-03-28T05:33:40.265 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.265 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.265 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.265 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.265 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.265 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.266 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.266 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.266 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.266 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.266 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.266 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.266 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.266 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.266 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.266 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.266 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.266 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.266 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.266 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.266 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.267 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.267 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.267 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.267 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.267 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.267 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.267 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.267 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.267 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.267 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.267 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.267 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.267 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.267 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.267 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.267 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.268 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.268 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.268 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.268 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.268 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.268 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.268 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.268 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.268 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.268 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.268 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.268 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.268 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.268 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.268 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.269 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.269 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.269 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.269 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.269 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.269 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.269 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.269 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.269 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.269 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.269 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.269 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.269 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.269 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.269 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.269 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.269 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.270 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.270 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.270 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.270 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.270 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.270 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.270 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.270 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.270 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.270 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.270 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.270 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telemetry", 2024-03-28T05:33:40.270 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.270 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.270 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.271 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_basic": { 2024-03-28T05:33:40.271 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_basic", 2024-03-28T05:33:40.271 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.271 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.271 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.271 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.271 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.271 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.271 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.271 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T05:33:40.271 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.271 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.271 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.271 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.271 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_crash": { 2024-03-28T05:33:40.271 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_crash", 2024-03-28T05:33:40.272 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.272 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.272 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.272 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.272 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.272 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.272 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.272 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T05:33:40.272 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.272 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.272 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.272 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.272 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_device": { 2024-03-28T05:33:40.272 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_device", 2024-03-28T05:33:40.272 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.272 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.273 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.273 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.273 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.273 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.273 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.273 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T05:33:40.273 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.273 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.273 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.273 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.273 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_ident": { 2024-03-28T05:33:40.273 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_ident", 2024-03-28T05:33:40.273 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.273 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.273 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.273 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.274 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.274 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.274 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.274 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T05:33:40.274 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.274 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.274 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.274 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.274 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_perf": { 2024-03-28T05:33:40.274 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_perf", 2024-03-28T05:33:40.274 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.274 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.274 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.274 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.274 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.275 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.275 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.275 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T05:33:40.275 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.275 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.275 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.275 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.275 INFO:teuthology.orchestra.run.smithi023.stdout: "contact": { 2024-03-28T05:33:40.275 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "contact", 2024-03-28T05:33:40.275 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.275 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.275 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.275 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.275 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.275 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.276 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.276 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.276 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.276 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.276 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.276 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.276 INFO:teuthology.orchestra.run.smithi023.stdout: "description": { 2024-03-28T05:33:40.276 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "description", 2024-03-28T05:33:40.276 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.276 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.276 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.276 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.276 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.276 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.276 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.277 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.277 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.277 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.277 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.277 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.277 INFO:teuthology.orchestra.run.smithi023.stdout: "device_url": { 2024-03-28T05:33:40.277 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_url", 2024-03-28T05:33:40.277 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.277 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.277 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.277 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T05:33:40.277 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.277 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.277 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.277 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.277 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.278 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.278 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.278 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.278 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-28T05:33:40.278 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-28T05:33:40.278 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.278 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.278 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.278 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.278 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.278 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.278 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.278 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.278 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.278 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.279 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.279 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.279 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T05:33:40.279 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T05:33:40.279 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.279 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.279 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.279 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "24", 2024-03-28T05:33:40.279 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "8", 2024-03-28T05:33:40.279 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.279 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.279 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.279 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.279 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.279 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.279 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.280 INFO:teuthology.orchestra.run.smithi023.stdout: "last_opt_revision": { 2024-03-28T05:33:40.280 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "last_opt_revision", 2024-03-28T05:33:40.280 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.280 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.280 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.280 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1", 2024-03-28T05:33:40.280 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.280 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.280 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.280 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.280 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.280 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.280 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.280 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.280 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard": { 2024-03-28T05:33:40.280 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard", 2024-03-28T05:33:40.281 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.281 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.281 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.281 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.281 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.281 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.281 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.281 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.281 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.281 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.281 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.281 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.281 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard_description": { 2024-03-28T05:33:40.281 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard_description", 2024-03-28T05:33:40.281 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.281 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.282 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.282 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.282 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.282 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.282 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.282 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.282 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.282 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.282 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.282 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.282 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.282 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.282 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.282 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.282 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.283 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.283 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.283 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.283 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.283 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.283 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.283 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.283 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.283 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.283 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.283 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.283 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.283 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.283 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.283 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.283 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.284 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.284 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.284 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.284 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.284 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.284 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.284 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.284 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.284 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.284 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.284 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.284 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.284 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.284 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.285 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.285 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.285 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.285 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.285 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.285 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.285 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.285 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.285 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.285 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.285 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.285 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.285 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.285 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.285 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.285 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.285 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.286 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.286 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.286 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.286 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.286 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.286 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.286 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.286 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.286 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.286 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.286 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.286 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.286 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.286 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.287 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.287 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.287 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.287 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.287 INFO:teuthology.orchestra.run.smithi023.stdout: "organization": { 2024-03-28T05:33:40.287 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "organization", 2024-03-28T05:33:40.287 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.287 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.287 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.287 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.287 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.287 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.287 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.287 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.287 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.287 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.288 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.288 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.288 INFO:teuthology.orchestra.run.smithi023.stdout: "proxy": { 2024-03-28T05:33:40.288 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "proxy", 2024-03-28T05:33:40.288 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.288 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.288 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.288 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.288 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.288 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.288 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.288 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.288 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.288 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.288 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.289 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.289 INFO:teuthology.orchestra.run.smithi023.stdout: "url": { 2024-03-28T05:33:40.289 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url", 2024-03-28T05:33:40.289 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.289 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.289 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.289 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T05:33:40.289 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.289 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.289 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.289 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.289 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.289 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.289 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.289 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.290 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.290 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.290 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.290 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "test_orchestrator", 2024-03-28T05:33:40.290 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.290 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.290 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.290 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.290 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.290 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.290 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.290 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.290 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.290 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.291 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.291 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.291 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.291 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.291 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.291 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.291 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.291 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.291 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.291 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.291 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.291 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.291 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.291 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.291 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.291 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.291 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.291 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.291 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.292 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.292 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.292 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.292 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.292 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.292 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.292 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.292 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.292 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.292 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.292 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.292 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.292 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.292 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.292 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.292 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.292 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.293 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.293 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.293 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.293 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.293 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.293 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.293 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.293 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.293 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.293 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.293 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.293 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.293 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.293 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.293 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.293 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.294 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.294 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.294 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.294 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.294 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.294 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.294 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.294 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.294 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.294 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.294 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.294 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.294 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.294 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.294 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2024-03-28T05:33:40.294 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.295 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.295 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.295 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.295 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.295 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.295 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.295 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.295 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.295 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.295 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.295 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.295 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.295 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.295 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.295 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.295 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.295 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.295 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.296 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.296 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.296 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.296 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.296 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.296 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.296 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.296 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.296 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.296 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.296 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.296 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.296 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.296 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.296 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.296 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.296 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.297 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.297 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.297 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.297 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.297 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.297 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.297 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.297 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.297 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.297 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.297 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.297 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.297 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.297 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.297 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.297 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.297 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.297 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.298 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.298 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.298 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.298 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.298 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.298 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.298 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.298 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.298 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.298 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.298 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.298 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.298 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.298 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.298 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.299 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.299 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.299 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.299 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.299 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_clones": { 2024-03-28T05:33:40.299 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_clones", 2024-03-28T05:33:40.299 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.299 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.299 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.299 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4", 2024-03-28T05:33:40.299 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.299 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.299 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.299 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T05:33:40.299 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.299 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.299 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.300 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.300 INFO:teuthology.orchestra.run.smithi023.stdout: "periodic_async_work": { 2024-03-28T05:33:40.300 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "periodic_async_work", 2024-03-28T05:33:40.300 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.300 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.300 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.300 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.300 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.300 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.300 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.300 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Periodically check for async work", 2024-03-28T05:33:40.300 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.300 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.300 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.300 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.300 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_delay": { 2024-03-28T05:33:40.300 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_delay", 2024-03-28T05:33:40.301 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.301 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.301 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.301 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T05:33:40.301 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.301 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.301 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.301 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T05:33:40.301 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.301 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.301 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.301 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.301 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_no_wait": { 2024-03-28T05:33:40.301 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T05:33:40.301 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.301 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.302 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.302 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T05:33:40.302 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.302 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.302 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.302 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T05:33:40.302 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.302 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.302 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.302 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.302 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.302 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.302 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.302 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix", 2024-03-28T05:33:40.302 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T05:33:40.302 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T05:33:40.303 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T05:33:40.303 INFO:teuthology.orchestra.run.smithi023.stdout: "discovery_interval": { 2024-03-28T05:33:40.303 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "discovery_interval", 2024-03-28T05:33:40.303 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T05:33:40.303 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.303 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.303 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "100", 2024-03-28T05:33:40.303 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.303 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.303 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.303 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.303 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.303 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.303 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.303 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.303 INFO:teuthology.orchestra.run.smithi023.stdout: "identifier": { 2024-03-28T05:33:40.303 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "identifier", 2024-03-28T05:33:40.303 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.304 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.304 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.304 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.304 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.304 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.304 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.304 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.304 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.304 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.304 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.304 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.304 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T05:33:40.304 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T05:33:40.304 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T05:33:40.304 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.304 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.304 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T05:33:40.305 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.305 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.305 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.305 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.305 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.305 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.305 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.305 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.305 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T05:33:40.305 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T05:33:40.305 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.305 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.305 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.305 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.305 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.305 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.305 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.305 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.306 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.306 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.306 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.306 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.306 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.306 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.306 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.306 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.306 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.306 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.306 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.306 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T05:33:40.306 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T05:33:40.306 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.306 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.306 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.307 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.307 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.307 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.307 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.307 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.307 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.307 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.307 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.307 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.307 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T05:33:40.307 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T05:33:40.307 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.307 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.307 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.307 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T05:33:40.307 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.307 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.307 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T05:33:40.308 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T05:33:40.308 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T05:33:40.308 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T05:33:40.308 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T05:33:40.308 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T05:33:40.308 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T05:33:40.308 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.308 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.308 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.308 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.308 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.308 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.308 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T05:33:40.308 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T05:33:40.308 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T05:33:40.308 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.308 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T05:33:40.308 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T05:33:40.309 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.309 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.309 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.309 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.309 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.309 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.309 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.309 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.309 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_host": { 2024-03-28T05:33:40.309 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_host", 2024-03-28T05:33:40.309 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.309 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.309 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.309 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T05:33:40.309 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.309 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.309 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.310 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.310 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.310 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.310 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.310 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.310 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_port": { 2024-03-28T05:33:40.310 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_port", 2024-03-28T05:33:40.310 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T05:33:40.310 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.310 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.310 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10051", 2024-03-28T05:33:40.310 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.310 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.310 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.310 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.311 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.311 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.311 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.311 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.311 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_sender": { 2024-03-28T05:33:40.311 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_sender", 2024-03-28T05:33:40.311 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T05:33:40.311 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T05:33:40.311 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T05:33:40.311 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T05:33:40.311 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T05:33:40.311 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T05:33:40.311 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T05:33:40.311 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T05:33:40.311 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T05:33:40.311 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T05:33:40.311 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T05:33:40.311 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.312 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.312 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.312 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.312 INFO:teuthology.orchestra.run.smithi023.stdout: "services": { 2024-03-28T05:33:40.312 INFO:teuthology.orchestra.run.smithi023.stdout: "dashboard": "https://172.21.15.23:8443/" 2024-03-28T05:33:40.312 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.312 INFO:teuthology.orchestra.run.smithi023.stdout: "always_on_modules": { 2024-03-28T05:33:40.312 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus": [ 2024-03-28T05:33:40.312 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-28T05:33:40.312 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-28T05:33:40.312 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-28T05:33:40.312 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-28T05:33:40.312 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-28T05:33:40.312 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-28T05:33:40.312 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-28T05:33:40.312 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-28T05:33:40.312 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-28T05:33:40.312 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-28T05:33:40.313 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.313 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific": [ 2024-03-28T05:33:40.313 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-28T05:33:40.313 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-28T05:33:40.313 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-28T05:33:40.313 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-28T05:33:40.313 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-28T05:33:40.313 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-28T05:33:40.313 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-28T05:33:40.313 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-28T05:33:40.313 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-28T05:33:40.313 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-28T05:33:40.313 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.313 INFO:teuthology.orchestra.run.smithi023.stdout: "quincy": [ 2024-03-28T05:33:40.313 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-28T05:33:40.313 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-28T05:33:40.313 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-28T05:33:40.313 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-28T05:33:40.314 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-28T05:33:40.314 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-28T05:33:40.314 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-28T05:33:40.314 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-28T05:33:40.314 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-28T05:33:40.314 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-28T05:33:40.314 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.314 INFO:teuthology.orchestra.run.smithi023.stdout: "reef": [ 2024-03-28T05:33:40.314 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-28T05:33:40.314 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-28T05:33:40.314 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-28T05:33:40.314 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-28T05:33:40.314 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-28T05:33:40.314 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-28T05:33:40.314 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-28T05:33:40.314 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-28T05:33:40.315 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-28T05:33:40.315 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-28T05:33:40.315 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T05:33:40.315 INFO:teuthology.orchestra.run.smithi023.stdout: "squid": [ 2024-03-28T05:33:40.315 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-28T05:33:40.315 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-28T05:33:40.315 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-28T05:33:40.315 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-28T05:33:40.315 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-28T05:33:40.315 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-28T05:33:40.315 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-28T05:33:40.315 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-28T05:33:40.315 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-28T05:33:40.315 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-28T05:33:40.315 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T05:33:40.315 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.315 INFO:teuthology.orchestra.run.smithi023.stdout: "last_failure_osd_epoch": 94, 2024-03-28T05:33:40.315 INFO:teuthology.orchestra.run.smithi023.stdout: "active_clients": [ 2024-03-28T05:33:40.316 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.316 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2024-03-28T05:33:40.316 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-28T05:33:40.316 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.316 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-28T05:33:40.316 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-28T05:33:40.316 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 388752193 2024-03-28T05:33:40.316 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.316 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T05:33:40.316 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.316 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.316 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "libcephsqlite", 2024-03-28T05:33:40.316 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-28T05:33:40.316 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.316 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-28T05:33:40.316 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-28T05:33:40.316 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3933268046 2024-03-28T05:33:40.317 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.317 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T05:33:40.317 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.317 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.317 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2024-03-28T05:33:40.317 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-28T05:33:40.317 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.317 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-28T05:33:40.317 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-28T05:33:40.317 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 922824272 2024-03-28T05:33:40.317 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.317 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T05:33:40.317 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.317 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.317 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-28T05:33:40.317 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-28T05:33:40.317 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.317 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-28T05:33:40.317 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-28T05:33:40.318 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 404195132 2024-03-28T05:33:40.318 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.318 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T05:33:40.318 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.318 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.318 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-28T05:33:40.318 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-28T05:33:40.318 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.318 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-28T05:33:40.318 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-28T05:33:40.318 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1411750300 2024-03-28T05:33:40.318 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.318 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T05:33:40.318 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:40.318 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.319 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2024-03-28T05:33:40.319 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-28T05:33:40.319 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T05:33:40.319 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-28T05:33:40.319 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-28T05:33:40.319 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1680570682 2024-03-28T05:33:40.319 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.319 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T05:33:40.319 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:40.319 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T05:33:40.319 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T05:33:40.320 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:39 smithi098 ceph-mon[36087]: pgmap v313: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:33:40.321 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:39 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1354037424' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T05:33:40.321 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:38 smithi113 ceph-mon[37164]: from='client.25927 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:33:40.321 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:38 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3825872786' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:33:40.321 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:38 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2104701992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:33:40.321 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:38 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3523010930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:33:40.321 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:38 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3981627898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:33:40.321 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:39 smithi113 ceph-mon[37164]: pgmap v313: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:33:40.321 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:39 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1354037424' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T05:33:40.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:39.198+0000 7f10c3b87640 1 -- 172.21.15.23:0/1354037424 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f10a80a29a0 msgr2=0x7f10a80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:40.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:39.198+0000 7f10c3b87640 1 --2- 172.21.15.23:0/1354037424 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f10a80a29a0 0x7f10a80a4e60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f10b8005d00 tx=0x7f10b800b410 comp rx=0 tx=0).stop 2024-03-28T05:33:40.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:39.198+0000 7f10c3b87640 1 -- 172.21.15.23:0/1354037424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10bc06a6f0 msgr2=0x7f10bc100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:33:40.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:39.198+0000 7f10c3b87640 1 --2- 172.21.15.23:0/1354037424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10bc06a6f0 0x7f10bc100b10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f10ac005940 tx=0x7f10ac02cc40 comp rx=0 tx=0).stop 2024-03-28T05:33:40.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:39.198+0000 7f10c3b87640 1 -- 172.21.15.23:0/1354037424 shutdown_connections 2024-03-28T05:33:40.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:39.198+0000 7f10c3b87640 1 --2- 172.21.15.23:0/1354037424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10bc1076a0 0x7f10bc101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:40.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:39.198+0000 7f10c3b87640 1 --2- 172.21.15.23:0/1354037424 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10bc06a6f0 0x7f10bc100b10 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:40.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:39.198+0000 7f10c3b87640 1 --2- 172.21.15.23:0/1354037424 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f10a80a29a0 0x7f10a80a4e60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:40.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:39.198+0000 7f10c3b87640 1 --2- 172.21.15.23:0/1354037424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10bc069d50 0x7f10bc1005d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:33:40.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:39.198+0000 7f10c3b87640 1 -- 172.21.15.23:0/1354037424 >> 172.21.15.23:0/1354037424 conn(0x7f10bc077640 msgr2=0x7f10bc0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:33:40.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:39.198+0000 7f10c3b87640 1 -- 172.21.15.23:0/1354037424 shutdown_connections 2024-03-28T05:33:40.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:33:39.198+0000 7f10c3b87640 1 -- 172.21.15.23:0/1354037424 wait complete. 2024-03-28T05:33:40.322 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:38 smithi023 ceph-mon[31954]: from='client.25927 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T05:33:40.323 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:38 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3825872786' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:33:40.323 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:38 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2104701992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:33:40.323 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:38 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3523010930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:33:40.323 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:38 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3981627898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:33:40.328 DEBUG:tasks.check_counter:Getting stats from a 2024-03-28T05:33:40.329 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-28T05:33:40.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:39 smithi023 ceph-mon[31954]: pgmap v313: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-03-28T05:33:40.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:39 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1354037424' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T05:33:40.581 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:33:42.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:41 smithi023 ceph-mon[31954]: pgmap v314: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-28T05:33:42.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:41 smithi098 ceph-mon[36087]: pgmap v314: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-28T05:33:42.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:41 smithi113 ceph-mon[37164]: pgmap v314: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-28T05:33:42.345 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T05:33:42.345 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T05:33:42.345 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 4, 2024-03-28T05:33:42.346 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 4, 2024-03-28T05:33:42.346 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 2992, 2024-03-28T05:33:42.346 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 448, 2024-03-28T05:33:42.346 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2024-03-28T05:33:42.346 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 0, 2024-03-28T05:33:42.346 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.001816718, 2024-03-28T05:33:42.346 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.000047979, 2024-03-28T05:33:42.346 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.000724074, 2024-03-28T05:33:42.346 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T05:33:42.346 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:33:42.346 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 3, 2024-03-28T05:33:42.347 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000849020, 2024-03-28T05:33:42.347 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000283006 2024-03-28T05:33:42.347 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:42.347 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:33:42.347 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T05:33:42.347 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T05:33:42.347 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T05:33:42.347 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:42.347 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-28T05:33:42.347 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T05:33:42.347 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:42.348 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T05:33:42.348 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 2, 2024-03-28T05:33:42.348 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 157, 2024-03-28T05:33:42.348 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 320, 2024-03-28T05:33:42.348 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 182752, 2024-03-28T05:33:42.348 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 5, 2024-03-28T05:33:42.348 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 3, 2024-03-28T05:33:42.348 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.012453972, 2024-03-28T05:33:42.348 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.008297770, 2024-03-28T05:33:42.348 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.002016542, 2024-03-28T05:33:42.348 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T05:33:42.348 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:33:42.348 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 154, 2024-03-28T05:33:42.349 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.012548786, 2024-03-28T05:33:42.349 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000081485 2024-03-28T05:33:42.349 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:42.349 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:33:42.349 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T05:33:42.349 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T05:33:42.349 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T05:33:42.349 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:42.349 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T05:33:42.349 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 182752 2024-03-28T05:33:42.349 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:42.349 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T05:33:42.350 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 216, 2024-03-28T05:33:42.350 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 205, 2024-03-28T05:33:42.350 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 1573104, 2024-03-28T05:33:42.350 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 122912, 2024-03-28T05:33:42.350 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 2, 2024-03-28T05:33:42.350 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 1, 2024-03-28T05:33:42.350 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.039521789, 2024-03-28T05:33:42.350 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.012419106, 2024-03-28T05:33:42.350 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.032772073, 2024-03-28T05:33:42.350 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.003970800, 2024-03-28T05:33:42.351 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:33:42.351 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 204, 2024-03-28T05:33:42.351 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.009331953, 2024-03-28T05:33:42.351 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000045744 2024-03-28T05:33:42.351 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:42.351 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:33:42.351 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T05:33:42.351 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T05:33:42.351 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T05:33:42.351 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:42.351 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 1573104, 2024-03-28T05:33:42.351 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 122912 2024-03-28T05:33:42.351 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:42.352 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-28T05:33:42.352 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 0, 2024-03-28T05:33:42.352 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-28T05:33:42.352 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:42.352 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-28T05:33:42.352 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-28T05:33:42.352 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-28T05:33:42.352 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T05:33:42.352 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-28T05:33:42.352 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T05:33:42.352 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-28T05:33:42.352 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T05:33:42.352 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T05:33:42.353 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T05:33:42.353 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T05:33:42.353 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T05:33:42.353 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-28T05:33:42.353 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T05:33:42.353 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T05:33:42.353 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-28T05:33:42.353 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-28T05:33:42.353 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-28T05:33:42.353 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-28T05:33:42.353 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T05:33:42.353 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T05:33:42.353 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T05:33:42.354 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T05:33:42.354 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T05:33:42.354 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-28T05:33:42.354 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-28T05:33:42.354 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-28T05:33:42.354 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T05:33:42.354 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T05:33:42.354 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-28T05:33:42.354 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-28T05:33:42.354 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-28T05:33:42.354 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-28T05:33:42.354 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T05:33:42.355 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-28T05:33:42.355 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T05:33:42.355 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-28T05:33:42.355 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 149057, 2024-03-28T05:33:42.355 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 32, 2024-03-28T05:33:42.355 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-28T05:33:42.355 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-28T05:33:42.355 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-28T05:33:42.355 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-28T05:33:42.355 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-28T05:33:42.355 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-28T05:33:42.355 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-28T05:33:42.355 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-28T05:33:42.356 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 0, 2024-03-28T05:33:42.356 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 0, 2024-03-28T05:33:42.356 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T05:33:42.356 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-28T05:33:42.356 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-28T05:33:42.356 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-28T05:33:42.356 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 928, 2024-03-28T05:33:42.356 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 2, 2024-03-28T05:33:42.356 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-28T05:33:42.356 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-28T05:33:42.356 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-28T05:33:42.356 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-28T05:33:42.356 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:42.357 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T05:33:42.357 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-28T05:33:42.357 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-28T05:33:42.357 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-28T05:33:42.357 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 218, 2024-03-28T05:33:42.357 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 1542673, 2024-03-28T05:33:42.357 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-28T05:33:42.357 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 218, 2024-03-28T05:33:42.357 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-28T05:33:42.357 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-28T05:33:42.357 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 218, 2024-03-28T05:33:42.357 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 1542673, 2024-03-28T05:33:42.357 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-28T05:33:42.358 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T05:33:42.358 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T05:33:42.358 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T05:33:42.358 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:42.358 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:42.358 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T05:33:42.992 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-03-28T05:33:42.992 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-03-28T05:33:42.992 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-03-28T05:33:42.992 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2024-03-28T05:33:42.992 DEBUG:tasks.check_counter:Getting stats from d 2024-03-28T05:33:42.993 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-28T05:33:43.240 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:33:44.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:43 smithi023 ceph-mon[31954]: pgmap v315: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T05:33:44.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:43 smithi098 ceph-mon[36087]: pgmap v315: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T05:33:44.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:43 smithi113 ceph-mon[37164]: pgmap v315: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T05:33:44.920 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T05:33:44.920 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T05:33:44.920 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 0, 2024-03-28T05:33:44.920 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 2, 2024-03-28T05:33:44.920 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 0, 2024-03-28T05:33:44.920 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 0, 2024-03-28T05:33:44.921 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2024-03-28T05:33:44.921 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 0, 2024-03-28T05:33:44.921 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.001189436, 2024-03-28T05:33:44.921 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.000000000, 2024-03-28T05:33:44.921 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.000260268, 2024-03-28T05:33:44.921 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T05:33:44.921 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:33:44.921 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T05:33:44.921 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T05:33:44.921 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T05:33:44.921 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:44.922 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:33:44.922 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T05:33:44.922 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T05:33:44.922 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T05:33:44.922 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:44.922 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T05:33:44.922 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T05:33:44.922 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:44.922 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T05:33:44.922 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 2, 2024-03-28T05:33:44.923 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 160, 2024-03-28T05:33:44.923 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 320, 2024-03-28T05:33:44.923 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 183280, 2024-03-28T05:33:44.923 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 5, 2024-03-28T05:33:44.923 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 3, 2024-03-28T05:33:44.923 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.011901715, 2024-03-28T05:33:44.923 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.007890484, 2024-03-28T05:33:44.923 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.001931057, 2024-03-28T05:33:44.923 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T05:33:44.923 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:33:44.923 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 157, 2024-03-28T05:33:44.924 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.009633656, 2024-03-28T05:33:44.924 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000061360 2024-03-28T05:33:44.924 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:44.924 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:33:44.924 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T05:33:44.924 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T05:33:44.924 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T05:33:44.924 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:44.924 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T05:33:44.924 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 183280 2024-03-28T05:33:44.924 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:44.925 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T05:33:44.925 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 226, 2024-03-28T05:33:44.925 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 213, 2024-03-28T05:33:44.925 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 1577792, 2024-03-28T05:33:44.925 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 126640, 2024-03-28T05:33:44.925 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 2, 2024-03-28T05:33:44.925 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 1, 2024-03-28T05:33:44.925 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.039290345, 2024-03-28T05:33:44.925 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.012615083, 2024-03-28T05:33:44.925 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.031876358, 2024-03-28T05:33:44.926 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.003942734, 2024-03-28T05:33:44.926 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:33:44.926 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 213, 2024-03-28T05:33:44.926 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.011477919, 2024-03-28T05:33:44.926 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000053886 2024-03-28T05:33:44.926 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:44.926 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:33:44.926 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T05:33:44.926 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T05:33:44.926 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T05:33:44.927 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:44.927 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 1577792, 2024-03-28T05:33:44.927 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 126640 2024-03-28T05:33:44.927 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:44.927 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-28T05:33:44.927 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 0, 2024-03-28T05:33:44.927 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-28T05:33:44.927 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:44.927 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-28T05:33:44.927 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-28T05:33:44.927 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-28T05:33:44.927 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T05:33:44.928 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-28T05:33:44.928 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T05:33:44.928 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-28T05:33:44.928 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T05:33:44.928 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T05:33:44.928 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T05:33:44.928 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T05:33:44.928 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T05:33:44.928 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-28T05:33:44.928 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T05:33:44.928 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T05:33:44.929 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-28T05:33:44.929 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-28T05:33:44.929 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-28T05:33:44.929 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-28T05:33:44.929 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T05:33:44.929 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T05:33:44.929 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T05:33:44.929 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T05:33:44.929 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T05:33:44.929 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-28T05:33:44.929 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-28T05:33:44.930 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-28T05:33:44.930 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T05:33:44.930 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T05:33:44.930 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-28T05:33:44.930 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-28T05:33:44.930 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-28T05:33:44.930 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-28T05:33:44.930 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T05:33:44.930 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-28T05:33:44.930 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T05:33:44.931 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-28T05:33:44.931 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 149057, 2024-03-28T05:33:44.931 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 32, 2024-03-28T05:33:44.931 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-28T05:33:44.931 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-28T05:33:44.931 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-28T05:33:44.931 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-28T05:33:44.931 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-28T05:33:44.931 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-28T05:33:44.931 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-28T05:33:44.931 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-28T05:33:44.931 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 0, 2024-03-28T05:33:44.932 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 0, 2024-03-28T05:33:44.932 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T05:33:44.932 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-28T05:33:44.932 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-28T05:33:44.932 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-28T05:33:44.932 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 928, 2024-03-28T05:33:44.932 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 2, 2024-03-28T05:33:44.932 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-28T05:33:44.932 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-28T05:33:44.932 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-28T05:33:44.932 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-28T05:33:44.933 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:44.933 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T05:33:44.933 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-28T05:33:44.933 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-28T05:33:44.933 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-28T05:33:44.933 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 224, 2024-03-28T05:33:44.933 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 1543517, 2024-03-28T05:33:44.933 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-28T05:33:44.933 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 224, 2024-03-28T05:33:44.933 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-28T05:33:44.933 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-28T05:33:44.934 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 224, 2024-03-28T05:33:44.934 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 1543517, 2024-03-28T05:33:44.934 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-28T05:33:44.934 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T05:33:44.934 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T05:33:44.934 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T05:33:44.934 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:44.934 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:44.934 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T05:33:45.675 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-28T05:33:45.675 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-28T05:33:45.676 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-28T05:33:45.676 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-03-28T05:33:45.676 DEBUG:tasks.check_counter:Getting stats from g 2024-03-28T05:33:45.676 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-28T05:33:45.924 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:33:46.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:45 smithi023 ceph-mon[31954]: pgmap v316: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T05:33:46.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:45 smithi098 ceph-mon[36087]: pgmap v316: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T05:33:46.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:45 smithi113 ceph-mon[37164]: pgmap v316: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T05:33:47.528 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T05:33:47.528 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T05:33:47.528 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 0, 2024-03-28T05:33:47.528 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 2, 2024-03-28T05:33:47.528 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 0, 2024-03-28T05:33:47.529 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 0, 2024-03-28T05:33:47.529 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2024-03-28T05:33:47.529 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 0, 2024-03-28T05:33:47.529 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.001338725, 2024-03-28T05:33:47.529 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.000000000, 2024-03-28T05:33:47.529 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.000292340, 2024-03-28T05:33:47.529 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T05:33:47.529 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:33:47.529 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T05:33:47.529 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T05:33:47.529 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T05:33:47.529 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:47.530 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:33:47.530 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T05:33:47.530 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T05:33:47.530 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T05:33:47.530 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:47.530 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T05:33:47.530 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T05:33:47.530 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:47.530 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T05:33:47.531 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 223, 2024-03-28T05:33:47.531 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 210, 2024-03-28T05:33:47.531 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 1576976, 2024-03-28T05:33:47.531 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 124768, 2024-03-28T05:33:47.531 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 2, 2024-03-28T05:33:47.531 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 1, 2024-03-28T05:33:47.531 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.039455957, 2024-03-28T05:33:47.531 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.012550365, 2024-03-28T05:33:47.531 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.032212362, 2024-03-28T05:33:47.532 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.003997143, 2024-03-28T05:33:47.532 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:33:47.532 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 210, 2024-03-28T05:33:47.532 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.008610585, 2024-03-28T05:33:47.532 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000041002 2024-03-28T05:33:47.532 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:47.532 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:33:47.532 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T05:33:47.532 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T05:33:47.532 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T05:33:47.532 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:47.532 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 1576976, 2024-03-28T05:33:47.533 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 124768 2024-03-28T05:33:47.533 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:47.533 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T05:33:47.533 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 2, 2024-03-28T05:33:47.533 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 158, 2024-03-28T05:33:47.533 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 320, 2024-03-28T05:33:47.533 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 182928, 2024-03-28T05:33:47.533 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 5, 2024-03-28T05:33:47.533 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 3, 2024-03-28T05:33:47.533 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.012191401, 2024-03-28T05:33:47.533 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.008338848, 2024-03-28T05:33:47.533 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.001679126, 2024-03-28T05:33:47.534 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T05:33:47.534 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:33:47.534 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 155, 2024-03-28T05:33:47.534 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.014171517, 2024-03-28T05:33:47.534 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000091429 2024-03-28T05:33:47.534 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:47.534 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:33:47.534 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T05:33:47.534 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T05:33:47.534 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T05:33:47.534 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:47.535 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T05:33:47.535 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 182928 2024-03-28T05:33:47.535 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:47.535 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-28T05:33:47.535 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 0, 2024-03-28T05:33:47.535 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-28T05:33:47.535 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:47.535 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-28T05:33:47.535 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-28T05:33:47.535 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-28T05:33:47.535 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T05:33:47.536 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-28T05:33:47.536 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T05:33:47.536 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-28T05:33:47.536 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T05:33:47.536 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T05:33:47.536 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T05:33:47.536 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T05:33:47.536 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T05:33:47.536 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-28T05:33:47.536 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T05:33:47.536 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T05:33:47.537 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-28T05:33:47.537 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-28T05:33:47.537 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-28T05:33:47.537 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-28T05:33:47.537 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T05:33:47.537 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T05:33:47.537 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T05:33:47.537 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T05:33:47.537 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T05:33:47.537 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-28T05:33:47.537 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-28T05:33:47.538 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-28T05:33:47.538 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T05:33:47.538 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T05:33:47.538 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-28T05:33:47.538 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-28T05:33:47.538 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-28T05:33:47.538 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-28T05:33:47.538 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T05:33:47.538 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-28T05:33:47.538 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T05:33:47.538 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-28T05:33:47.539 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 148995, 2024-03-28T05:33:47.539 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 31, 2024-03-28T05:33:47.539 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-28T05:33:47.539 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-28T05:33:47.539 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-28T05:33:47.539 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-28T05:33:47.539 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-28T05:33:47.539 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-28T05:33:47.539 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-28T05:33:47.539 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-28T05:33:47.539 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 0, 2024-03-28T05:33:47.539 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 0, 2024-03-28T05:33:47.540 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T05:33:47.540 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-28T05:33:47.540 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-28T05:33:47.540 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-28T05:33:47.540 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 928, 2024-03-28T05:33:47.540 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 2, 2024-03-28T05:33:47.540 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-28T05:33:47.540 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-28T05:33:47.540 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-28T05:33:47.540 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-28T05:33:47.540 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:47.540 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T05:33:47.541 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-28T05:33:47.541 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-28T05:33:47.541 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-28T05:33:47.541 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 221, 2024-03-28T05:33:47.541 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 1543127, 2024-03-28T05:33:47.541 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-28T05:33:47.541 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 221, 2024-03-28T05:33:47.541 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-28T05:33:47.541 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-28T05:33:47.541 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 221, 2024-03-28T05:33:47.541 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 1543127, 2024-03-28T05:33:47.541 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-28T05:33:47.542 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T05:33:47.542 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T05:33:47.542 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T05:33:47.542 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:47.542 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:47.542 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T05:33:48.149 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:47 smithi023 ceph-mon[31954]: pgmap v317: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T05:33:48.150 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-03-28T05:33:48.150 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-03-28T05:33:48.150 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-03-28T05:33:48.150 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2024-03-28T05:33:48.151 DEBUG:tasks.check_counter:Getting stats from j 2024-03-28T05:33:48.151 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-28T05:33:48.243 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:47 smithi113 ceph-mon[37164]: pgmap v317: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T05:33:48.367 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:47 smithi098 ceph-mon[36087]: pgmap v317: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T05:33:48.394 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T05:33:50.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:49 smithi023 ceph-mon[31954]: pgmap v318: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T05:33:50.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:49 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:33:50.064 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T05:33:50.065 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T05:33:50.065 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 0, 2024-03-28T05:33:50.065 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 2, 2024-03-28T05:33:50.065 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 0, 2024-03-28T05:33:50.065 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 0, 2024-03-28T05:33:50.065 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2024-03-28T05:33:50.065 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 0, 2024-03-28T05:33:50.065 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.001262519, 2024-03-28T05:33:50.065 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.000000000, 2024-03-28T05:33:50.065 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.000318213, 2024-03-28T05:33:50.065 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T05:33:50.065 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:33:50.065 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T05:33:50.065 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T05:33:50.065 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T05:33:50.066 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:50.066 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:33:50.066 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T05:33:50.066 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T05:33:50.066 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T05:33:50.066 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:50.066 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T05:33:50.066 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T05:33:50.066 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:50.066 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T05:33:50.066 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 228, 2024-03-28T05:33:50.066 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 215, 2024-03-28T05:33:50.066 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 1578336, 2024-03-28T05:33:50.066 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 127888, 2024-03-28T05:33:50.066 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 2, 2024-03-28T05:33:50.067 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 1, 2024-03-28T05:33:50.067 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.041731528, 2024-03-28T05:33:50.067 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.012604066, 2024-03-28T05:33:50.067 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.034845913, 2024-03-28T05:33:50.067 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.004164680, 2024-03-28T05:33:50.067 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:33:50.067 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 215, 2024-03-28T05:33:50.067 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.010320450, 2024-03-28T05:33:50.067 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000048002 2024-03-28T05:33:50.067 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:50.067 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:33:50.067 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T05:33:50.067 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T05:33:50.067 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T05:33:50.067 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:50.067 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 1578336, 2024-03-28T05:33:50.067 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 127888 2024-03-28T05:33:50.068 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:50.068 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T05:33:50.068 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 2, 2024-03-28T05:33:50.068 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 163, 2024-03-28T05:33:50.068 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 320, 2024-03-28T05:33:50.068 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 183808, 2024-03-28T05:33:50.068 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 5, 2024-03-28T05:33:50.068 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 3, 2024-03-28T05:33:50.068 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.012698744, 2024-03-28T05:33:50.068 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.008375011, 2024-03-28T05:33:50.068 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.001976827, 2024-03-28T05:33:50.068 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T05:33:50.068 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:33:50.068 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 160, 2024-03-28T05:33:50.068 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.013595410, 2024-03-28T05:33:50.068 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000084971 2024-03-28T05:33:50.068 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:50.068 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:33:50.069 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T05:33:50.069 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T05:33:50.069 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T05:33:50.069 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:50.069 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T05:33:50.069 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 183808 2024-03-28T05:33:50.069 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:50.069 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-28T05:33:50.069 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 0, 2024-03-28T05:33:50.069 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-28T05:33:50.069 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:50.069 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-28T05:33:50.069 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-28T05:33:50.069 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-28T05:33:50.069 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T05:33:50.069 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-28T05:33:50.069 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T05:33:50.070 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-28T05:33:50.070 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T05:33:50.070 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T05:33:50.070 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T05:33:50.070 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T05:33:50.070 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T05:33:50.070 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-28T05:33:50.070 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T05:33:50.070 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T05:33:50.070 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-28T05:33:50.070 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-28T05:33:50.070 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-28T05:33:50.070 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-28T05:33:50.070 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T05:33:50.070 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T05:33:50.070 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T05:33:50.070 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T05:33:50.071 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T05:33:50.071 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-28T05:33:50.071 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-28T05:33:50.071 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-28T05:33:50.071 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T05:33:50.071 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T05:33:50.071 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-28T05:33:50.071 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-28T05:33:50.071 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-28T05:33:50.071 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-28T05:33:50.071 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T05:33:50.071 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-28T05:33:50.071 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T05:33:50.071 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-28T05:33:50.071 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 149057, 2024-03-28T05:33:50.071 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 32, 2024-03-28T05:33:50.071 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-28T05:33:50.072 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-28T05:33:50.072 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-28T05:33:50.072 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-28T05:33:50.072 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-28T05:33:50.072 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-28T05:33:50.072 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-28T05:33:50.072 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-28T05:33:50.072 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 0, 2024-03-28T05:33:50.072 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 0, 2024-03-28T05:33:50.072 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T05:33:50.072 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-28T05:33:50.072 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-28T05:33:50.072 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-28T05:33:50.072 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 928, 2024-03-28T05:33:50.072 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 2, 2024-03-28T05:33:50.072 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-28T05:33:50.072 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-28T05:33:50.072 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-28T05:33:50.073 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-28T05:33:50.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T05:33:50.073 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T05:33:50.073 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-28T05:33:50.073 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-28T05:33:50.073 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-28T05:33:50.073 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 226, 2024-03-28T05:33:50.073 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 1543777, 2024-03-28T05:33:50.073 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-28T05:33:50.073 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 226, 2024-03-28T05:33:50.073 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-28T05:33:50.073 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-28T05:33:50.073 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 226, 2024-03-28T05:33:50.073 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 1543777, 2024-03-28T05:33:50.073 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-28T05:33:50.073 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T05:33:50.073 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T05:33:50.074 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T05:33:50.074 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:50.074 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T05:33:50.074 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T05:33:50.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:49 smithi113 ceph-mon[37164]: pgmap v318: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T05:33:50.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:49 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:33:50.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:49 smithi098 ceph-mon[36087]: pgmap v318: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T05:33:50.366 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:49 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T05:33:50.743 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-28T05:33:50.743 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-28T05:33:50.743 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-28T05:33:50.743 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-03-28T05:33:50.743 DEBUG:tasks.check_counter:Getting stats from b 2024-03-28T05:33:50.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-28T05:33:51.232 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T05:33:52.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:51 smithi098 ceph-mon[36087]: pgmap v319: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T05:33:52.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:51 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:33:52.117 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:51 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:33:52.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:51 smithi113 ceph-mon[37164]: pgmap v319: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T05:33:52.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:51 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:33:52.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:51 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:33:52.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:51 smithi023 ceph-mon[31954]: pgmap v319: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T05:33:52.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:51 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:33:52.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:51 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:33:53.108 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:52 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:52 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:52 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:52 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:52 smithi098 ceph-mon[36087]: Adjusting osd_memory_target on smithi113 to 174.1M 2024-03-28T05:33:53.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:52 smithi098 ceph-mon[36087]: Unable to set osd_memory_target on smithi113 to 182619852: error parsing value: Value '182619852' is below minimum 939524096 2024-03-28T05:33:53.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:52 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:33:53.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:52 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:33:53.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:52 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.109 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:52 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.110 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:52 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.110 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:52 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.110 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-28T05:33:53.110 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T05:33:53.111 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 1628, 2024-03-28T05:33:53.111 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 1630, 2024-03-28T05:33:53.111 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 113721405, 2024-03-28T05:33:53.111 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 517714, 2024-03-28T05:33:53.111 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 8, 2024-03-28T05:33:53.111 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 4, 2024-03-28T05:33:53.111 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.195820612, 2024-03-28T05:33:53.111 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.052712128, 2024-03-28T05:33:53.111 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.085379287, 2024-03-28T05:33:53.111 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.044199625, 2024-03-28T05:33:53.111 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:33:53.112 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1628, 2024-03-28T05:33:53.112 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.062968195, 2024-03-28T05:33:53.112 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000038678 2024-03-28T05:33:53.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.112 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:33:53.112 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.112 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.112 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.112 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T05:33:53.113 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T05:33:53.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.113 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T05:33:53.113 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 592, 2024-03-28T05:33:53.113 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 748, 2024-03-28T05:33:53.113 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 101641246, 2024-03-28T05:33:53.113 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 624022, 2024-03-28T05:33:53.113 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 9, 2024-03-28T05:33:53.113 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 7, 2024-03-28T05:33:53.113 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.117504700, 2024-03-28T05:33:53.114 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.019172137, 2024-03-28T05:33:53.114 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.077370834, 2024-03-28T05:33:53.114 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.015136086, 2024-03-28T05:33:53.114 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:33:53.114 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 745, 2024-03-28T05:33:53.114 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.039374180, 2024-03-28T05:33:53.114 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000052851 2024-03-28T05:33:53.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.114 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:33:53.114 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.115 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.115 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.115 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T05:33:53.115 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 436336 2024-03-28T05:33:53.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.115 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T05:33:53.115 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 942, 2024-03-28T05:33:53.115 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 928, 2024-03-28T05:33:53.115 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 115032723, 2024-03-28T05:33:53.115 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 350331, 2024-03-28T05:33:53.116 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 5, 2024-03-28T05:33:53.116 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 4, 2024-03-28T05:33:53.116 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.159743866, 2024-03-28T05:33:53.116 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.026983380, 2024-03-28T05:33:53.116 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.111784456, 2024-03-28T05:33:53.116 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.024384504, 2024-03-28T05:33:53.116 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:33:53.116 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 928, 2024-03-28T05:33:53.116 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.033207609, 2024-03-28T05:33:53.116 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000035784 2024-03-28T05:33:53.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.117 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:33:53.117 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.117 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.117 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.117 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1600416, 2024-03-28T05:33:53.117 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 128048 2024-03-28T05:33:53.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.117 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-03-28T05:33:53.117 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 1, 2024-03-28T05:33:53.117 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-03-28T05:33:53.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.118 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-MDSRank": { 2024-03-28T05:33:53.118 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-03-28T05:33:53.118 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-03-28T05:33:53.118 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 4713, 2024-03-28T05:33:53.118 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.537119251, 2024-03-28T05:33:53.118 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000113965 2024-03-28T05:33:53.118 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-28T05:33:53.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.118 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-PurgeQueue": { 2024-03-28T05:33:53.118 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-03-28T05:33:53.119 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-03-28T05:33:53.119 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.119 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.119 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.119 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-28T05:33:53.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.119 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": { 2024-03-28T05:33:53.119 INFO:teuthology.orchestra.run.smithi098.stdout: "request": 0, 2024-03-28T05:33:53.119 INFO:teuthology.orchestra.run.smithi098.stdout: "reply": 0, 2024-03-28T05:33:53.119 INFO:teuthology.orchestra.run.smithi098.stdout: "reply_latency": { 2024-03-28T05:33:53.119 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.120 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.120 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.122 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_reply": 0, 2024-03-28T05:33:53.122 INFO:teuthology.orchestra.run.smithi098.stdout: "forward": 0, 2024-03-28T05:33:53.122 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_complete": 0, 2024-03-28T05:33:53.122 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_keys": 0, 2024-03-28T05:33:53.122 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_commit": 0, 2024-03-28T05:33:53.122 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_split": 0, 2024-03-28T05:33:53.122 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_merge": 0, 2024-03-28T05:33:53.123 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes": 56407, 2024-03-28T05:33:53.123 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_top": 35480, 2024-03-28T05:33:53.123 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_bottom": 20927, 2024-03-28T05:33:53.123 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pin_tail": 0, 2024-03-28T05:33:53.123 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pinned": 5721, 2024-03-28T05:33:53.123 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_expired": 0, 2024-03-28T05:33:53.123 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_with_caps": 0, 2024-03-28T05:33:53.123 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": 0, 2024-03-28T05:33:53.123 INFO:teuthology.orchestra.run.smithi098.stdout: "subtrees": 2, 2024-03-28T05:33:53.123 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse": 0, 2024-03-28T05:33:53.123 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_hit": 0, 2024-03-28T05:33:53.123 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_forward": 0, 2024-03-28T05:33:53.124 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_discover": 0, 2024-03-28T05:33:53.124 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_dir_fetch": 0, 2024-03-28T05:33:53.124 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_remote_ino": 0, 2024-03-28T05:33:53.124 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_lock": 0, 2024-03-28T05:33:53.124 INFO:teuthology.orchestra.run.smithi098.stdout: "load_cent": 0, 2024-03-28T05:33:53.124 INFO:teuthology.orchestra.run.smithi098.stdout: "q": 0, 2024-03-28T05:33:53.124 INFO:teuthology.orchestra.run.smithi098.stdout: "exported": 0, 2024-03-28T05:33:53.124 INFO:teuthology.orchestra.run.smithi098.stdout: "exported_inodes": 0, 2024-03-28T05:33:53.124 INFO:teuthology.orchestra.run.smithi098.stdout: "imported": 0, 2024-03-28T05:33:53.124 INFO:teuthology.orchestra.run.smithi098.stdout: "imported_inodes": 0, 2024-03-28T05:33:53.124 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_dir_fetch": 0, 2024-03-28T05:33:53.125 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_backtrace_fetch": 0, 2024-03-28T05:33:53.125 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_peer_discover": 0, 2024-03-28T05:33:53.125 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rfiles": 4, 2024-03-28T05:33:53.125 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rbytes": 234, 2024-03-28T05:33:53.125 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rsnaps": 6, 2024-03-28T05:33:53.125 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T05:33:53.125 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_set_tag": 0, 2024-03-28T05:33:53.125 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T05:33:53.125 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_inotable_repaired": 0, 2024-03-28T05:33:53.125 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_inodes": 0, 2024-03-28T05:33:53.125 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T05:33:53.125 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T05:33:53.126 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_file_inodes": 0, 2024-03-28T05:33:53.126 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_inode_file_caps": 0, 2024-03-28T05:33:53.126 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T05:33:53.126 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_grant": 0, 2024-03-28T05:33:53.126 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T05:33:53.126 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T05:33:53.126 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T05:33:53.126 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps": 0, 2024-03-28T05:33:53.126 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps_dirty": 0, 2024-03-28T05:33:53.126 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_cap_release": 0, 2024-03-28T05:33:53.126 INFO:teuthology.orchestra.run.smithi098.stdout: "process_request_cap_release": 0 2024-03-28T05:33:53.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.127 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache": { 2024-03-28T05:33:53.127 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update": 0, 2024-03-28T05:33:53.127 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update_receipt": 0, 2024-03-28T05:33:53.127 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_try_discover": 0, 2024-03-28T05:33:53.127 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_send_discover": 0, 2024-03-28T05:33:53.127 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_handle_discover": 0, 2024-03-28T05:33:53.127 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays": 0, 2024-03-28T05:33:53.127 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_delayed": 0, 2024-03-28T05:33:53.127 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_enqueuing": 0, 2024-03-28T05:33:53.127 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_created": 0, 2024-03-28T05:33:53.127 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_enqueued": 0, 2024-03-28T05:33:53.128 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_reintegrated": 0, 2024-03-28T05:33:53.128 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_migrated": 0, 2024-03-28T05:33:53.128 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_processing": 0, 2024-03-28T05:33:53.128 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_enqueued": 0, 2024-03-28T05:33:53.128 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_prioritized": 0, 2024-03-28T05:33:53.128 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_started": 0, 2024-03-28T05:33:53.128 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_completed": 0, 2024-03-28T05:33:53.128 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T05:33:53.128 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_exportdir": 0, 2024-03-28T05:33:53.128 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_flush": 0, 2024-03-28T05:33:53.128 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragmentdir": 0, 2024-03-28T05:33:53.128 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragstats": 0, 2024-03-28T05:33:53.129 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_inodestats": 0 2024-03-28T05:33:53.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.129 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log": { 2024-03-28T05:33:53.129 INFO:teuthology.orchestra.run.smithi098.stdout: "evlrg": 0, 2024-03-28T05:33:53.129 INFO:teuthology.orchestra.run.smithi098.stdout: "evadd": 0, 2024-03-28T05:33:53.129 INFO:teuthology.orchestra.run.smithi098.stdout: "evex": 0, 2024-03-28T05:33:53.129 INFO:teuthology.orchestra.run.smithi098.stdout: "evtrm": 0, 2024-03-28T05:33:53.129 INFO:teuthology.orchestra.run.smithi098.stdout: "ev": 97194, 2024-03-28T05:33:53.129 INFO:teuthology.orchestra.run.smithi098.stdout: "evexg": 0, 2024-03-28T05:33:53.129 INFO:teuthology.orchestra.run.smithi098.stdout: "evexd": 0, 2024-03-28T05:33:53.129 INFO:teuthology.orchestra.run.smithi098.stdout: "segadd": 0, 2024-03-28T05:33:53.129 INFO:teuthology.orchestra.run.smithi098.stdout: "segex": 0, 2024-03-28T05:33:53.130 INFO:teuthology.orchestra.run.smithi098.stdout: "segtrm": 0, 2024-03-28T05:33:53.130 INFO:teuthology.orchestra.run.smithi098.stdout: "seg": 8, 2024-03-28T05:33:53.130 INFO:teuthology.orchestra.run.smithi098.stdout: "segmjr": 36, 2024-03-28T05:33:53.130 INFO:teuthology.orchestra.run.smithi098.stdout: "segexg": 0, 2024-03-28T05:33:53.130 INFO:teuthology.orchestra.run.smithi098.stdout: "segexd": 0, 2024-03-28T05:33:53.130 INFO:teuthology.orchestra.run.smithi098.stdout: "expos": 316852387, 2024-03-28T05:33:53.130 INFO:teuthology.orchestra.run.smithi098.stdout: "wrpos": 332174728, 2024-03-28T05:33:53.130 INFO:teuthology.orchestra.run.smithi098.stdout: "rdpos": 332174592, 2024-03-28T05:33:53.130 INFO:teuthology.orchestra.run.smithi098.stdout: "jlat": { 2024-03-28T05:33:53.130 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.130 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.131 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.131 INFO:teuthology.orchestra.run.smithi098.stdout: "replayed": 97194 2024-03-28T05:33:53.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.131 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_mem": { 2024-03-28T05:33:53.131 INFO:teuthology.orchestra.run.smithi098.stdout: "ino": 16758, 2024-03-28T05:33:53.131 INFO:teuthology.orchestra.run.smithi098.stdout: "ino+": 34709, 2024-03-28T05:33:53.131 INFO:teuthology.orchestra.run.smithi098.stdout: "ino-": 17951, 2024-03-28T05:33:53.131 INFO:teuthology.orchestra.run.smithi098.stdout: "dir": 498, 2024-03-28T05:33:53.131 INFO:teuthology.orchestra.run.smithi098.stdout: "dir+": 559, 2024-03-28T05:33:53.131 INFO:teuthology.orchestra.run.smithi098.stdout: "dir-": 61, 2024-03-28T05:33:53.131 INFO:teuthology.orchestra.run.smithi098.stdout: "dn": 56407, 2024-03-28T05:33:53.132 INFO:teuthology.orchestra.run.smithi098.stdout: "dn+": 56429, 2024-03-28T05:33:53.132 INFO:teuthology.orchestra.run.smithi098.stdout: "dn-": 22, 2024-03-28T05:33:53.132 INFO:teuthology.orchestra.run.smithi098.stdout: "cap": 0, 2024-03-28T05:33:53.132 INFO:teuthology.orchestra.run.smithi098.stdout: "cap+": 0, 2024-03-28T05:33:53.132 INFO:teuthology.orchestra.run.smithi098.stdout: "cap-": 0, 2024-03-28T05:33:53.132 INFO:teuthology.orchestra.run.smithi098.stdout: "rss": 146880, 2024-03-28T05:33:53.132 INFO:teuthology.orchestra.run.smithi098.stdout: "heap": 215420 2024-03-28T05:33:53.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.132 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_server": { 2024-03-28T05:33:53.132 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_client_request": 0, 2024-03-28T05:33:53.132 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_server_request": 0, 2024-03-28T05:33:53.132 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_request": 0, 2024-03-28T05:33:53.132 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_session": 0, 2024-03-28T05:33:53.133 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_peer_request": 0, 2024-03-28T05:33:53.133 INFO:teuthology.orchestra.run.smithi098.stdout: "req_create_latency": { 2024-03-28T05:33:53.133 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.133 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.133 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.133 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getattr_latency": { 2024-03-28T05:33:53.133 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.133 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.133 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.133 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getfilelock_latency": { 2024-03-28T05:33:53.133 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.134 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.134 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.134 INFO:teuthology.orchestra.run.smithi098.stdout: "req_link_latency": { 2024-03-28T05:33:53.134 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.134 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.134 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.134 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookup_latency": { 2024-03-28T05:33:53.134 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.134 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.134 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.135 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookuphash_latency": { 2024-03-28T05:33:53.135 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.135 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.135 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.135 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupino_latency": { 2024-03-28T05:33:53.135 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.135 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.135 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.135 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupname_latency": { 2024-03-28T05:33:53.135 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.136 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.136 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.136 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupparent_latency": { 2024-03-28T05:33:53.136 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.136 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.136 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.136 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupsnap_latency": { 2024-03-28T05:33:53.136 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.136 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.136 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.137 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lssnap_latency": { 2024-03-28T05:33:53.137 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.137 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.137 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.137 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mkdir_latency": { 2024-03-28T05:33:53.137 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.137 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.137 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.137 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mknod_latency": { 2024-03-28T05:33:53.137 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.138 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.138 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.138 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mksnap_latency": { 2024-03-28T05:33:53.138 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.138 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.138 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.138 INFO:teuthology.orchestra.run.smithi098.stdout: "req_open_latency": { 2024-03-28T05:33:53.138 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.138 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.138 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.139 INFO:teuthology.orchestra.run.smithi098.stdout: "req_readdir_latency": { 2024-03-28T05:33:53.139 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.139 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.139 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.139 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rename_latency": { 2024-03-28T05:33:53.139 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.139 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.139 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.139 INFO:teuthology.orchestra.run.smithi098.stdout: "req_renamesnap_latency": { 2024-03-28T05:33:53.140 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.140 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.140 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.140 INFO:teuthology.orchestra.run.smithi098.stdout: "req_snapdiff_latency": { 2024-03-28T05:33:53.140 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.140 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.140 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.140 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmdir_latency": { 2024-03-28T05:33:53.140 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.140 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.141 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.141 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmsnap_latency": { 2024-03-28T05:33:53.141 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.141 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.141 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.141 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmxattr_latency": { 2024-03-28T05:33:53.141 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.141 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.141 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.142 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setattr_latency": { 2024-03-28T05:33:53.142 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.142 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.142 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.142 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setdirlayout_latency": { 2024-03-28T05:33:53.142 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.142 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.142 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.143 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setfilelock_latency": { 2024-03-28T05:33:53.143 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.143 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.143 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.143 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setlayout_latency": { 2024-03-28T05:33:53.143 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.143 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.143 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.143 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setxattr_latency": { 2024-03-28T05:33:53.144 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.144 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.144 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.144 INFO:teuthology.orchestra.run.smithi098.stdout: "req_symlink_latency": { 2024-03-28T05:33:53.144 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.144 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.144 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.144 INFO:teuthology.orchestra.run.smithi098.stdout: "req_unlink_latency": { 2024-03-28T05:33:53.144 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.144 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.145 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.145 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_revoke_eviction": 0, 2024-03-28T05:33:53.145 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_acquisition_throttle": 0, 2024-03-28T05:33:53.145 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getvxattr_latency": { 2024-03-28T05:33:53.145 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.145 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.145 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.145 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-28T05:33:53.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.145 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_sessions": { 2024-03-28T05:33:53.145 INFO:teuthology.orchestra.run.smithi098.stdout: "session_count": 2, 2024-03-28T05:33:53.146 INFO:teuthology.orchestra.run.smithi098.stdout: "session_add": 6, 2024-03-28T05:33:53.146 INFO:teuthology.orchestra.run.smithi098.stdout: "session_remove": 4, 2024-03-28T05:33:53.146 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_open": 4, 2024-03-28T05:33:53.146 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_stale": 0, 2024-03-28T05:33:53.146 INFO:teuthology.orchestra.run.smithi098.stdout: "total_load": 0, 2024-03-28T05:33:53.146 INFO:teuthology.orchestra.run.smithi098.stdout: "average_load": 0, 2024-03-28T05:33:53.146 INFO:teuthology.orchestra.run.smithi098.stdout: "avg_session_uptime": 3458, 2024-03-28T05:33:53.146 INFO:teuthology.orchestra.run.smithi098.stdout: "mdthresh_evicted": 0 2024-03-28T05:33:53.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.146 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-03-28T05:33:53.146 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-03-28T05:33:53.147 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-03-28T05:33:53.147 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T05:33:53.147 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-03-28T05:33:53.147 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T05:33:53.147 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-03-28T05:33:53.147 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T05:33:53.147 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T05:33:53.147 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T05:33:53.147 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T05:33:53.147 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T05:33:53.147 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-03-28T05:33:53.148 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T05:33:53.148 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T05:33:53.148 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-03-28T05:33:53.148 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-03-28T05:33:53.148 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-03-28T05:33:53.148 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-03-28T05:33:53.148 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T05:33:53.148 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T05:33:53.148 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T05:33:53.148 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T05:33:53.148 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T05:33:53.148 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-03-28T05:33:53.149 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-03-28T05:33:53.149 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-03-28T05:33:53.149 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T05:33:53.149 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T05:33:53.149 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-03-28T05:33:53.149 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-03-28T05:33:53.149 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-03-28T05:33:53.149 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-03-28T05:33:53.149 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T05:33:53.149 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-03-28T05:33:53.149 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T05:33:53.149 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-03-28T05:33:53.150 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 869036, 2024-03-28T05:33:53.150 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 17095, 2024-03-28T05:33:53.150 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-03-28T05:33:53.150 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-03-28T05:33:53.150 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-03-28T05:33:53.150 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-03-28T05:33:53.150 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-03-28T05:33:53.150 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-03-28T05:33:53.150 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-03-28T05:33:53.150 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-03-28T05:33:53.150 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 9608, 2024-03-28T05:33:53.151 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 184, 2024-03-28T05:33:53.151 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T05:33:53.151 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-03-28T05:33:53.151 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-03-28T05:33:53.151 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-03-28T05:33:53.151 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 79192346, 2024-03-28T05:33:53.151 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 841666, 2024-03-28T05:33:53.151 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-03-28T05:33:53.151 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-03-28T05:33:53.151 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-03-28T05:33:53.151 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-03-28T05:33:53.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.152 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter": { 2024-03-28T05:33:53.152 INFO:teuthology.orchestra.run.smithi098.stdout: "op_active": 0, 2024-03-28T05:33:53.152 INFO:teuthology.orchestra.run.smithi098.stdout: "op_laggy": 0, 2024-03-28T05:33:53.152 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send": 2917, 2024-03-28T05:33:53.152 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send_bytes": 26, 2024-03-28T05:33:53.152 INFO:teuthology.orchestra.run.smithi098.stdout: "op_resend": 4, 2024-03-28T05:33:53.152 INFO:teuthology.orchestra.run.smithi098.stdout: "op_reply": 2917, 2024-03-28T05:33:53.152 INFO:teuthology.orchestra.run.smithi098.stdout: "op_latency": { 2024-03-28T05:33:53.152 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 2917, 2024-03-28T05:33:53.152 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 5.990920191, 2024-03-28T05:33:53.152 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.002053795 2024-03-28T05:33:53.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.152 INFO:teuthology.orchestra.run.smithi098.stdout: "op_inflight": 0, 2024-03-28T05:33:53.152 INFO:teuthology.orchestra.run.smithi098.stdout: "oplen_avg": { 2024-03-28T05:33:53.153 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 2917, 2024-03-28T05:33:53.153 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 2918 2024-03-28T05:33:53.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.153 INFO:teuthology.orchestra.run.smithi098.stdout: "op": 2917, 2024-03-28T05:33:53.153 INFO:teuthology.orchestra.run.smithi098.stdout: "op_r": 2917, 2024-03-28T05:33:53.153 INFO:teuthology.orchestra.run.smithi098.stdout: "op_w": 0, 2024-03-28T05:33:53.153 INFO:teuthology.orchestra.run.smithi098.stdout: "op_rmw": 0, 2024-03-28T05:33:53.153 INFO:teuthology.orchestra.run.smithi098.stdout: "op_pg": 0, 2024-03-28T05:33:53.153 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_stat": 1708, 2024-03-28T05:33:53.153 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_create": 0, 2024-03-28T05:33:53.153 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_read": 1206, 2024-03-28T05:33:53.153 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_write": 0, 2024-03-28T05:33:53.154 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writefull": 0, 2024-03-28T05:33:53.154 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writesame": 0, 2024-03-28T05:33:53.154 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_append": 0, 2024-03-28T05:33:53.154 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_zero": 0, 2024-03-28T05:33:53.154 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_truncate": 0, 2024-03-28T05:33:53.154 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_delete": 0, 2024-03-28T05:33:53.154 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_mapext": 0, 2024-03-28T05:33:53.154 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_sparse_read": 0, 2024-03-28T05:33:53.154 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_clonerange": 0, 2024-03-28T05:33:53.154 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_getxattr": 2, 2024-03-28T05:33:53.154 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_setxattr": 0, 2024-03-28T05:33:53.154 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_cmpxattr": 0, 2024-03-28T05:33:53.154 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_rmxattr": 0, 2024-03-28T05:33:53.155 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_resetxattrs": 0, 2024-03-28T05:33:53.155 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_call": 0, 2024-03-28T05:33:53.155 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_watch": 0, 2024-03-28T05:33:53.155 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_notify": 0, 2024-03-28T05:33:53.155 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T05:33:53.155 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls": 0, 2024-03-28T05:33:53.155 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls_filter": 0, 2024-03-28T05:33:53.155 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_other": 0, 2024-03-28T05:33:53.155 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_active": 0, 2024-03-28T05:33:53.155 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_send": 0, 2024-03-28T05:33:53.155 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_resend": 0, 2024-03-28T05:33:53.155 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_ping": 0, 2024-03-28T05:33:53.155 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_active": 0, 2024-03-28T05:33:53.156 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_send": 0, 2024-03-28T05:33:53.156 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_resend": 0, 2024-03-28T05:33:53.156 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_active": 0, 2024-03-28T05:33:53.156 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_send": 0, 2024-03-28T05:33:53.156 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_resend": 0, 2024-03-28T05:33:53.156 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_active": 0, 2024-03-28T05:33:53.156 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_send": 0, 2024-03-28T05:33:53.156 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_resend": 0, 2024-03-28T05:33:53.156 INFO:teuthology.orchestra.run.smithi098.stdout: "command_active": 0, 2024-03-28T05:33:53.156 INFO:teuthology.orchestra.run.smithi098.stdout: "command_send": 0, 2024-03-28T05:33:53.156 INFO:teuthology.orchestra.run.smithi098.stdout: "command_resend": 0, 2024-03-28T05:33:53.156 INFO:teuthology.orchestra.run.smithi098.stdout: "map_epoch": 102, 2024-03-28T05:33:53.156 INFO:teuthology.orchestra.run.smithi098.stdout: "map_full": 0, 2024-03-28T05:33:53.156 INFO:teuthology.orchestra.run.smithi098.stdout: "map_inc": 13, 2024-03-28T05:33:53.157 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_sessions": 11, 2024-03-28T05:33:53.157 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_open": 11, 2024-03-28T05:33:53.157 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_close": 0, 2024-03-28T05:33:53.157 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_laggy": 0, 2024-03-28T05:33:53.157 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_wr": 0, 2024-03-28T05:33:53.157 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_rd": 2, 2024-03-28T05:33:53.157 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_del": 0 2024-03-28T05:33:53.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.157 INFO:teuthology.orchestra.run.smithi098.stdout: "oft": { 2024-03-28T05:33:53.157 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_objs": 0, 2024-03-28T05:33:53.157 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_kv_pairs": 0, 2024-03-28T05:33:53.157 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_updates": 0, 2024-03-28T05:33:53.158 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_removes": 0 2024-03-28T05:33:53.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.158 INFO:teuthology.orchestra.run.smithi098.stdout: "osd-slow-ops": { 2024-03-28T05:33:53.158 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_ops_count": 0 2024-03-28T05:33:53.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.158 INFO:teuthology.orchestra.run.smithi098.stdout: "purge_queue": { 2024-03-28T05:33:53.158 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops": 0, 2024-03-28T05:33:53.158 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T05:33:53.158 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing": 0, 2024-03-28T05:33:53.158 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_high_water": 0, 2024-03-28T05:33:53.158 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed_ops": 0, 2024-03-28T05:33:53.158 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed": 0, 2024-03-28T05:33:53.158 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_item_in_journal": 0 2024-03-28T05:33:53.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.159 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T05:33:53.159 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-28T05:33:53.159 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-28T05:33:53.159 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-28T05:33:53.159 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 3158, 2024-03-28T05:33:53.159 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 330095942, 2024-03-28T05:33:53.159 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-28T05:33:53.159 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 3158, 2024-03-28T05:33:53.159 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-28T05:33:53.159 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-28T05:33:53.159 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 3158, 2024-03-28T05:33:53.159 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 330095942, 2024-03-28T05:33:53.159 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-28T05:33:53.160 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.160 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.160 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.160 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-28T05:33:53.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.160 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_bytes": { 2024-03-28T05:33:53.160 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-28T05:33:53.160 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-28T05:33:53.160 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-28T05:33:53.160 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-03-28T05:33:53.160 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-03-28T05:33:53.160 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-28T05:33:53.160 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-03-28T05:33:53.161 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 2917, 2024-03-28T05:33:53.161 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 327989982, 2024-03-28T05:33:53.161 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 2917, 2024-03-28T05:33:53.161 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 327989982, 2024-03-28T05:33:53.161 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-28T05:33:53.161 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.162 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.162 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.162 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-28T05:33:53.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.162 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_ops": { 2024-03-28T05:33:53.162 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-28T05:33:53.162 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 1024, 2024-03-28T05:33:53.162 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-28T05:33:53.162 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-03-28T05:33:53.162 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-03-28T05:33:53.162 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-28T05:33:53.163 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-03-28T05:33:53.163 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 2917, 2024-03-28T05:33:53.163 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 2917, 2024-03-28T05:33:53.163 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 2917, 2024-03-28T05:33:53.163 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 2917, 2024-03-28T05:33:53.163 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-28T05:33:53.163 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.163 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.163 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.163 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-28T05:33:53.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.163 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle": { 2024-03-28T05:33:53.163 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-28T05:33:53.163 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-03-28T05:33:53.164 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-28T05:33:53.164 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-03-28T05:33:53.164 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-03-28T05:33:53.164 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-28T05:33:53.164 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-03-28T05:33:53.164 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-28T05:33:53.164 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-28T05:33:53.164 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-03-28T05:33:53.164 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-03-28T05:33:53.164 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-28T05:33:53.164 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.164 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.164 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.165 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-28T05:33:53.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:53.165 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle-0x563908d9c6e0": { 2024-03-28T05:33:53.165 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-28T05:33:53.165 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-03-28T05:33:53.165 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-28T05:33:53.165 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-03-28T05:33:53.165 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-03-28T05:33:53.165 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-28T05:33:53.165 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-03-28T05:33:53.165 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-28T05:33:53.165 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-28T05:33:53.165 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-03-28T05:33:53.165 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-03-28T05:33:53.166 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-28T05:33:53.166 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:53.166 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:53.166 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:53.166 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-28T05:33:53.166 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-28T05:33:53.166 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-28T05:33:53.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:52 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:52 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:52 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:52 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:52 smithi113 ceph-mon[37164]: Adjusting osd_memory_target on smithi113 to 174.1M 2024-03-28T05:33:53.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:52 smithi113 ceph-mon[37164]: Unable to set osd_memory_target on smithi113 to 182619852: error parsing value: Value '182619852' is below minimum 939524096 2024-03-28T05:33:53.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:52 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:33:53.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:52 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:33:53.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:52 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:52 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.245 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:52 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.245 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:52 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:52 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:52 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:52 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:52 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:52 smithi023 ceph-mon[31954]: Adjusting osd_memory_target on smithi113 to 174.1M 2024-03-28T05:33:53.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:52 smithi023 ceph-mon[31954]: Unable to set osd_memory_target on smithi113 to 182619852: error parsing value: Value '182619852' is below minimum 939524096 2024-03-28T05:33:53.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:52 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:33:53.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:52 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:33:53.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:52 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:52 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:52 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.283 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:52 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:53.925 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-28T05:33:53.926 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=0 2024-03-28T05:33:53.926 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T05:33:53.926 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=0 2024-03-28T05:33:53.926 DEBUG:tasks.check_counter:Getting stats from e 2024-03-28T05:33:53.926 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-28T05:33:54.193 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T05:33:54.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:53 smithi113 ceph-mon[37164]: pgmap v320: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T05:33:54.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:53 smithi113 ceph-mon[37164]: Adjusting osd_memory_target on smithi023 to 176.2M 2024-03-28T05:33:54.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:53 smithi113 ceph-mon[37164]: Unable to set osd_memory_target on smithi023 to 184790323: error parsing value: Value '184790323' is below minimum 939524096 2024-03-28T05:33:54.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:53 smithi023 ceph-mon[31954]: pgmap v320: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T05:33:54.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:53 smithi023 ceph-mon[31954]: Adjusting osd_memory_target on smithi023 to 176.2M 2024-03-28T05:33:54.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:53 smithi023 ceph-mon[31954]: Unable to set osd_memory_target on smithi023 to 184790323: error parsing value: Value '184790323' is below minimum 939524096 2024-03-28T05:33:54.318 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:53 smithi098 ceph-mon[36087]: pgmap v320: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-28T05:33:54.318 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:53 smithi098 ceph-mon[36087]: Adjusting osd_memory_target on smithi023 to 176.2M 2024-03-28T05:33:54.318 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:53 smithi098 ceph-mon[36087]: Unable to set osd_memory_target on smithi023 to 184790323: error parsing value: Value '184790323' is below minimum 939524096 2024-03-28T05:33:55.367 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:55 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:33:55.367 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:55 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:33:55.367 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:55 smithi098 ceph-mon[36087]: pgmap v321: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:33:55.367 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:55 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:55.367 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:55 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:55.367 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:55 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:55.367 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:55 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:55.367 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:55 smithi098 ceph-mon[36087]: Adjusting osd_memory_target on smithi098 to 174.1M 2024-03-28T05:33:55.367 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:55 smithi098 ceph-mon[36087]: Unable to set osd_memory_target on smithi098 to 182633472: error parsing value: Value '182633472' is below minimum 939524096 2024-03-28T05:33:55.368 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:55 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:33:55.368 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:55 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:33:55.368 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:55 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:33:55.494 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:55 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:33:55.494 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:55 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:33:55.494 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:55 smithi113 ceph-mon[37164]: pgmap v321: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:33:55.494 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:55 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:55.494 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:55 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:55.494 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:55 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:55.494 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:55 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:55.494 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:55 smithi113 ceph-mon[37164]: Adjusting osd_memory_target on smithi098 to 174.1M 2024-03-28T05:33:55.495 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:55 smithi113 ceph-mon[37164]: Unable to set osd_memory_target on smithi098 to 182633472: error parsing value: Value '182633472' is below minimum 939524096 2024-03-28T05:33:55.495 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:55 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:33:55.495 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:55 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:33:55.495 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:55 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:33:55.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:55 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:33:55.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:55 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:33:55.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:55 smithi023 ceph-mon[31954]: pgmap v321: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:33:55.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:55 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:55.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:55 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:55.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:55 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:55.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:55 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T05:33:55.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:55 smithi023 ceph-mon[31954]: Adjusting osd_memory_target on smithi098 to 174.1M 2024-03-28T05:33:55.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:55 smithi023 ceph-mon[31954]: Unable to set osd_memory_target on smithi098 to 182633472: error parsing value: Value '182633472' is below minimum 939524096 2024-03-28T05:33:55.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:55 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T05:33:55.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:55 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T05:33:55.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:55 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' 2024-03-28T05:33:56.059 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-28T05:33:56.060 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T05:33:56.060 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 0, 2024-03-28T05:33:56.060 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2, 2024-03-28T05:33:56.060 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 0, 2024-03-28T05:33:56.060 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 0, 2024-03-28T05:33:56.060 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-03-28T05:33:56.060 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-03-28T05:33:56.060 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.001076342, 2024-03-28T05:33:56.060 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000000000, 2024-03-28T05:33:56.061 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000252274, 2024-03-28T05:33:56.061 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T05:33:56.061 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:33:56.061 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:56.061 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:56.061 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:56.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:56.061 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:33:56.061 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:56.061 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:56.061 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T05:33:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T05:33:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T05:33:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 6, 2024-03-28T05:33:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 165, 2024-03-28T05:33:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 3312, 2024-03-28T05:33:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 184256, 2024-03-28T05:33:56.062 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 5, 2024-03-28T05:33:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 3, 2024-03-28T05:33:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.011321738, 2024-03-28T05:33:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.007214570, 2024-03-28T05:33:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.002010562, 2024-03-28T05:33:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T05:33:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:33:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 163, 2024-03-28T05:33:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.013005722, 2024-03-28T05:33:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000079789 2024-03-28T05:33:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:33:56.063 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-28T05:33:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 184256 2024-03-28T05:33:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T05:33:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 224, 2024-03-28T05:33:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 213, 2024-03-28T05:33:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 1575344, 2024-03-28T05:33:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 127440, 2024-03-28T05:33:56.064 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-03-28T05:33:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-03-28T05:33:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.043155154, 2024-03-28T05:33:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.012772370, 2024-03-28T05:33:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.033921818, 2024-03-28T05:33:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.005178924, 2024-03-28T05:33:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:33:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 212, 2024-03-28T05:33:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.011605472, 2024-03-28T05:33:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000054742 2024-03-28T05:33:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:56.065 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:33:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1575344, 2024-03-28T05:33:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 127440 2024-03-28T05:33:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-03-28T05:33:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-03-28T05:33:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-03-28T05:33:56.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:56.067 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-03-28T05:33:56.067 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-03-28T05:33:56.067 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-03-28T05:33:56.067 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T05:33:56.067 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-03-28T05:33:56.067 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T05:33:56.067 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-03-28T05:33:56.067 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T05:33:56.067 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T05:33:56.067 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T05:33:56.067 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T05:33:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T05:33:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-03-28T05:33:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T05:33:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T05:33:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-03-28T05:33:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-03-28T05:33:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-03-28T05:33:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-03-28T05:33:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T05:33:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T05:33:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T05:33:56.068 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T05:33:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T05:33:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-03-28T05:33:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-03-28T05:33:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-03-28T05:33:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T05:33:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T05:33:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-03-28T05:33:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-03-28T05:33:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-03-28T05:33:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-03-28T05:33:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T05:33:56.069 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-03-28T05:33:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T05:33:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-03-28T05:33:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 149017, 2024-03-28T05:33:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 31, 2024-03-28T05:33:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-03-28T05:33:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-03-28T05:33:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-03-28T05:33:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-03-28T05:33:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-03-28T05:33:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-03-28T05:33:56.070 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-03-28T05:33:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-03-28T05:33:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-03-28T05:33:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-03-28T05:33:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T05:33:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-03-28T05:33:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-03-28T05:33:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-03-28T05:33:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-03-28T05:33:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-03-28T05:33:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-03-28T05:33:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-03-28T05:33:56.071 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-03-28T05:33:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-03-28T05:33:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T05:33:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-28T05:33:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-28T05:33:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-28T05:33:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 226, 2024-03-28T05:33:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 1543777, 2024-03-28T05:33:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-28T05:33:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 226, 2024-03-28T05:33:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-28T05:33:56.072 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-28T05:33:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 226, 2024-03-28T05:33:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1543777, 2024-03-28T05:33:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-28T05:33:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-28T05:33:56.073 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-28T05:33:56.073 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-28T05:33:56.879 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.e 2024-03-28T05:33:56.879 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.e 2024-03-28T05:33:56.879 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.e 2024-03-28T05:33:56.879 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.e 2024-03-28T05:33:56.879 DEBUG:tasks.check_counter:Getting stats from h 2024-03-28T05:33:56.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-28T05:33:57.121 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T05:33:57.494 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:57 smithi113 ceph-mon[37164]: pgmap v322: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:33:57.518 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:57 smithi098 ceph-mon[36087]: pgmap v322: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:33:57.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:57 smithi023 ceph-mon[31954]: pgmap v322: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:33:58.884 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-28T05:33:58.885 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T05:33:58.886 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 227, 2024-03-28T05:33:58.886 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 214, 2024-03-28T05:33:58.886 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 1578064, 2024-03-28T05:33:58.886 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 127264, 2024-03-28T05:33:58.886 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-03-28T05:33:58.886 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-03-28T05:33:58.886 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.042948174, 2024-03-28T05:33:58.886 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.013034265, 2024-03-28T05:33:58.886 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.031847319, 2024-03-28T05:33:58.887 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.005195053, 2024-03-28T05:33:58.887 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:33:58.887 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 214, 2024-03-28T05:33:58.887 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.012021568, 2024-03-28T05:33:58.887 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000056175 2024-03-28T05:33:58.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:58.887 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:33:58.887 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:58.887 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:58.887 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:58.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:58.888 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1578064, 2024-03-28T05:33:58.888 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 127264 2024-03-28T05:33:58.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:58.888 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T05:33:58.888 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 2, 2024-03-28T05:33:58.888 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 161, 2024-03-28T05:33:58.888 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 320, 2024-03-28T05:33:58.888 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 183456, 2024-03-28T05:33:58.888 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 5, 2024-03-28T05:33:58.888 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 3, 2024-03-28T05:33:58.888 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.009767147, 2024-03-28T05:33:58.888 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.006239963, 2024-03-28T05:33:58.889 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.001627586, 2024-03-28T05:33:58.889 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T05:33:58.889 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:33:58.889 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 158, 2024-03-28T05:33:58.889 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.017223399, 2024-03-28T05:33:58.889 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000109008 2024-03-28T05:33:58.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:58.889 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:33:58.889 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:58.889 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:58.889 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:58.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:58.890 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T05:33:58.890 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 183456 2024-03-28T05:33:58.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:58.890 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T05:33:58.890 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 0, 2024-03-28T05:33:58.890 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2, 2024-03-28T05:33:58.890 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 0, 2024-03-28T05:33:58.890 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 0, 2024-03-28T05:33:58.890 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-03-28T05:33:58.890 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-03-28T05:33:58.890 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.000424645, 2024-03-28T05:33:58.891 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000000000, 2024-03-28T05:33:58.891 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000351087, 2024-03-28T05:33:58.891 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T05:33:58.891 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:33:58.891 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:58.891 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:58.891 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:58.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:58.891 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:33:58.891 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:58.891 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:58.891 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:58.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:58.892 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T05:33:58.892 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T05:33:58.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:58.892 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-03-28T05:33:58.892 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-03-28T05:33:58.892 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-03-28T05:33:58.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:58.892 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-03-28T05:33:58.892 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-03-28T05:33:58.892 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-03-28T05:33:58.892 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T05:33:58.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-03-28T05:33:58.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T05:33:58.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-03-28T05:33:58.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T05:33:58.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T05:33:58.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T05:33:58.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T05:33:58.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T05:33:58.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-03-28T05:33:58.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T05:33:58.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T05:33:58.893 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-03-28T05:33:58.894 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-03-28T05:33:58.894 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-03-28T05:33:58.894 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-03-28T05:33:58.894 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T05:33:58.894 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T05:33:58.894 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T05:33:58.894 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T05:33:58.894 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T05:33:58.894 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-03-28T05:33:58.894 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-03-28T05:33:58.894 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-03-28T05:33:58.894 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T05:33:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T05:33:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-03-28T05:33:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-03-28T05:33:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-03-28T05:33:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-03-28T05:33:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T05:33:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-03-28T05:33:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T05:33:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-03-28T05:33:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 148977, 2024-03-28T05:33:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 30, 2024-03-28T05:33:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-03-28T05:33:58.895 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-03-28T05:33:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-03-28T05:33:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-03-28T05:33:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-03-28T05:33:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-03-28T05:33:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-03-28T05:33:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-03-28T05:33:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-03-28T05:33:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-03-28T05:33:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T05:33:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-03-28T05:33:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-03-28T05:33:58.896 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-03-28T05:33:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-03-28T05:33:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-03-28T05:33:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-03-28T05:33:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-03-28T05:33:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-03-28T05:33:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-03-28T05:33:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:33:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T05:33:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-28T05:33:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-28T05:33:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-28T05:33:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 225, 2024-03-28T05:33:58.897 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 1543647, 2024-03-28T05:33:58.898 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-28T05:33:58.898 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 225, 2024-03-28T05:33:58.898 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-28T05:33:58.898 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-28T05:33:58.898 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 225, 2024-03-28T05:33:58.898 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1543647, 2024-03-28T05:33:58.898 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-28T05:33:58.898 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:33:58.898 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:33:58.898 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:33:58.898 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-28T05:33:58.899 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-28T05:33:58.899 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-28T05:33:59.494 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:33:59 smithi113 ceph-mon[37164]: pgmap v323: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:33:59.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:33:59 smithi023 ceph-mon[31954]: pgmap v323: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:33:59.534 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:33:59 smithi098 ceph-mon[36087]: pgmap v323: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:33:59.562 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.h 2024-03-28T05:33:59.562 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.h 2024-03-28T05:33:59.562 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.h 2024-03-28T05:33:59.562 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.h 2024-03-28T05:33:59.562 DEBUG:tasks.check_counter:Getting stats from k 2024-03-28T05:33:59.563 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-28T05:33:59.800 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T05:34:01.464 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:01 smithi098 ceph-mon[36087]: pgmap v324: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:34:01.494 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:01 smithi113 ceph-mon[37164]: pgmap v324: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:34:01.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:01 smithi023 ceph-mon[31954]: pgmap v324: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-28T05:34:01.729 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-28T05:34:01.729 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T05:34:01.729 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 4, 2024-03-28T05:34:01.729 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 4, 2024-03-28T05:34:01.729 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 2992, 2024-03-28T05:34:01.729 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 448, 2024-03-28T05:34:01.729 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-03-28T05:34:01.729 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-03-28T05:34:01.730 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.001510451, 2024-03-28T05:34:01.730 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000033871, 2024-03-28T05:34:01.730 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000723039, 2024-03-28T05:34:01.730 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T05:34:01.730 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:34:01.730 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 3, 2024-03-28T05:34:01.730 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000785985, 2024-03-28T05:34:01.730 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000261995 2024-03-28T05:34:01.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:34:01.730 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:34:01.730 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:34:01.731 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:34:01.731 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:34:01.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:34:01.731 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-28T05:34:01.731 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T05:34:01.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:34:01.731 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T05:34:01.731 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 228, 2024-03-28T05:34:01.731 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 217, 2024-03-28T05:34:01.731 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 1576432, 2024-03-28T05:34:01.731 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 129936, 2024-03-28T05:34:01.732 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-03-28T05:34:01.732 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-03-28T05:34:01.732 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.044197735, 2024-03-28T05:34:01.732 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.013539139, 2024-03-28T05:34:01.732 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.034435458, 2024-03-28T05:34:01.732 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.005256966, 2024-03-28T05:34:01.732 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:34:01.732 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 216, 2024-03-28T05:34:01.732 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.012720019, 2024-03-28T05:34:01.732 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000058888 2024-03-28T05:34:01.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:34:01.732 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:34:01.733 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:34:01.733 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:34:01.733 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:34:01.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:34:01.733 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1576432, 2024-03-28T05:34:01.733 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 129936 2024-03-28T05:34:01.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:34:01.733 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T05:34:01.733 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 2, 2024-03-28T05:34:01.733 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 166, 2024-03-28T05:34:01.733 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 320, 2024-03-28T05:34:01.734 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 184336, 2024-03-28T05:34:01.734 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 5, 2024-03-28T05:34:01.734 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 3, 2024-03-28T05:34:01.734 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.010802134, 2024-03-28T05:34:01.734 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.006887790, 2024-03-28T05:34:01.734 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.001777309, 2024-03-28T05:34:01.734 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T05:34:01.734 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:34:01.734 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 163, 2024-03-28T05:34:01.734 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.018737218, 2024-03-28T05:34:01.734 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000114952 2024-03-28T05:34:01.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:34:01.735 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:34:01.735 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:34:01.735 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:34:01.735 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:34:01.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:34:01.735 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T05:34:01.735 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 184336 2024-03-28T05:34:01.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:34:01.735 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-03-28T05:34:01.735 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-03-28T05:34:01.736 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-03-28T05:34:01.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:34:01.736 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-03-28T05:34:01.736 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-03-28T05:34:01.736 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-03-28T05:34:01.736 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T05:34:01.736 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-03-28T05:34:01.736 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T05:34:01.736 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-03-28T05:34:01.736 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T05:34:01.736 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T05:34:01.737 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T05:34:01.737 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T05:34:01.737 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T05:34:01.737 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-03-28T05:34:01.737 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T05:34:01.737 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T05:34:01.737 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-03-28T05:34:01.737 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-03-28T05:34:01.737 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-03-28T05:34:01.737 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-03-28T05:34:01.737 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T05:34:01.738 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T05:34:01.738 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T05:34:01.738 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T05:34:01.738 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T05:34:01.738 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-03-28T05:34:01.738 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-03-28T05:34:01.738 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-03-28T05:34:01.738 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T05:34:01.738 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T05:34:01.738 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-03-28T05:34:01.738 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-03-28T05:34:01.739 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-03-28T05:34:01.739 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-03-28T05:34:01.739 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T05:34:01.739 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-03-28T05:34:01.739 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T05:34:01.739 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-03-28T05:34:01.739 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 149017, 2024-03-28T05:34:01.739 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 31, 2024-03-28T05:34:01.739 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-03-28T05:34:01.739 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-03-28T05:34:01.739 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-03-28T05:34:01.739 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-03-28T05:34:01.740 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-03-28T05:34:01.740 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-03-28T05:34:01.740 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-03-28T05:34:01.740 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-03-28T05:34:01.740 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-03-28T05:34:01.740 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-03-28T05:34:01.740 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T05:34:01.740 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-03-28T05:34:01.740 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-03-28T05:34:01.740 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-03-28T05:34:01.740 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-03-28T05:34:01.741 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-03-28T05:34:01.741 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-03-28T05:34:01.741 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-03-28T05:34:01.741 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-03-28T05:34:01.741 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-03-28T05:34:01.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-28T05:34:01.741 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T05:34:01.741 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-03-28T05:34:01.741 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-03-28T05:34:01.741 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-03-28T05:34:01.741 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 230, 2024-03-28T05:34:01.741 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 1544297, 2024-03-28T05:34:01.742 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-03-28T05:34:01.742 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 230, 2024-03-28T05:34:01.742 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-03-28T05:34:01.742 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-03-28T05:34:01.742 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 230, 2024-03-28T05:34:01.742 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1544297, 2024-03-28T05:34:01.742 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-03-28T05:34:01.742 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-03-28T05:34:01.742 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-03-28T05:34:01.742 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-03-28T05:34:01.742 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-28T05:34:01.742 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-28T05:34:01.743 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-28T05:34:02.421 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-03-28T05:34:02.421 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-03-28T05:34:02.421 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-03-28T05:34:02.421 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2024-03-28T05:34:02.421 DEBUG:tasks.check_counter:Getting stats from c 2024-03-28T05:34:02.421 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-28T05:34:02.704 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T05:34:03.494 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:03 smithi113 ceph-mon[37164]: pgmap v325: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-03-28T05:34:03.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:03 smithi023 ceph-mon[31954]: pgmap v325: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-03-28T05:34:03.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:03 smithi098 ceph-mon[36087]: pgmap v325: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 2 op/s 2024-03-28T05:34:04.536 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-28T05:34:04.536 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T05:34:04.536 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 225, 2024-03-28T05:34:04.536 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 214, 2024-03-28T05:34:04.536 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 1575616, 2024-03-28T05:34:04.536 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 128064, 2024-03-28T05:34:04.536 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 4, 2024-03-28T05:34:04.536 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 0, 2024-03-28T05:34:04.536 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 0.034941177, 2024-03-28T05:34:04.536 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 0.008427099, 2024-03-28T05:34:04.536 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.032587291, 2024-03-28T05:34:04.537 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 0.003916659, 2024-03-28T05:34:04.537 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:34:04.537 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 213, 2024-03-28T05:34:04.537 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.007765036, 2024-03-28T05:34:04.537 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000036455 2024-03-28T05:34:04.537 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:04.537 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:34:04.537 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:04.537 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:04.537 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:04.537 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:04.537 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 1575616, 2024-03-28T05:34:04.537 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 128064 2024-03-28T05:34:04.537 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:04.538 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T05:34:04.538 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 6, 2024-03-28T05:34:04.538 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 165, 2024-03-28T05:34:04.538 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 3312, 2024-03-28T05:34:04.538 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 184256, 2024-03-28T05:34:04.538 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 5, 2024-03-28T05:34:04.538 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 3, 2024-03-28T05:34:04.538 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 0.008118181, 2024-03-28T05:34:04.538 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 0.004600667, 2024-03-28T05:34:04.538 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.002144777, 2024-03-28T05:34:04.538 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T05:34:04.538 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:34:04.538 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 163, 2024-03-28T05:34:04.538 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.009119167, 2024-03-28T05:34:04.539 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000055945 2024-03-28T05:34:04.539 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:04.539 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:34:04.539 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:04.539 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:04.539 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:04.539 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:04.539 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-28T05:34:04.539 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 184256 2024-03-28T05:34:04.539 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:04.539 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T05:34:04.539 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 0, 2024-03-28T05:34:04.539 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 2, 2024-03-28T05:34:04.539 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 0, 2024-03-28T05:34:04.539 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 0, 2024-03-28T05:34:04.540 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 2, 2024-03-28T05:34:04.540 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 1, 2024-03-28T05:34:04.540 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 0.000303275, 2024-03-28T05:34:04.540 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 0.000000000, 2024-03-28T05:34:04.540 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.000243260, 2024-03-28T05:34:04.540 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T05:34:04.540 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:34:04.540 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:04.540 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:04.540 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:04.540 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:04.540 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:34:04.540 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:04.540 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:04.540 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:04.541 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:04.541 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T05:34:04.541 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T05:34:04.541 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:04.541 INFO:teuthology.orchestra.run.smithi113.stdout: "cct": { 2024-03-28T05:34:04.541 INFO:teuthology.orchestra.run.smithi113.stdout: "total_workers": 0, 2024-03-28T05:34:04.541 INFO:teuthology.orchestra.run.smithi113.stdout: "unhealthy_workers": 0 2024-03-28T05:34:04.541 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:04.541 INFO:teuthology.orchestra.run.smithi113.stdout: "mempool": { 2024-03-28T05:34:04.541 INFO:teuthology.orchestra.run.smithi113.stdout: "bloom_filter_bytes": 0, 2024-03-28T05:34:04.541 INFO:teuthology.orchestra.run.smithi113.stdout: "bloom_filter_items": 0, 2024-03-28T05:34:04.541 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T05:34:04.541 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_alloc_items": 0, 2024-03-28T05:34:04.541 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T05:34:04.541 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_data_items": 0, 2024-03-28T05:34:04.542 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T05:34:04.542 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T05:34:04.542 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T05:34:04.542 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T05:34:04.542 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T05:34:04.542 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_other_items": 0, 2024-03-28T05:34:04.542 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T05:34:04.542 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T05:34:04.542 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_extent_bytes": 0, 2024-03-28T05:34:04.542 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_extent_items": 0, 2024-03-28T05:34:04.542 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_blob_bytes": 0, 2024-03-28T05:34:04.542 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_blob_items": 0, 2024-03-28T05:34:04.542 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T05:34:04.542 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T05:34:04.543 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T05:34:04.543 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T05:34:04.543 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T05:34:04.543 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_fsck_items": 0, 2024-03-28T05:34:04.543 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_txc_bytes": 0, 2024-03-28T05:34:04.543 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_txc_items": 0, 2024-03-28T05:34:04.543 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T05:34:04.543 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T05:34:04.543 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_bytes": 0, 2024-03-28T05:34:04.543 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_items": 0, 2024-03-28T05:34:04.543 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_bytes": 0, 2024-03-28T05:34:04.543 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_items": 0, 2024-03-28T05:34:04.543 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T05:34:04.543 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_reader_items": 0, 2024-03-28T05:34:04.543 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T05:34:04.543 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_writer_items": 0, 2024-03-28T05:34:04.544 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_anon_bytes": 148977, 2024-03-28T05:34:04.544 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_anon_items": 30, 2024-03-28T05:34:04.544 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_meta_bytes": 0, 2024-03-28T05:34:04.544 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_meta_items": 0, 2024-03-28T05:34:04.544 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_bytes": 0, 2024-03-28T05:34:04.544 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_items": 0, 2024-03-28T05:34:04.544 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_mapbl_bytes": 0, 2024-03-28T05:34:04.544 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_mapbl_items": 0, 2024-03-28T05:34:04.544 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_pglog_bytes": 0, 2024-03-28T05:34:04.544 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_pglog_items": 0, 2024-03-28T05:34:04.544 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_bytes": 0, 2024-03-28T05:34:04.544 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_items": 0, 2024-03-28T05:34:04.544 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T05:34:04.544 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_mapping_items": 0, 2024-03-28T05:34:04.545 INFO:teuthology.orchestra.run.smithi113.stdout: "pgmap_bytes": 0, 2024-03-28T05:34:04.545 INFO:teuthology.orchestra.run.smithi113.stdout: "pgmap_items": 0, 2024-03-28T05:34:04.545 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_co_bytes": 928, 2024-03-28T05:34:04.545 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_co_items": 2, 2024-03-28T05:34:04.545 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_1_bytes": 0, 2024-03-28T05:34:04.545 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_1_items": 0, 2024-03-28T05:34:04.545 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_2_bytes": 0, 2024-03-28T05:34:04.545 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_2_items": 0 2024-03-28T05:34:04.545 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:04.545 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T05:34:04.545 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-28T05:34:04.545 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 104857600, 2024-03-28T05:34:04.545 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-28T05:34:04.545 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 227, 2024-03-28T05:34:04.545 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 1543907, 2024-03-28T05:34:04.546 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-28T05:34:04.546 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 227, 2024-03-28T05:34:04.546 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 0, 2024-03-28T05:34:04.546 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 0, 2024-03-28T05:34:04.546 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 227, 2024-03-28T05:34:04.546 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 1543907, 2024-03-28T05:34:04.546 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-28T05:34:04.546 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:04.546 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:04.546 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:04.546 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-28T05:34:04.546 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-28T05:34:04.546 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-28T05:34:05.258 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.c 2024-03-28T05:34:05.258 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.c 2024-03-28T05:34:05.258 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.c 2024-03-28T05:34:05.258 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.c 2024-03-28T05:34:05.258 DEBUG:tasks.check_counter:Getting stats from f 2024-03-28T05:34:05.258 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-28T05:34:05.493 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:05 smithi113 ceph-mon[37164]: pgmap v326: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:34:05.521 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T05:34:05.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:05 smithi023 ceph-mon[31954]: pgmap v326: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:34:05.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:05 smithi098 ceph-mon[36087]: pgmap v326: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-28T05:34:07.379 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:07 smithi113 ceph-mon[37164]: pgmap v327: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-28T05:34:07.442 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-28T05:34:07.442 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T05:34:07.442 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 0, 2024-03-28T05:34:07.442 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 2, 2024-03-28T05:34:07.443 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 0, 2024-03-28T05:34:07.443 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 0, 2024-03-28T05:34:07.443 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 4, 2024-03-28T05:34:07.443 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 0, 2024-03-28T05:34:07.443 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 0.000994311, 2024-03-28T05:34:07.443 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 0.000000000, 2024-03-28T05:34:07.443 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.000164697, 2024-03-28T05:34:07.443 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T05:34:07.443 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:34:07.443 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:07.443 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:07.443 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:07.444 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:07.444 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:34:07.444 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:07.444 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:07.444 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:07.444 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:07.444 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T05:34:07.444 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T05:34:07.444 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:07.444 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T05:34:07.444 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 2, 2024-03-28T05:34:07.444 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 164, 2024-03-28T05:34:07.445 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 320, 2024-03-28T05:34:07.445 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 183984, 2024-03-28T05:34:07.445 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 5, 2024-03-28T05:34:07.445 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 3, 2024-03-28T05:34:07.445 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 0.007195376, 2024-03-28T05:34:07.445 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 0.004334854, 2024-03-28T05:34:07.445 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.001493437, 2024-03-28T05:34:07.445 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T05:34:07.445 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:34:07.445 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 161, 2024-03-28T05:34:07.445 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.012639873, 2024-03-28T05:34:07.445 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000078508 2024-03-28T05:34:07.446 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:07.446 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:34:07.446 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:07.446 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:07.446 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:07.446 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:07.446 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T05:34:07.446 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 183984 2024-03-28T05:34:07.446 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:07.446 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T05:34:07.446 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 231, 2024-03-28T05:34:07.447 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 218, 2024-03-28T05:34:07.447 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 1579152, 2024-03-28T05:34:07.447 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 129760, 2024-03-28T05:34:07.447 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 2, 2024-03-28T05:34:07.447 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 1, 2024-03-28T05:34:07.447 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 0.037118190, 2024-03-28T05:34:07.447 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 0.008776762, 2024-03-28T05:34:07.447 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.035478685, 2024-03-28T05:34:07.447 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 0.003893908, 2024-03-28T05:34:07.447 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:34:07.447 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 218, 2024-03-28T05:34:07.447 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.007516186, 2024-03-28T05:34:07.447 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000034477 2024-03-28T05:34:07.448 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:07.448 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:34:07.448 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:07.448 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:07.448 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:07.448 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:07.448 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 1579152, 2024-03-28T05:34:07.448 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 129760 2024-03-28T05:34:07.448 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:07.448 INFO:teuthology.orchestra.run.smithi113.stdout: "cct": { 2024-03-28T05:34:07.448 INFO:teuthology.orchestra.run.smithi113.stdout: "total_workers": 0, 2024-03-28T05:34:07.449 INFO:teuthology.orchestra.run.smithi113.stdout: "unhealthy_workers": 0 2024-03-28T05:34:07.449 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:07.449 INFO:teuthology.orchestra.run.smithi113.stdout: "mempool": { 2024-03-28T05:34:07.449 INFO:teuthology.orchestra.run.smithi113.stdout: "bloom_filter_bytes": 0, 2024-03-28T05:34:07.449 INFO:teuthology.orchestra.run.smithi113.stdout: "bloom_filter_items": 0, 2024-03-28T05:34:07.449 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T05:34:07.449 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_alloc_items": 0, 2024-03-28T05:34:07.449 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T05:34:07.449 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_data_items": 0, 2024-03-28T05:34:07.449 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T05:34:07.449 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T05:34:07.449 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T05:34:07.450 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T05:34:07.450 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T05:34:07.450 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_other_items": 0, 2024-03-28T05:34:07.450 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T05:34:07.450 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T05:34:07.450 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_extent_bytes": 0, 2024-03-28T05:34:07.450 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_extent_items": 0, 2024-03-28T05:34:07.450 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_blob_bytes": 0, 2024-03-28T05:34:07.450 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_blob_items": 0, 2024-03-28T05:34:07.450 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T05:34:07.450 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T05:34:07.451 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T05:34:07.451 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T05:34:07.451 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T05:34:07.451 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_fsck_items": 0, 2024-03-28T05:34:07.451 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_txc_bytes": 0, 2024-03-28T05:34:07.451 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_txc_items": 0, 2024-03-28T05:34:07.451 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T05:34:07.451 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T05:34:07.451 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_bytes": 0, 2024-03-28T05:34:07.451 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_items": 0, 2024-03-28T05:34:07.451 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_bytes": 0, 2024-03-28T05:34:07.451 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_items": 0, 2024-03-28T05:34:07.451 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T05:34:07.452 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_reader_items": 0, 2024-03-28T05:34:07.452 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T05:34:07.452 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_writer_items": 0, 2024-03-28T05:34:07.452 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_anon_bytes": 149017, 2024-03-28T05:34:07.452 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_anon_items": 31, 2024-03-28T05:34:07.452 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_meta_bytes": 0, 2024-03-28T05:34:07.452 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_meta_items": 0, 2024-03-28T05:34:07.452 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_bytes": 0, 2024-03-28T05:34:07.452 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_items": 0, 2024-03-28T05:34:07.452 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_mapbl_bytes": 0, 2024-03-28T05:34:07.452 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_mapbl_items": 0, 2024-03-28T05:34:07.452 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_pglog_bytes": 0, 2024-03-28T05:34:07.453 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_pglog_items": 0, 2024-03-28T05:34:07.453 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_bytes": 0, 2024-03-28T05:34:07.453 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_items": 0, 2024-03-28T05:34:07.453 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T05:34:07.453 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_mapping_items": 0, 2024-03-28T05:34:07.453 INFO:teuthology.orchestra.run.smithi113.stdout: "pgmap_bytes": 0, 2024-03-28T05:34:07.453 INFO:teuthology.orchestra.run.smithi113.stdout: "pgmap_items": 0, 2024-03-28T05:34:07.453 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_co_bytes": 928, 2024-03-28T05:34:07.453 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_co_items": 2, 2024-03-28T05:34:07.453 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_1_bytes": 0, 2024-03-28T05:34:07.453 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_1_items": 0, 2024-03-28T05:34:07.453 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_2_bytes": 0, 2024-03-28T05:34:07.454 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_2_items": 0 2024-03-28T05:34:07.454 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:07.454 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T05:34:07.454 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-28T05:34:07.454 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 104857600, 2024-03-28T05:34:07.454 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-28T05:34:07.454 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 229, 2024-03-28T05:34:07.454 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 1544167, 2024-03-28T05:34:07.454 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-28T05:34:07.454 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 229, 2024-03-28T05:34:07.455 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 0, 2024-03-28T05:34:07.455 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 0, 2024-03-28T05:34:07.455 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 229, 2024-03-28T05:34:07.455 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 1544167, 2024-03-28T05:34:07.455 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-28T05:34:07.455 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:07.455 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:07.455 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:07.455 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-28T05:34:07.455 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-28T05:34:07.455 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-28T05:34:07.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:07 smithi023 ceph-mon[31954]: pgmap v327: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-28T05:34:07.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:07 smithi098 ceph-mon[36087]: pgmap v327: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-28T05:34:08.168 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.f 2024-03-28T05:34:08.168 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.f 2024-03-28T05:34:08.169 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.f 2024-03-28T05:34:08.169 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.f 2024-03-28T05:34:08.169 DEBUG:tasks.check_counter:Getting stats from i 2024-03-28T05:34:08.169 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-28T05:34:08.427 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T05:34:09.494 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:09 smithi113 ceph-mon[37164]: pgmap v328: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 18 op/s 2024-03-28T05:34:09.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:09 smithi023 ceph-mon[31954]: pgmap v328: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 18 op/s 2024-03-28T05:34:09.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:09 smithi098 ceph-mon[36087]: pgmap v328: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 18 op/s 2024-03-28T05:34:10.160 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-28T05:34:10.160 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T05:34:10.160 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 36257, 2024-03-28T05:34:10.160 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 35667, 2024-03-28T05:34:10.160 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 11874497, 2024-03-28T05:34:10.160 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 123106284, 2024-03-28T05:34:10.160 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 11, 2024-03-28T05:34:10.161 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 5, 2024-03-28T05:34:10.161 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 3.270433219, 2024-03-28T05:34:10.161 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 1.395673424, 2024-03-28T05:34:10.161 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.552513236, 2024-03-28T05:34:10.161 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 1.151758995, 2024-03-28T05:34:10.161 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:34:10.161 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 35666, 2024-03-28T05:34:10.161 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 29.348216303, 2024-03-28T05:34:10.161 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000822862 2024-03-28T05:34:10.161 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.161 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:34:10.162 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 900, 2024-03-28T05:34:10.162 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000119355, 2024-03-28T05:34:10.162 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000132 2024-03-28T05:34:10.162 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.162 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 1939312, 2024-03-28T05:34:10.162 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 238224 2024-03-28T05:34:10.162 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.162 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T05:34:10.162 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 37139, 2024-03-28T05:34:10.162 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 37001, 2024-03-28T05:34:10.162 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 12446767, 2024-03-28T05:34:10.162 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 204782066, 2024-03-28T05:34:10.162 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 76, 2024-03-28T05:34:10.163 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 5, 2024-03-28T05:34:10.163 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 3.227075486, 2024-03-28T05:34:10.163 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 1.234006670, 2024-03-28T05:34:10.163 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.531290344, 2024-03-28T05:34:10.163 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 1.281095564, 2024-03-28T05:34:10.163 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:34:10.163 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 36999, 2024-03-28T05:34:10.163 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 30.902171290, 2024-03-28T05:34:10.163 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000835216 2024-03-28T05:34:10.163 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.163 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:34:10.163 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 274, 2024-03-28T05:34:10.163 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000032261, 2024-03-28T05:34:10.163 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000117 2024-03-28T05:34:10.163 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.164 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 89440, 2024-03-28T05:34:10.164 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 144976 2024-03-28T05:34:10.164 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.164 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T05:34:10.164 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 630924, 2024-03-28T05:34:10.164 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 736999, 2024-03-28T05:34:10.164 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 216415104, 2024-03-28T05:34:10.164 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 618554903, 2024-03-28T05:34:10.164 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 16, 2024-03-28T05:34:10.164 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 6, 2024-03-28T05:34:10.164 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 27.727218600, 2024-03-28T05:34:10.164 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 12.710204818, 2024-03-28T05:34:10.164 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 21.653102913, 2024-03-28T05:34:10.165 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 1.201773160, 2024-03-28T05:34:10.165 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:34:10.165 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 736997, 2024-03-28T05:34:10.165 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 58.753373333, 2024-03-28T05:34:10.165 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000079719 2024-03-28T05:34:10.165 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.165 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:34:10.165 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 1196349, 2024-03-28T05:34:10.165 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.152200610, 2024-03-28T05:34:10.165 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000127 2024-03-28T05:34:10.165 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.165 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 207357152, 2024-03-28T05:34:10.165 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 527410176 2024-03-28T05:34:10.165 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.166 INFO:teuthology.orchestra.run.smithi113.stdout: "cct": { 2024-03-28T05:34:10.166 INFO:teuthology.orchestra.run.smithi113.stdout: "total_workers": 1, 2024-03-28T05:34:10.166 INFO:teuthology.orchestra.run.smithi113.stdout: "unhealthy_workers": 0 2024-03-28T05:34:10.166 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.166 INFO:teuthology.orchestra.run.smithi113.stdout: "finisher-MDSRank": { 2024-03-28T05:34:10.166 INFO:teuthology.orchestra.run.smithi113.stdout: "queue_len": 0, 2024-03-28T05:34:10.166 INFO:teuthology.orchestra.run.smithi113.stdout: "complete_latency": { 2024-03-28T05:34:10.166 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 45549, 2024-03-28T05:34:10.166 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 117.678315562, 2024-03-28T05:34:10.166 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.002583554 2024-03-28T05:34:10.166 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-28T05:34:10.166 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.166 INFO:teuthology.orchestra.run.smithi113.stdout: "finisher-PurgeQueue": { 2024-03-28T05:34:10.166 INFO:teuthology.orchestra.run.smithi113.stdout: "queue_len": 0, 2024-03-28T05:34:10.167 INFO:teuthology.orchestra.run.smithi113.stdout: "complete_latency": { 2024-03-28T05:34:10.167 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 23067, 2024-03-28T05:34:10.167 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 8.129860621, 2024-03-28T05:34:10.167 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000352445 2024-03-28T05:34:10.167 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-28T05:34:10.167 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.167 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-03-28T05:34:10.167 INFO:teuthology.orchestra.run.smithi113.stdout: "request": 470438, 2024-03-28T05:34:10.167 INFO:teuthology.orchestra.run.smithi113.stdout: "reply": 470438, 2024-03-28T05:34:10.167 INFO:teuthology.orchestra.run.smithi113.stdout: "reply_latency": { 2024-03-28T05:34:10.167 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 470438, 2024-03-28T05:34:10.167 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 15870.239227479, 2024-03-28T05:34:10.167 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.033735028 2024-03-28T05:34:10.167 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.167 INFO:teuthology.orchestra.run.smithi113.stdout: "slow_reply": 0, 2024-03-28T05:34:10.168 INFO:teuthology.orchestra.run.smithi113.stdout: "forward": 0, 2024-03-28T05:34:10.168 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_fetch_complete": 0, 2024-03-28T05:34:10.168 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_fetch_keys": 0, 2024-03-28T05:34:10.168 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_commit": 5012, 2024-03-28T05:34:10.168 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_split": 57, 2024-03-28T05:34:10.168 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_merge": 0, 2024-03-28T05:34:10.168 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes": 28037, 2024-03-28T05:34:10.168 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_top": 16914, 2024-03-28T05:34:10.168 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_bottom": 11123, 2024-03-28T05:34:10.168 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_pin_tail": 0, 2024-03-28T05:34:10.168 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_pinned": 3873, 2024-03-28T05:34:10.168 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_expired": 0, 2024-03-28T05:34:10.168 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_with_caps": 3, 2024-03-28T05:34:10.168 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": 4, 2024-03-28T05:34:10.168 INFO:teuthology.orchestra.run.smithi113.stdout: "subtrees": 2, 2024-03-28T05:34:10.169 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse": 965883, 2024-03-28T05:34:10.169 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_hit": 705854, 2024-03-28T05:34:10.169 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_forward": 0, 2024-03-28T05:34:10.169 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_discover": 0, 2024-03-28T05:34:10.169 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_dir_fetch": 0, 2024-03-28T05:34:10.169 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_remote_ino": 0, 2024-03-28T05:34:10.169 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_lock": 1231, 2024-03-28T05:34:10.169 INFO:teuthology.orchestra.run.smithi113.stdout: "load_cent": 0, 2024-03-28T05:34:10.169 INFO:teuthology.orchestra.run.smithi113.stdout: "q": 0, 2024-03-28T05:34:10.169 INFO:teuthology.orchestra.run.smithi113.stdout: "exported": 0, 2024-03-28T05:34:10.169 INFO:teuthology.orchestra.run.smithi113.stdout: "exported_inodes": 0, 2024-03-28T05:34:10.169 INFO:teuthology.orchestra.run.smithi113.stdout: "imported": 0, 2024-03-28T05:34:10.169 INFO:teuthology.orchestra.run.smithi113.stdout: "imported_inodes": 0, 2024-03-28T05:34:10.169 INFO:teuthology.orchestra.run.smithi113.stdout: "openino_dir_fetch": 0, 2024-03-28T05:34:10.169 INFO:teuthology.orchestra.run.smithi113.stdout: "openino_backtrace_fetch": 0, 2024-03-28T05:34:10.170 INFO:teuthology.orchestra.run.smithi113.stdout: "openino_peer_discover": 0, 2024-03-28T05:34:10.170 INFO:teuthology.orchestra.run.smithi113.stdout: "root_rfiles": 4, 2024-03-28T05:34:10.170 INFO:teuthology.orchestra.run.smithi113.stdout: "root_rbytes": 234, 2024-03-28T05:34:10.170 INFO:teuthology.orchestra.run.smithi113.stdout: "root_rsnaps": 6, 2024-03-28T05:34:10.170 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T05:34:10.170 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_set_tag": 0, 2024-03-28T05:34:10.170 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T05:34:10.170 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_inotable_repaired": 0, 2024-03-28T05:34:10.170 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_dir_inodes": 0, 2024-03-28T05:34:10.170 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T05:34:10.170 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T05:34:10.170 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_file_inodes": 0, 2024-03-28T05:34:10.170 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_inode_file_caps": 0, 2024-03-28T05:34:10.170 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_revoke": 37024, 2024-03-28T05:34:10.171 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_grant": 58777, 2024-03-28T05:34:10.171 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_trunc": 8, 2024-03-28T05:34:10.171 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T05:34:10.171 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T05:34:10.171 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_caps": 54376, 2024-03-28T05:34:10.171 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_caps_dirty": 19492, 2024-03-28T05:34:10.171 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_cap_release": 1141, 2024-03-28T05:34:10.171 INFO:teuthology.orchestra.run.smithi113.stdout: "process_request_cap_release": 75132 2024-03-28T05:34:10.171 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.171 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_cache": { 2024-03-28T05:34:10.171 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_update": 0, 2024-03-28T05:34:10.171 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_update_receipt": 0, 2024-03-28T05:34:10.171 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_try_discover": 0, 2024-03-28T05:34:10.171 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_send_discover": 0, 2024-03-28T05:34:10.171 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_handle_discover": 0, 2024-03-28T05:34:10.172 INFO:teuthology.orchestra.run.smithi113.stdout: "num_strays": 81, 2024-03-28T05:34:10.172 INFO:teuthology.orchestra.run.smithi113.stdout: "num_strays_delayed": 0, 2024-03-28T05:34:10.172 INFO:teuthology.orchestra.run.smithi113.stdout: "num_strays_enqueuing": 0, 2024-03-28T05:34:10.172 INFO:teuthology.orchestra.run.smithi113.stdout: "strays_created": 18032, 2024-03-28T05:34:10.172 INFO:teuthology.orchestra.run.smithi113.stdout: "strays_enqueued": 17951, 2024-03-28T05:34:10.172 INFO:teuthology.orchestra.run.smithi113.stdout: "strays_reintegrated": 0, 2024-03-28T05:34:10.172 INFO:teuthology.orchestra.run.smithi113.stdout: "strays_migrated": 0, 2024-03-28T05:34:10.172 INFO:teuthology.orchestra.run.smithi113.stdout: "num_recovering_processing": 0, 2024-03-28T05:34:10.172 INFO:teuthology.orchestra.run.smithi113.stdout: "num_recovering_enqueued": 0, 2024-03-28T05:34:10.172 INFO:teuthology.orchestra.run.smithi113.stdout: "num_recovering_prioritized": 0, 2024-03-28T05:34:10.172 INFO:teuthology.orchestra.run.smithi113.stdout: "recovery_started": 0, 2024-03-28T05:34:10.172 INFO:teuthology.orchestra.run.smithi113.stdout: "recovery_completed": 0, 2024-03-28T05:34:10.172 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T05:34:10.172 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_exportdir": 0, 2024-03-28T05:34:10.172 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_flush": 0, 2024-03-28T05:34:10.173 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_fragmentdir": 57, 2024-03-28T05:34:10.173 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_fragstats": 0, 2024-03-28T05:34:10.173 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_inodestats": 0 2024-03-28T05:34:10.173 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.173 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_log": { 2024-03-28T05:34:10.173 INFO:teuthology.orchestra.run.smithi113.stdout: "evlrg": 0, 2024-03-28T05:34:10.173 INFO:teuthology.orchestra.run.smithi113.stdout: "evadd": 97202, 2024-03-28T05:34:10.173 INFO:teuthology.orchestra.run.smithi113.stdout: "evex": 92982, 2024-03-28T05:34:10.173 INFO:teuthology.orchestra.run.smithi113.stdout: "evtrm": 92982, 2024-03-28T05:34:10.173 INFO:teuthology.orchestra.run.smithi113.stdout: "ev": 4220, 2024-03-28T05:34:10.173 INFO:teuthology.orchestra.run.smithi113.stdout: "evexg": 0, 2024-03-28T05:34:10.173 INFO:teuthology.orchestra.run.smithi113.stdout: "evexd": 2953, 2024-03-28T05:34:10.173 INFO:teuthology.orchestra.run.smithi113.stdout: "segadd": 164, 2024-03-28T05:34:10.173 INFO:teuthology.orchestra.run.smithi113.stdout: "segex": 156, 2024-03-28T05:34:10.173 INFO:teuthology.orchestra.run.smithi113.stdout: "segtrm": 156, 2024-03-28T05:34:10.174 INFO:teuthology.orchestra.run.smithi113.stdout: "seg": 8, 2024-03-28T05:34:10.174 INFO:teuthology.orchestra.run.smithi113.stdout: "segmjr": 1, 2024-03-28T05:34:10.174 INFO:teuthology.orchestra.run.smithi113.stdout: "segexg": 0, 2024-03-28T05:34:10.174 INFO:teuthology.orchestra.run.smithi113.stdout: "segexd": 5, 2024-03-28T05:34:10.174 INFO:teuthology.orchestra.run.smithi113.stdout: "expos": 316852387, 2024-03-28T05:34:10.174 INFO:teuthology.orchestra.run.smithi113.stdout: "wrpos": 332181778, 2024-03-28T05:34:10.174 INFO:teuthology.orchestra.run.smithi113.stdout: "rdpos": 0, 2024-03-28T05:34:10.174 INFO:teuthology.orchestra.run.smithi113.stdout: "jlat": { 2024-03-28T05:34:10.174 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 23723, 2024-03-28T05:34:10.174 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 142.117632067, 2024-03-28T05:34:10.174 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.005990710 2024-03-28T05:34:10.174 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.174 INFO:teuthology.orchestra.run.smithi113.stdout: "replayed": 0 2024-03-28T05:34:10.174 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.174 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_mem": { 2024-03-28T05:34:10.175 INFO:teuthology.orchestra.run.smithi113.stdout: "ino": 16758, 2024-03-28T05:34:10.175 INFO:teuthology.orchestra.run.smithi113.stdout: "ino+": 34709, 2024-03-28T05:34:10.175 INFO:teuthology.orchestra.run.smithi113.stdout: "ino-": 17951, 2024-03-28T05:34:10.175 INFO:teuthology.orchestra.run.smithi113.stdout: "dir": 498, 2024-03-28T05:34:10.175 INFO:teuthology.orchestra.run.smithi113.stdout: "dir+": 559, 2024-03-28T05:34:10.175 INFO:teuthology.orchestra.run.smithi113.stdout: "dir-": 61, 2024-03-28T05:34:10.175 INFO:teuthology.orchestra.run.smithi113.stdout: "dn": 28037, 2024-03-28T05:34:10.175 INFO:teuthology.orchestra.run.smithi113.stdout: "dn+": 57859, 2024-03-28T05:34:10.175 INFO:teuthology.orchestra.run.smithi113.stdout: "dn-": 29822, 2024-03-28T05:34:10.175 INFO:teuthology.orchestra.run.smithi113.stdout: "cap": 4, 2024-03-28T05:34:10.175 INFO:teuthology.orchestra.run.smithi113.stdout: "cap+": 26575, 2024-03-28T05:34:10.175 INFO:teuthology.orchestra.run.smithi113.stdout: "cap-": 26571, 2024-03-28T05:34:10.175 INFO:teuthology.orchestra.run.smithi113.stdout: "rss": 178848, 2024-03-28T05:34:10.175 INFO:teuthology.orchestra.run.smithi113.stdout: "heap": 231804 2024-03-28T05:34:10.175 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.176 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_server": { 2024-03-28T05:34:10.176 INFO:teuthology.orchestra.run.smithi113.stdout: "dispatch_client_request": 528600, 2024-03-28T05:34:10.176 INFO:teuthology.orchestra.run.smithi113.stdout: "dispatch_server_request": 0, 2024-03-28T05:34:10.176 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_request": 470438, 2024-03-28T05:34:10.176 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_session": 36, 2024-03-28T05:34:10.176 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_peer_request": 0, 2024-03-28T05:34:10.176 INFO:teuthology.orchestra.run.smithi113.stdout: "req_create_latency": { 2024-03-28T05:34:10.176 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 17953, 2024-03-28T05:34:10.176 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 743.294329018, 2024-03-28T05:34:10.176 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.041402235 2024-03-28T05:34:10.176 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.176 INFO:teuthology.orchestra.run.smithi113.stdout: "req_getattr_latency": { 2024-03-28T05:34:10.176 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 141, 2024-03-28T05:34:10.177 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.394149526, 2024-03-28T05:34:10.177 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.002795386 2024-03-28T05:34:10.177 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.177 INFO:teuthology.orchestra.run.smithi113.stdout: "req_getfilelock_latency": { 2024-03-28T05:34:10.177 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:10.177 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:10.177 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:10.177 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.177 INFO:teuthology.orchestra.run.smithi113.stdout: "req_link_latency": { 2024-03-28T05:34:10.177 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:10.177 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:10.177 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:10.177 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.178 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookup_latency": { 2024-03-28T05:34:10.178 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 407736, 2024-03-28T05:34:10.178 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 13759.392707330, 2024-03-28T05:34:10.178 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.033745837 2024-03-28T05:34:10.178 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.178 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookuphash_latency": { 2024-03-28T05:34:10.178 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:10.178 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:10.178 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:10.178 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.178 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookupino_latency": { 2024-03-28T05:34:10.178 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 2, 2024-03-28T05:34:10.178 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000251770, 2024-03-28T05:34:10.178 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000125885 2024-03-28T05:34:10.179 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.179 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookupname_latency": { 2024-03-28T05:34:10.179 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:10.179 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:10.179 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:10.179 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.179 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookupparent_latency": { 2024-03-28T05:34:10.179 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:10.179 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:10.179 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:10.179 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.179 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookupsnap_latency": { 2024-03-28T05:34:10.179 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:10.179 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:10.179 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:10.180 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.180 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lssnap_latency": { 2024-03-28T05:34:10.180 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 11, 2024-03-28T05:34:10.180 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.005122173, 2024-03-28T05:34:10.180 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000465652 2024-03-28T05:34:10.180 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.180 INFO:teuthology.orchestra.run.smithi113.stdout: "req_mkdir_latency": { 2024-03-28T05:34:10.180 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 91, 2024-03-28T05:34:10.181 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 3.035506532, 2024-03-28T05:34:10.181 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.033357214 2024-03-28T05:34:10.181 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.181 INFO:teuthology.orchestra.run.smithi113.stdout: "req_mknod_latency": { 2024-03-28T05:34:10.181 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:10.181 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:10.181 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:10.181 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.181 INFO:teuthology.orchestra.run.smithi113.stdout: "req_mksnap_latency": { 2024-03-28T05:34:10.181 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 11, 2024-03-28T05:34:10.181 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 8.678444699, 2024-03-28T05:34:10.181 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.788949518 2024-03-28T05:34:10.181 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.181 INFO:teuthology.orchestra.run.smithi113.stdout: "req_open_latency": { 2024-03-28T05:34:10.181 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 15160, 2024-03-28T05:34:10.182 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 367.762516028, 2024-03-28T05:34:10.182 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.024258741 2024-03-28T05:34:10.182 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.182 INFO:teuthology.orchestra.run.smithi113.stdout: "req_readdir_latency": { 2024-03-28T05:34:10.182 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 541, 2024-03-28T05:34:10.182 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.714483091, 2024-03-28T05:34:10.182 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.001320671 2024-03-28T05:34:10.182 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.182 INFO:teuthology.orchestra.run.smithi113.stdout: "req_rename_latency": { 2024-03-28T05:34:10.182 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 17570, 2024-03-28T05:34:10.182 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 925.735328193, 2024-03-28T05:34:10.182 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.052688407 2024-03-28T05:34:10.182 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.182 INFO:teuthology.orchestra.run.smithi113.stdout: "req_renamesnap_latency": { 2024-03-28T05:34:10.183 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:10.183 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:10.183 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:10.183 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.183 INFO:teuthology.orchestra.run.smithi113.stdout: "req_snapdiff_latency": { 2024-03-28T05:34:10.183 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:10.183 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:10.183 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:10.183 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.183 INFO:teuthology.orchestra.run.smithi113.stdout: "req_rmdir_latency": { 2024-03-28T05:34:10.183 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 85, 2024-03-28T05:34:10.183 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.617511137, 2024-03-28T05:34:10.183 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.007264836 2024-03-28T05:34:10.183 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.183 INFO:teuthology.orchestra.run.smithi113.stdout: "req_rmsnap_latency": { 2024-03-28T05:34:10.184 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 5, 2024-03-28T05:34:10.184 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 1.322420100, 2024-03-28T05:34:10.184 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.264484020 2024-03-28T05:34:10.184 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.184 INFO:teuthology.orchestra.run.smithi113.stdout: "req_rmxattr_latency": { 2024-03-28T05:34:10.184 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:10.184 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:10.184 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:10.184 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.184 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setattr_latency": { 2024-03-28T05:34:10.184 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 14, 2024-03-28T05:34:10.184 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.158850168, 2024-03-28T05:34:10.184 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.011346440 2024-03-28T05:34:10.184 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.185 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setdirlayout_latency": { 2024-03-28T05:34:10.185 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:10.185 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:10.185 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:10.185 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.185 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setfilelock_latency": { 2024-03-28T05:34:10.185 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:10.185 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:10.185 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:10.185 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.185 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setlayout_latency": { 2024-03-28T05:34:10.185 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:10.185 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:10.185 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:10.185 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.186 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setxattr_latency": { 2024-03-28T05:34:10.186 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 8, 2024-03-28T05:34:10.186 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.019974730, 2024-03-28T05:34:10.186 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.002496841 2024-03-28T05:34:10.186 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.186 INFO:teuthology.orchestra.run.smithi113.stdout: "req_symlink_latency": { 2024-03-28T05:34:10.186 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 3, 2024-03-28T05:34:10.186 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.007892355, 2024-03-28T05:34:10.186 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.002630785 2024-03-28T05:34:10.186 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.186 INFO:teuthology.orchestra.run.smithi113.stdout: "req_unlink_latency": { 2024-03-28T05:34:10.186 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 11107, 2024-03-28T05:34:10.186 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 59.099740629, 2024-03-28T05:34:10.186 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.005320945 2024-03-28T05:34:10.186 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.187 INFO:teuthology.orchestra.run.smithi113.stdout: "cap_revoke_eviction": 0, 2024-03-28T05:34:10.187 INFO:teuthology.orchestra.run.smithi113.stdout: "cap_acquisition_throttle": 0, 2024-03-28T05:34:10.187 INFO:teuthology.orchestra.run.smithi113.stdout: "req_getvxattr_latency": { 2024-03-28T05:34:10.187 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:10.187 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:10.187 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:10.187 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-28T05:34:10.187 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.187 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_sessions": { 2024-03-28T05:34:10.187 INFO:teuthology.orchestra.run.smithi113.stdout: "session_count": 2, 2024-03-28T05:34:10.187 INFO:teuthology.orchestra.run.smithi113.stdout: "session_add": 6, 2024-03-28T05:34:10.187 INFO:teuthology.orchestra.run.smithi113.stdout: "session_remove": 4, 2024-03-28T05:34:10.187 INFO:teuthology.orchestra.run.smithi113.stdout: "sessions_open": 2, 2024-03-28T05:34:10.187 INFO:teuthology.orchestra.run.smithi113.stdout: "sessions_stale": 0, 2024-03-28T05:34:10.187 INFO:teuthology.orchestra.run.smithi113.stdout: "total_load": 10829, 2024-03-28T05:34:10.188 INFO:teuthology.orchestra.run.smithi113.stdout: "average_load": 5414, 2024-03-28T05:34:10.188 INFO:teuthology.orchestra.run.smithi113.stdout: "avg_session_uptime": 622, 2024-03-28T05:34:10.188 INFO:teuthology.orchestra.run.smithi113.stdout: "mdthresh_evicted": 0 2024-03-28T05:34:10.188 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.188 INFO:teuthology.orchestra.run.smithi113.stdout: "mempool": { 2024-03-28T05:34:10.188 INFO:teuthology.orchestra.run.smithi113.stdout: "bloom_filter_bytes": 0, 2024-03-28T05:34:10.188 INFO:teuthology.orchestra.run.smithi113.stdout: "bloom_filter_items": 0, 2024-03-28T05:34:10.188 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T05:34:10.188 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_alloc_items": 0, 2024-03-28T05:34:10.188 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T05:34:10.188 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_data_items": 0, 2024-03-28T05:34:10.188 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T05:34:10.188 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T05:34:10.188 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T05:34:10.188 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T05:34:10.188 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T05:34:10.189 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_other_items": 0, 2024-03-28T05:34:10.189 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T05:34:10.189 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T05:34:10.189 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_extent_bytes": 0, 2024-03-28T05:34:10.189 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_extent_items": 0, 2024-03-28T05:34:10.189 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_blob_bytes": 0, 2024-03-28T05:34:10.189 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_blob_items": 0, 2024-03-28T05:34:10.189 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T05:34:10.189 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T05:34:10.189 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T05:34:10.189 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T05:34:10.189 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T05:34:10.189 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_fsck_items": 0, 2024-03-28T05:34:10.189 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_txc_bytes": 0, 2024-03-28T05:34:10.189 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_txc_items": 0, 2024-03-28T05:34:10.190 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T05:34:10.190 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T05:34:10.190 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_bytes": 0, 2024-03-28T05:34:10.190 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_items": 0, 2024-03-28T05:34:10.190 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_bytes": 0, 2024-03-28T05:34:10.190 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_items": 0, 2024-03-28T05:34:10.190 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T05:34:10.190 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_reader_items": 0, 2024-03-28T05:34:10.190 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T05:34:10.190 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_writer_items": 0, 2024-03-28T05:34:10.190 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_anon_bytes": 1496959, 2024-03-28T05:34:10.190 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_anon_items": 14483, 2024-03-28T05:34:10.190 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_meta_bytes": 0, 2024-03-28T05:34:10.190 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_meta_items": 0, 2024-03-28T05:34:10.190 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_bytes": 0, 2024-03-28T05:34:10.190 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_items": 0, 2024-03-28T05:34:10.191 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_mapbl_bytes": 0, 2024-03-28T05:34:10.191 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_mapbl_items": 0, 2024-03-28T05:34:10.191 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_pglog_bytes": 0, 2024-03-28T05:34:10.191 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_pglog_items": 0, 2024-03-28T05:34:10.191 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_bytes": 9608, 2024-03-28T05:34:10.191 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_items": 184, 2024-03-28T05:34:10.191 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T05:34:10.191 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_mapping_items": 0, 2024-03-28T05:34:10.191 INFO:teuthology.orchestra.run.smithi113.stdout: "pgmap_bytes": 0, 2024-03-28T05:34:10.191 INFO:teuthology.orchestra.run.smithi113.stdout: "pgmap_items": 0, 2024-03-28T05:34:10.191 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_co_bytes": 59251703, 2024-03-28T05:34:10.191 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_co_items": 574105, 2024-03-28T05:34:10.191 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_1_bytes": 0, 2024-03-28T05:34:10.191 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_1_items": 0, 2024-03-28T05:34:10.192 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_2_bytes": 0, 2024-03-28T05:34:10.192 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_2_items": 0 2024-03-28T05:34:10.192 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.192 INFO:teuthology.orchestra.run.smithi113.stdout: "objecter": { 2024-03-28T05:34:10.192 INFO:teuthology.orchestra.run.smithi113.stdout: "op_active": 0, 2024-03-28T05:34:10.192 INFO:teuthology.orchestra.run.smithi113.stdout: "op_laggy": 0, 2024-03-28T05:34:10.192 INFO:teuthology.orchestra.run.smithi113.stdout: "op_send": 103915, 2024-03-28T05:34:10.192 INFO:teuthology.orchestra.run.smithi113.stdout: "op_send_bytes": 383977437, 2024-03-28T05:34:10.192 INFO:teuthology.orchestra.run.smithi113.stdout: "op_resend": 32, 2024-03-28T05:34:10.192 INFO:teuthology.orchestra.run.smithi113.stdout: "op_reply": 103915, 2024-03-28T05:34:10.192 INFO:teuthology.orchestra.run.smithi113.stdout: "op_latency": { 2024-03-28T05:34:10.192 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 103915, 2024-03-28T05:34:10.192 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 8195.134992786, 2024-03-28T05:34:10.192 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.078863830 2024-03-28T05:34:10.192 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.192 INFO:teuthology.orchestra.run.smithi113.stdout: "op_inflight": 0, 2024-03-28T05:34:10.193 INFO:teuthology.orchestra.run.smithi113.stdout: "oplen_avg": { 2024-03-28T05:34:10.193 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 103915, 2024-03-28T05:34:10.193 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 172854 2024-03-28T05:34:10.193 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.193 INFO:teuthology.orchestra.run.smithi113.stdout: "op": 103915, 2024-03-28T05:34:10.193 INFO:teuthology.orchestra.run.smithi113.stdout: "op_r": 413, 2024-03-28T05:34:10.193 INFO:teuthology.orchestra.run.smithi113.stdout: "op_w": 103502, 2024-03-28T05:34:10.193 INFO:teuthology.orchestra.run.smithi113.stdout: "op_rmw": 0, 2024-03-28T05:34:10.193 INFO:teuthology.orchestra.run.smithi113.stdout: "op_pg": 0, 2024-03-28T05:34:10.193 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_stat": 4457, 2024-03-28T05:34:10.193 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_create": 36924, 2024-03-28T05:34:10.193 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_read": 413, 2024-03-28T05:34:10.193 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_write": 24136, 2024-03-28T05:34:10.193 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_writefull": 334, 2024-03-28T05:34:10.193 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_writesame": 0, 2024-03-28T05:34:10.194 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_append": 0, 2024-03-28T05:34:10.194 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_zero": 0, 2024-03-28T05:34:10.194 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_truncate": 55, 2024-03-28T05:34:10.194 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_delete": 36561, 2024-03-28T05:34:10.194 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_mapext": 0, 2024-03-28T05:34:10.194 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_sparse_read": 0, 2024-03-28T05:34:10.194 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_clonerange": 0, 2024-03-28T05:34:10.194 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_getxattr": 0, 2024-03-28T05:34:10.194 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_setxattr": 56478, 2024-03-28T05:34:10.194 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_cmpxattr": 0, 2024-03-28T05:34:10.194 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_rmxattr": 0, 2024-03-28T05:34:10.194 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_resetxattrs": 0, 2024-03-28T05:34:10.194 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_call": 0, 2024-03-28T05:34:10.194 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_watch": 0, 2024-03-28T05:34:10.194 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_notify": 0, 2024-03-28T05:34:10.194 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T05:34:10.195 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_pgls": 0, 2024-03-28T05:34:10.195 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_pgls_filter": 0, 2024-03-28T05:34:10.195 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_other": 8, 2024-03-28T05:34:10.195 INFO:teuthology.orchestra.run.smithi113.stdout: "linger_active": 0, 2024-03-28T05:34:10.195 INFO:teuthology.orchestra.run.smithi113.stdout: "linger_send": 0, 2024-03-28T05:34:10.195 INFO:teuthology.orchestra.run.smithi113.stdout: "linger_resend": 0, 2024-03-28T05:34:10.195 INFO:teuthology.orchestra.run.smithi113.stdout: "linger_ping": 0, 2024-03-28T05:34:10.195 INFO:teuthology.orchestra.run.smithi113.stdout: "poolop_active": 0, 2024-03-28T05:34:10.195 INFO:teuthology.orchestra.run.smithi113.stdout: "poolop_send": 0, 2024-03-28T05:34:10.195 INFO:teuthology.orchestra.run.smithi113.stdout: "poolop_resend": 0, 2024-03-28T05:34:10.195 INFO:teuthology.orchestra.run.smithi113.stdout: "poolstat_active": 0, 2024-03-28T05:34:10.195 INFO:teuthology.orchestra.run.smithi113.stdout: "poolstat_send": 0, 2024-03-28T05:34:10.195 INFO:teuthology.orchestra.run.smithi113.stdout: "poolstat_resend": 0, 2024-03-28T05:34:10.195 INFO:teuthology.orchestra.run.smithi113.stdout: "statfs_active": 0, 2024-03-28T05:34:10.195 INFO:teuthology.orchestra.run.smithi113.stdout: "statfs_send": 0, 2024-03-28T05:34:10.195 INFO:teuthology.orchestra.run.smithi113.stdout: "statfs_resend": 0, 2024-03-28T05:34:10.196 INFO:teuthology.orchestra.run.smithi113.stdout: "command_active": 0, 2024-03-28T05:34:10.196 INFO:teuthology.orchestra.run.smithi113.stdout: "command_send": 0, 2024-03-28T05:34:10.196 INFO:teuthology.orchestra.run.smithi113.stdout: "command_resend": 0, 2024-03-28T05:34:10.196 INFO:teuthology.orchestra.run.smithi113.stdout: "map_epoch": 102, 2024-03-28T05:34:10.196 INFO:teuthology.orchestra.run.smithi113.stdout: "map_full": 0, 2024-03-28T05:34:10.196 INFO:teuthology.orchestra.run.smithi113.stdout: "map_inc": 18, 2024-03-28T05:34:10.196 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_sessions": 12, 2024-03-28T05:34:10.196 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_session_open": 12, 2024-03-28T05:34:10.196 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_session_close": 0, 2024-03-28T05:34:10.196 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_laggy": 0, 2024-03-28T05:34:10.196 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_wr": 10470, 2024-03-28T05:34:10.196 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_rd": 0, 2024-03-28T05:34:10.196 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_del": 3018 2024-03-28T05:34:10.196 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.196 INFO:teuthology.orchestra.run.smithi113.stdout: "oft": { 2024-03-28T05:34:10.196 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_total_objs": 1, 2024-03-28T05:34:10.197 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_total_kv_pairs": 0, 2024-03-28T05:34:10.197 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_total_updates": 0, 2024-03-28T05:34:10.197 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_total_removes": 0 2024-03-28T05:34:10.197 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.197 INFO:teuthology.orchestra.run.smithi113.stdout: "osd-slow-ops": { 2024-03-28T05:34:10.197 INFO:teuthology.orchestra.run.smithi113.stdout: "slow_ops_count": 10 2024-03-28T05:34:10.197 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.197 INFO:teuthology.orchestra.run.smithi113.stdout: "purge_queue": { 2024-03-28T05:34:10.197 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executing_ops": 0, 2024-03-28T05:34:10.197 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executing_ops_high_water": 49, 2024-03-28T05:34:10.197 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executing": 0, 2024-03-28T05:34:10.197 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executing_high_water": 26, 2024-03-28T05:34:10.197 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executed_ops": 35895, 2024-03-28T05:34:10.197 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executed": 17951, 2024-03-28T05:34:10.197 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_item_in_journal": 0 2024-03-28T05:34:10.197 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.198 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T05:34:10.198 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-28T05:34:10.198 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 104857600, 2024-03-28T05:34:10.198 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-28T05:34:10.198 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 704316, 2024-03-28T05:34:10.198 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 148789142, 2024-03-28T05:34:10.198 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-28T05:34:10.198 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 704316, 2024-03-28T05:34:10.198 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 0, 2024-03-28T05:34:10.198 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 0, 2024-03-28T05:34:10.198 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 704316, 2024-03-28T05:34:10.198 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 148789142, 2024-03-28T05:34:10.198 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-28T05:34:10.198 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:10.198 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:10.199 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:10.199 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-28T05:34:10.199 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.199 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-objecter_bytes": { 2024-03-28T05:34:10.199 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-28T05:34:10.199 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 104857600, 2024-03-28T05:34:10.199 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-28T05:34:10.199 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 0, 2024-03-28T05:34:10.199 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 0, 2024-03-28T05:34:10.199 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-28T05:34:10.199 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 0, 2024-03-28T05:34:10.199 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 103915, 2024-03-28T05:34:10.199 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 386224856, 2024-03-28T05:34:10.199 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 103915, 2024-03-28T05:34:10.199 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 386224856, 2024-03-28T05:34:10.199 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-28T05:34:10.200 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:10.200 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:10.200 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:10.200 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-28T05:34:10.200 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.200 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-objecter_ops": { 2024-03-28T05:34:10.200 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-28T05:34:10.200 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 1024, 2024-03-28T05:34:10.200 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-28T05:34:10.200 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 0, 2024-03-28T05:34:10.200 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 0, 2024-03-28T05:34:10.200 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-28T05:34:10.200 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 0, 2024-03-28T05:34:10.200 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 103915, 2024-03-28T05:34:10.200 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 103915, 2024-03-28T05:34:10.200 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 103915, 2024-03-28T05:34:10.200 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 103915, 2024-03-28T05:34:10.201 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-28T05:34:10.201 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:10.201 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:10.201 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:10.201 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-28T05:34:10.201 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.201 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-write_buf_throttle": { 2024-03-28T05:34:10.201 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-28T05:34:10.201 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 3758096384, 2024-03-28T05:34:10.201 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-28T05:34:10.201 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 17951, 2024-03-28T05:34:10.201 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 2247419, 2024-03-28T05:34:10.201 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-28T05:34:10.201 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 17951, 2024-03-28T05:34:10.201 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 0, 2024-03-28T05:34:10.201 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 0, 2024-03-28T05:34:10.202 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 413, 2024-03-28T05:34:10.202 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 2247419, 2024-03-28T05:34:10.202 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-28T05:34:10.202 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:10.202 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:10.202 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:10.202 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-28T05:34:10.202 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:10.202 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-write_buf_throttle-0x564392410500": { 2024-03-28T05:34:10.202 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 60, 2024-03-28T05:34:10.202 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 3758096384, 2024-03-28T05:34:10.202 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-28T05:34:10.202 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 97202, 2024-03-28T05:34:10.202 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 327987474, 2024-03-28T05:34:10.202 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-28T05:34:10.203 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 97202, 2024-03-28T05:34:10.203 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 0, 2024-03-28T05:34:10.203 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 0, 2024-03-28T05:34:10.203 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 23723, 2024-03-28T05:34:10.203 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 327987414, 2024-03-28T05:34:10.203 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-28T05:34:10.203 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:10.203 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:10.203 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:10.203 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-28T05:34:10.203 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-28T05:34:10.203 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-28T05:34:10.856 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-03-28T05:34:10.856 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=11 2024-03-28T05:34:10.856 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=5 2024-03-28T05:34:10.856 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=57 2024-03-28T05:34:10.856 DEBUG:tasks.check_counter:Getting stats from l 2024-03-28T05:34:10.857 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-28T05:34:11.107 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T05:34:11.494 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:11 smithi113 ceph-mon[37164]: pgmap v329: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-28T05:34:11.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:11 smithi023 ceph-mon[31954]: pgmap v329: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-28T05:34:11.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:11 smithi098 ceph-mon[36087]: pgmap v329: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-28T05:34:12.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:12 smithi023 ceph-mon[31954]: osdmap e103: 12 total, 12 up, 12 in 2024-03-28T05:34:12.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:12 smithi098 ceph-mon[36087]: osdmap e103: 12 total, 12 up, 12 in 2024-03-28T05:34:12.633 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:12 smithi113 ceph-mon[37164]: osdmap e103: 12 total, 12 up, 12 in 2024-03-28T05:34:13.138 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-28T05:34:13.138 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T05:34:13.138 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 0, 2024-03-28T05:34:13.138 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 2, 2024-03-28T05:34:13.138 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 0, 2024-03-28T05:34:13.138 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 0, 2024-03-28T05:34:13.138 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 4, 2024-03-28T05:34:13.138 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 0, 2024-03-28T05:34:13.138 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 0.001106451, 2024-03-28T05:34:13.138 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 0.000000000, 2024-03-28T05:34:13.138 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.000323753, 2024-03-28T05:34:13.139 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T05:34:13.139 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:34:13.139 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:13.139 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:13.139 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:13.139 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:13.139 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:34:13.139 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:13.139 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:13.139 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:13.139 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:13.139 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T05:34:13.139 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T05:34:13.140 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:13.140 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T05:34:13.140 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 6, 2024-03-28T05:34:13.140 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 170, 2024-03-28T05:34:13.140 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 3312, 2024-03-28T05:34:13.140 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 185136, 2024-03-28T05:34:13.140 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 5, 2024-03-28T05:34:13.140 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 3, 2024-03-28T05:34:13.140 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 0.007850551, 2024-03-28T05:34:13.140 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 0.004558291, 2024-03-28T05:34:13.140 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.001723538, 2024-03-28T05:34:13.140 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T05:34:13.140 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:34:13.140 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 168, 2024-03-28T05:34:13.141 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.017149256, 2024-03-28T05:34:13.141 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000102078 2024-03-28T05:34:13.141 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:13.141 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:34:13.141 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:13.141 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:13.141 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:13.141 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:13.141 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-28T05:34:13.141 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 185136 2024-03-28T05:34:13.141 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:13.141 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T05:34:13.141 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 232, 2024-03-28T05:34:13.141 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 221, 2024-03-28T05:34:13.142 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 1577520, 2024-03-28T05:34:13.142 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 132432, 2024-03-28T05:34:13.142 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 2, 2024-03-28T05:34:13.142 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 1, 2024-03-28T05:34:13.142 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 0.036292220, 2024-03-28T05:34:13.142 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 0.008787431, 2024-03-28T05:34:13.142 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.033970087, 2024-03-28T05:34:13.142 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 0.004048492, 2024-03-28T05:34:13.142 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T05:34:13.142 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 220, 2024-03-28T05:34:13.142 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.006710019, 2024-03-28T05:34:13.142 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000030500 2024-03-28T05:34:13.142 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:13.142 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-28T05:34:13.143 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:13.143 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:13.143 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:13.143 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:13.143 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 1577520, 2024-03-28T05:34:13.143 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 132432 2024-03-28T05:34:13.143 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:13.143 INFO:teuthology.orchestra.run.smithi113.stdout: "cct": { 2024-03-28T05:34:13.143 INFO:teuthology.orchestra.run.smithi113.stdout: "total_workers": 0, 2024-03-28T05:34:13.143 INFO:teuthology.orchestra.run.smithi113.stdout: "unhealthy_workers": 0 2024-03-28T05:34:13.143 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:13.143 INFO:teuthology.orchestra.run.smithi113.stdout: "mempool": { 2024-03-28T05:34:13.143 INFO:teuthology.orchestra.run.smithi113.stdout: "bloom_filter_bytes": 0, 2024-03-28T05:34:13.143 INFO:teuthology.orchestra.run.smithi113.stdout: "bloom_filter_items": 0, 2024-03-28T05:34:13.143 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T05:34:13.144 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_alloc_items": 0, 2024-03-28T05:34:13.144 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T05:34:13.144 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_data_items": 0, 2024-03-28T05:34:13.144 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T05:34:13.144 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T05:34:13.144 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T05:34:13.144 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T05:34:13.144 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T05:34:13.144 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_other_items": 0, 2024-03-28T05:34:13.144 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T05:34:13.144 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T05:34:13.144 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_extent_bytes": 0, 2024-03-28T05:34:13.144 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_extent_items": 0, 2024-03-28T05:34:13.144 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_blob_bytes": 0, 2024-03-28T05:34:13.145 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_blob_items": 0, 2024-03-28T05:34:13.145 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T05:34:13.145 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T05:34:13.145 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T05:34:13.145 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T05:34:13.145 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T05:34:13.145 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_fsck_items": 0, 2024-03-28T05:34:13.145 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_txc_bytes": 0, 2024-03-28T05:34:13.145 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_txc_items": 0, 2024-03-28T05:34:13.145 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T05:34:13.145 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T05:34:13.145 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_bytes": 0, 2024-03-28T05:34:13.145 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_items": 0, 2024-03-28T05:34:13.145 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_bytes": 0, 2024-03-28T05:34:13.146 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_items": 0, 2024-03-28T05:34:13.146 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T05:34:13.146 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_reader_items": 0, 2024-03-28T05:34:13.146 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T05:34:13.146 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_writer_items": 0, 2024-03-28T05:34:13.146 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_anon_bytes": 148754, 2024-03-28T05:34:13.146 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_anon_items": 31, 2024-03-28T05:34:13.146 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_meta_bytes": 0, 2024-03-28T05:34:13.146 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_meta_items": 0, 2024-03-28T05:34:13.146 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_bytes": 0, 2024-03-28T05:34:13.146 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_items": 0, 2024-03-28T05:34:13.146 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_mapbl_bytes": 0, 2024-03-28T05:34:13.146 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_mapbl_items": 0, 2024-03-28T05:34:13.146 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_pglog_bytes": 0, 2024-03-28T05:34:13.147 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_pglog_items": 0, 2024-03-28T05:34:13.147 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_bytes": 0, 2024-03-28T05:34:13.147 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_items": 0, 2024-03-28T05:34:13.147 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T05:34:13.147 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_mapping_items": 0, 2024-03-28T05:34:13.147 INFO:teuthology.orchestra.run.smithi113.stdout: "pgmap_bytes": 0, 2024-03-28T05:34:13.147 INFO:teuthology.orchestra.run.smithi113.stdout: "pgmap_items": 0, 2024-03-28T05:34:13.147 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_co_bytes": 928, 2024-03-28T05:34:13.147 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_co_items": 2, 2024-03-28T05:34:13.147 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_1_bytes": 0, 2024-03-28T05:34:13.147 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_1_items": 0, 2024-03-28T05:34:13.147 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_2_bytes": 0, 2024-03-28T05:34:13.147 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_2_items": 0 2024-03-28T05:34:13.147 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-28T05:34:13.147 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T05:34:13.148 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-28T05:34:13.148 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 104857600, 2024-03-28T05:34:13.148 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-28T05:34:13.148 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 234, 2024-03-28T05:34:13.148 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 1544817, 2024-03-28T05:34:13.148 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-28T05:34:13.148 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 234, 2024-03-28T05:34:13.148 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 0, 2024-03-28T05:34:13.148 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 0, 2024-03-28T05:34:13.148 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 234, 2024-03-28T05:34:13.148 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 1544817, 2024-03-28T05:34:13.148 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-28T05:34:13.148 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-28T05:34:13.149 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-28T05:34:13.149 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-28T05:34:13.149 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-28T05:34:13.149 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-28T05:34:13.149 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-28T05:34:13.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:13 smithi023 ceph-mon[31954]: pgmap v331: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-03-28T05:34:13.560 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:13 smithi113 ceph-mon[37164]: pgmap v331: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-03-28T05:34:13.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:13 smithi098 ceph-mon[36087]: pgmap v331: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-03-28T05:34:13.897 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-03-28T05:34:13.897 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-03-28T05:34:13.897 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-03-28T05:34:13.897 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2024-03-28T05:34:13.898 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-28T05:34:13.912 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-28T05:34:13.912 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T05:34:13.912 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T05:34:13.939 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T05:34:13.939 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T05:34:14.015 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T05:34:14.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.231+0000 7fd2e4fab640 1 -- 172.21.15.23:0/2824726602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2e010a160 msgr2=0x7fd2e010a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:14.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.231+0000 7fd2e4fab640 1 --2- 172.21.15.23:0/2824726602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2e010a160 0x7fd2e010a5e0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0009890 tx=0x7fd2d002bc30 comp rx=0 tx=0).stop 2024-03-28T05:34:14.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.232+0000 7fd2e4fab640 1 -- 172.21.15.23:0/2824726602 shutdown_connections 2024-03-28T05:34:14.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.232+0000 7fd2e4fab640 1 --2- 172.21.15.23:0/2824726602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2e010a160 0x7fd2e010a5e0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:14.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.232+0000 7fd2e4fab640 1 --2- 172.21.15.23:0/2824726602 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2e01097a0 0x7fd2e0109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:14.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.232+0000 7fd2e4fab640 1 --2- 172.21.15.23:0/2824726602 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e0108df0 0x7fd2e01091d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:14.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.232+0000 7fd2e4fab640 1 -- 172.21.15.23:0/2824726602 >> 172.21.15.23:0/2824726602 conn(0x7fd2e006cbf0 msgr2=0x7fd2e006d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:14.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.235+0000 7fd2e4fab640 1 -- 172.21.15.23:0/2824726602 shutdown_connections 2024-03-28T05:34:14.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.235+0000 7fd2e4fab640 1 -- 172.21.15.23:0/2824726602 wait complete. 2024-03-28T05:34:14.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.236+0000 7fd2e4fab640 1 Processor -- start 2024-03-28T05:34:14.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.240+0000 7fd2e4fab640 1 -- start start 2024-03-28T05:34:14.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.240+0000 7fd2e4fab640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2e0108df0 0x7fd2e0118fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:14.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.240+0000 7fd2e4fab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e01097a0 0x7fd2e0116080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:14.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.240+0000 7fd2e4fab640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2e010a160 0x7fd2e01165c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:14.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.240+0000 7fd2e4fab640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e011c460 con 0x7fd2e010a160 2024-03-28T05:34:14.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.240+0000 7fd2e4fab640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e011c2e0 con 0x7fd2e01097a0 2024-03-28T05:34:14.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.240+0000 7fd2e4fab640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e011c5e0 con 0x7fd2e0108df0 2024-03-28T05:34:14.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.241+0000 7fd2ded9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2e010a160 0x7fd2e01165c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:14.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.241+0000 7fd2ded9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2e010a160 0x7fd2e01165c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41532/0 (socket says 172.21.15.23:41532) 2024-03-28T05:34:14.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.241+0000 7fd2ded9d640 1 -- 172.21.15.23:0/4043736528 learned_addr learned my addr 172.21.15.23:0/4043736528 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:34:14.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.241+0000 7fd2ddd9b640 1 --2- 172.21.15.23:0/4043736528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e01097a0 0x7fd2e0116080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:14.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.243+0000 7fd2ded9d640 1 -- 172.21.15.23:0/4043736528 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2e0108df0 msgr2=0x7fd2e0118fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:14.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.243+0000 7fd2de59c640 1 --2- 172.21.15.23:0/4043736528 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2e0108df0 0x7fd2e0118fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:14.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.244+0000 7fd2ded9d640 1 --2- 172.21.15.23:0/4043736528 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2e0108df0 0x7fd2e0118fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:14.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.244+0000 7fd2ded9d640 1 -- 172.21.15.23:0/4043736528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e01097a0 msgr2=0x7fd2e0116080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:14.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.244+0000 7fd2ded9d640 1 --2- 172.21.15.23:0/4043736528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e01097a0 0x7fd2e0116080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:14.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.244+0000 7fd2ded9d640 1 -- 172.21.15.23:0/4043736528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e0116e30 con 0x7fd2e010a160 2024-03-28T05:34:14.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.244+0000 7fd2ddd9b640 1 --2- 172.21.15.23:0/4043736528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e01097a0 0x7fd2e0116080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:34:14.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.244+0000 7fd2de59c640 1 --2- 172.21.15.23:0/4043736528 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2e0108df0 0x7fd2e0118fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:34:14.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.244+0000 7fd2ded9d640 1 --2- 172.21.15.23:0/4043736528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2e010a160 0x7fd2e01165c0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd2d002c1c0 tx=0x7fd2d0034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:14.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.246+0000 7fd2cf7fe640 1 -- 172.21.15.23:0/4043736528 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0041030 con 0x7fd2e010a160 2024-03-28T05:34:14.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.246+0000 7fd2cf7fe640 1 -- 172.21.15.23:0/4043736528 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2d002c9f0 con 0x7fd2e010a160 2024-03-28T05:34:14.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.246+0000 7fd2cf7fe640 1 -- 172.21.15.23:0/4043736528 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0039040 con 0x7fd2e010a160 2024-03-28T05:34:14.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.246+0000 7fd2e4fab640 1 -- 172.21.15.23:0/4043736528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e01a1930 con 0x7fd2e010a160 2024-03-28T05:34:14.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.246+0000 7fd2e4fab640 1 -- 172.21.15.23:0/4043736528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e01a1e00 con 0x7fd2e010a160 2024-03-28T05:34:14.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.247+0000 7fd2e4fab640 1 -- 172.21.15.23:0/4043736528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e006dc60 con 0x7fd2e010a160 2024-03-28T05:34:14.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.260+0000 7fd2cf7fe640 1 -- 172.21.15.23:0/4043736528 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fd2d0044070 con 0x7fd2e010a160 2024-03-28T05:34:14.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.261+0000 7fd2cf7fe640 1 --2- 172.21.15.23:0/4043736528 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fd2c80a2b70 0x7fd2c80a5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:14.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.261+0000 7fd2cf7fe640 1 -- 172.21.15.23:0/4043736528 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9613+0+0 (secure 0 0 0) 0x7fd2d00f0e90 con 0x7fd2e010a160 2024-03-28T05:34:14.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.261+0000 7fd2cf7fe640 1 -- 172.21.15.23:0/4043736528 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd2d00f1350 con 0x7fd2e010a160 2024-03-28T05:34:14.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.262+0000 7fd2de59c640 1 --2- 172.21.15.23:0/4043736528 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fd2c80a2b70 0x7fd2c80a5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:14.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.270+0000 7fd2de59c640 1 --2- 172.21.15.23:0/4043736528 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fd2c80a2b70 0x7fd2c80a5030 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8005fd0 tx=0x7fd2d80093e0 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:14.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.618+0000 7fd2e4fab640 1 -- 172.21.15.23:0/4043736528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2e010e170 con 0x7fd2e010a160 2024-03-28T05:34:14.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.621+0000 7fd2cf7fe640 1 -- 172.21.15.23:0/4043736528 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+25389 (secure 0 0 0) 0x7fd2d00bd960 con 0x7fd2e010a160 2024-03-28T05:34:14.622 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:34:14.622 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":103,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","created":"2024-03-28T05:13:13.453653+0000","modified":"2024-03-28T05:34:11.228836+0000","last_up_change":"2024-03-28T05:20:20.221527+0000","last_in_change":"2024-03-28T05:20:00.365478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T05:16:32.986974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T05:21:06.103894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T05:21:06.827190+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T05:21:10.507418+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"18150942-59a4-4c3b-83d4-c1fc3ccf1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6803","nonce":1219956881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6805","nonce":1219956881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6809","nonce":1219956881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6807","nonce":1219956881}]},"public_addr":"172.21.15.23:6803/1219956881","cluster_addr":"172.21.15.23:6805/1219956881","heartbeat_back_addr":"172.21.15.23:6809/1219956881","heartbeat_front_addr":"172.21.15.23:6807/1219956881","state":["exists","up"]},{"osd":1,"uuid":"ba6d1a95-5135-4670-95f9-7878c0e7bd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6801","nonce":555491234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6803","nonce":555491234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6807","nonce":555491234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6805","nonce":555491234}]},"public_addr":"172.21.15.98:6801/555491234","cluster_addr":"172.21.15.98:6803/555491234","heartbeat_back_addr":"172.21.15.98:6807/555491234","heartbeat_front_addr":"172.21.15.98:6805/555491234","state":["exists","up"]},{"osd":2,"uuid":"873bdd60-2a47-403b-b1f4-ee9226a95067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6801","nonce":1535856914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6803","nonce":1535856914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6807","nonce":1535856914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6805","nonce":1535856914}]},"public_addr":"172.21.15.113:6801/1535856914","cluster_addr":"172.21.15.113:6803/1535856914","heartbeat_back_addr":"172.21.15.113:6807/1535856914","heartbeat_front_addr":"172.21.15.113:6805/1535856914","state":["exists","up"]},{"osd":3,"uuid":"83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6811","nonce":2949415020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6813","nonce":2949415020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6817","nonce":2949415020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6815","nonce":2949415020}]},"public_addr":"172.21.15.23:6811/2949415020","cluster_addr":"172.21.15.23:6813/2949415020","heartbeat_back_addr":"172.21.15.23:6817/2949415020","heartbeat_front_addr":"172.21.15.23:6815/2949415020","state":["exists","up"]},{"osd":4,"uuid":"76a90bac-b221-40a3-8238-40e8049cec36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6809","nonce":2615902819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6811","nonce":2615902819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6815","nonce":2615902819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6813","nonce":2615902819}]},"public_addr":"172.21.15.98:6809/2615902819","cluster_addr":"172.21.15.98:6811/2615902819","heartbeat_back_addr":"172.21.15.98:6815/2615902819","heartbeat_front_addr":"172.21.15.98:6813/2615902819","state":["exists","up"]},{"osd":5,"uuid":"b53fdb55-a8df-40a8-bd1b-8bcef634e26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6809","nonce":625271528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6811","nonce":625271528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6815","nonce":625271528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6813","nonce":625271528}]},"public_addr":"172.21.15.113:6809/625271528","cluster_addr":"172.21.15.113:6811/625271528","heartbeat_back_addr":"172.21.15.113:6815/625271528","heartbeat_front_addr":"172.21.15.113:6813/625271528","state":["exists","up"]},{"osd":6,"uuid":"f00283d2-8d67-4e97-a575-9a9d3b55d5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6819","nonce":4188464176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6821","nonce":4188464176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6825","nonce":4188464176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6823","nonce":4188464176}]},"public_addr":"172.21.15.23:6819/4188464176","cluster_addr":"172.21.15.23:6821/4188464176","heartbeat_back_addr":"172.21.15.23:6825/4188464176","heartbeat_front_addr":"172.21.15.23:6823/4188464176","state":["exists","up"]},{"osd":7,"uuid":"edc008a7-b07b-40a8-88b7-1614ba38a775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6817","nonce":1985546028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6819","nonce":1985546028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6823","nonce":1985546028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6821","nonce":1985546028}]},"public_addr":"172.21.15.98:6817/1985546028","cluster_addr":"172.21.15.98:6819/1985546028","heartbeat_back_addr":"172.21.15.98:6823/1985546028","heartbeat_front_addr":"172.21.15.98:6821/1985546028","state":["exists","up"]},{"osd":8,"uuid":"56e7b2d1-81f0-47b2-8638-f29832bed8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6817","nonce":3534876081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6819","nonce":3534876081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6823","nonce":3534876081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6821","nonce":3534876081}]},"public_addr":"172.21.15.113:6817/3534876081","cluster_addr":"172.21.15.113:6819/3534876081","heartbeat_back_addr":"172.21.15.113:6823/3534876081","heartbeat_front_addr":"172.21.15.113:6821/3534876081","state":["exists","up"]},{"osd":9,"uuid":"a2e90caa-6e9c-42bf-9f18-81f7f47880cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6827","nonce":672264207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6829","nonce":672264207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6833","nonce":672264207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6831","nonce":672264207}]},"public_addr":"172.21.15.23:6827/672264207","cluster_addr":"172.21.15.23:6829/672264207","heartbeat_back_addr":"172.21.15.23:6833/672264207","heartbeat_front_addr":"172.21.15.23:6831/672264207","state":["exists","up"]},{"osd":10,"uuid":"0eda917f-020c-437f-95ae-b26203f6c68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6825","nonce":2699920595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6827","nonce":2699920595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6831","nonce":2699920595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6829","nonce":2699920595}]},"public_addr":"172.21.15.98:6825/2699920595","cluster_addr":"172.21.15.98:6827/2699920595","heartbeat_back_addr":"172.21.15.98:6831/2699920595","heartbeat_front_addr":"172.21.15.98:6829/2699920595","state":["exists","up"]},{"osd":11,"uuid":"aae78501-78a8-469a-be20-3006f5ec5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6825","nonce":2113644020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6827","nonce":2113644020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6831","nonce":2113644020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6829","nonce":2113644020}]},"public_addr":"172.21.15.113:6825/2113644020","cluster_addr":"172.21.15.113:6827/2113644020","heartbeat_back_addr":"172.21.15.113:6831/2113644020","heartbeat_front_addr":"172.21.15.113:6829/2113644020","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:15:43.527475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:07.006538+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:30.594176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:55.810999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:20.595811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:45.474138+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:10.500409+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:35.787638+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:01.001188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:26.761115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:52.952752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:20:18.559832+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":7}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2772113246":"2024-03-29T05:23:33.950262+0000","172.21.15.23:0/84692814":"2024-03-29T05:23:33.950262+0000","172.21.15.23:6801/222178108":"2024-03-29T05:23:33.950262+0000","172.21.15.23:6801/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:6800/222178108":"2024-03-29T05:23:33.950262+0000","172.21.15.23:6800/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1876451381":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2253559862":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2653562762":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/3337463830":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/944304909":"2024-03-29T05:23:33.950262+0000","172.21.15.23:6800/1735221490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/2784771635":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1135235867":"2024-03-29T05:23:33.950262+0000","172.21.15.23:0/2389902490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6801/1735221490":"2024-03-29T05:13:42.579607+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T05:34:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.626+0000 7fd2e4fab640 1 -- 172.21.15.23:0/4043736528 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fd2c80a2b70 msgr2=0x7fd2c80a5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.626+0000 7fd2e4fab640 1 --2- 172.21.15.23:0/4043736528 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fd2c80a2b70 0x7fd2c80a5030 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8005fd0 tx=0x7fd2d80093e0 comp rx=0 tx=0).stop 2024-03-28T05:34:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.626+0000 7fd2e4fab640 1 -- 172.21.15.23:0/4043736528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2e010a160 msgr2=0x7fd2e01165c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.626+0000 7fd2e4fab640 1 --2- 172.21.15.23:0/4043736528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2e010a160 0x7fd2e01165c0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd2d002c1c0 tx=0x7fd2d0034670 comp rx=0 tx=0).stop 2024-03-28T05:34:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.627+0000 7fd2e4fab640 1 -- 172.21.15.23:0/4043736528 shutdown_connections 2024-03-28T05:34:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.627+0000 7fd2e4fab640 1 --2- 172.21.15.23:0/4043736528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2e010a160 0x7fd2e01165c0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.627+0000 7fd2e4fab640 1 --2- 172.21.15.23:0/4043736528 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fd2c80a2b70 0x7fd2c80a5030 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.627+0000 7fd2e4fab640 1 --2- 172.21.15.23:0/4043736528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2e01097a0 0x7fd2e0116080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.627+0000 7fd2e4fab640 1 --2- 172.21.15.23:0/4043736528 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2e0108df0 0x7fd2e0118fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.627+0000 7fd2e4fab640 1 -- 172.21.15.23:0/4043736528 >> 172.21.15.23:0/4043736528 conn(0x7fd2e006cbf0 msgr2=0x7fd2e00722a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.627+0000 7fd2e4fab640 1 -- 172.21.15.23:0/4043736528 shutdown_connections 2024-03-28T05:34:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.627+0000 7fd2e4fab640 1 -- 172.21.15.23:0/4043736528 wait complete. 2024-03-28T05:34:14.644 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T05:16:32.986974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T05:21:06.103894+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T05:21:06.827190+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 103, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T05:21:10.507418+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 103, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T05:34:14.644 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T05:34:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.830+0000 7f1829f77640 1 -- 172.21.15.23:0/1776953205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18241065a0 msgr2=0x7f1824106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.830+0000 7f1829f77640 1 --2- 172.21.15.23:0/1776953205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18241065a0 0x7f1824106980 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f181c01acb0 tx=0x7f181c03d1e0 comp rx=0 tx=0).stop 2024-03-28T05:34:14.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.835+0000 7f1829f77640 1 -- 172.21.15.23:0/1776953205 shutdown_connections 2024-03-28T05:34:14.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.835+0000 7f1829f77640 1 --2- 172.21.15.23:0/1776953205 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18241004e0 0x7f1824105290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:14.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.835+0000 7f1829f77640 1 --2- 172.21.15.23:0/1776953205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18240ffb40 0x7f18240fffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:14.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.835+0000 7f1829f77640 1 --2- 172.21.15.23:0/1776953205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18241065a0 0x7f1824106980 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:14.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.835+0000 7f1829f77640 1 -- 172.21.15.23:0/1776953205 >> 172.21.15.23:0/1776953205 conn(0x7f1824076ac0 msgr2=0x7f18240fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:14.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.838+0000 7f1829f77640 1 -- 172.21.15.23:0/1776953205 shutdown_connections 2024-03-28T05:34:14.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.840+0000 7f1829f77640 1 -- 172.21.15.23:0/1776953205 wait complete. 2024-03-28T05:34:14.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.841+0000 7f1829f77640 1 Processor -- start 2024-03-28T05:34:14.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.841+0000 7f1829f77640 1 -- start start 2024-03-28T05:34:14.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.841+0000 7f1829f77640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18240ffb40 0x7f18240721c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:14.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.841+0000 7f1829f77640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18241004e0 0x7f1824072700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:14.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.841+0000 7f1829f77640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18241065a0 0x7f18241a2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:14.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.841+0000 7f1829f77640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1824109840 con 0x7f18241004e0 2024-03-28T05:34:14.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.841+0000 7f1829f77640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18241096c0 con 0x7f18241065a0 2024-03-28T05:34:14.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.841+0000 7f1829f77640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18241099c0 con 0x7f18240ffb40 2024-03-28T05:34:14.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.842+0000 7f1823fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18241065a0 0x7f18241a2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:14.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.842+0000 7f1823fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18241065a0 0x7f18241a2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.23:42122/0 (socket says 172.21.15.23:42122) 2024-03-28T05:34:14.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.842+0000 7f1823fff640 1 -- 172.21.15.23:0/3174649317 learned_addr learned my addr 172.21.15.23:0/3174649317 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:34:14.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.842+0000 7f18237fe640 1 --2- 172.21.15.23:0/3174649317 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18240ffb40 0x7f18240721c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:14.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.842+0000 7f1822ffd640 1 --2- 172.21.15.23:0/3174649317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18241004e0 0x7f1824072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:14.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.843+0000 7f18237fe640 1 -- 172.21.15.23:0/3174649317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18241065a0 msgr2=0x7f18241a2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:14.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.843+0000 7f18237fe640 1 --2- 172.21.15.23:0/3174649317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18241065a0 0x7f18241a2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:14.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.843+0000 7f18237fe640 1 -- 172.21.15.23:0/3174649317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18241004e0 msgr2=0x7f1824072700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:14.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.843+0000 7f18237fe640 1 --2- 172.21.15.23:0/3174649317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18241004e0 0x7f1824072700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:14.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.843+0000 7f18237fe640 1 -- 172.21.15.23:0/3174649317 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18241a35c0 con 0x7f18240ffb40 2024-03-28T05:34:14.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.845+0000 7f18237fe640 1 --2- 172.21.15.23:0/3174649317 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18240ffb40 0x7f18240721c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f181c03d7f0 tx=0x7f181c0045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:14.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.847+0000 7f1820ff9640 1 -- 172.21.15.23:0/3174649317 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c052020 con 0x7f18240ffb40 2024-03-28T05:34:14.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.848+0000 7f1820ff9640 1 -- 172.21.15.23:0/3174649317 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f181c004750 con 0x7f18240ffb40 2024-03-28T05:34:14.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.848+0000 7f1829f77640 1 -- 172.21.15.23:0/3174649317 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18241a3850 con 0x7f18240ffb40 2024-03-28T05:34:14.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.848+0000 7f1829f77640 1 -- 172.21.15.23:0/3174649317 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18241ab180 con 0x7f18240ffb40 2024-03-28T05:34:14.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.849+0000 7f1820ff9640 1 -- 172.21.15.23:0/3174649317 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c04a040 con 0x7f18240ffb40 2024-03-28T05:34:14.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.851+0000 7f1820ff9640 1 -- 172.21.15.23:0/3174649317 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f181c055020 con 0x7f18240ffb40 2024-03-28T05:34:14.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.852+0000 7f1820ff9640 1 --2- 172.21.15.23:0/3174649317 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f180c0a2b70 0x7f180c0a5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:14.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.853+0000 7f1820ff9640 1 -- 172.21.15.23:0/3174649317 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9613+0+0 (secure 0 0 0) 0x7f181c1029a0 con 0x7f18240ffb40 2024-03-28T05:34:14.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.853+0000 7f1822ffd640 1 --2- 172.21.15.23:0/3174649317 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f180c0a2b70 0x7f180c0a5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:14.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.853+0000 7f1829f77640 1 -- 172.21.15.23:0/3174649317 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1810005290 con 0x7f18240ffb40 2024-03-28T05:34:14.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.854+0000 7f1822ffd640 1 --2- 172.21.15.23:0/3174649317 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f180c0a2b70 0x7f180c0a5030 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f18241047f0 tx=0x7f181800a2d0 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:14.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:14.867+0000 7f1820ff9640 1 -- 172.21.15.23:0/3174649317 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f181c0cf370 con 0x7f18240ffb40 2024-03-28T05:34:15.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.237+0000 7f1829f77640 1 -- 172.21.15.23:0/3174649317 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1810005850 con 0x7f18240ffb40 2024-03-28T05:34:15.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.238+0000 7f1820ff9640 1 -- 172.21.15.23:0/3174649317 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) v1 ==== 93+0+10 (secure 0 0 0) 0x7f181c0cec70 con 0x7f18240ffb40 2024-03-28T05:34:15.238 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-28T05:34:15.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.242+0000 7f1829f77640 1 -- 172.21.15.23:0/3174649317 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f180c0a2b70 msgr2=0x7f180c0a5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:15.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.242+0000 7f1829f77640 1 --2- 172.21.15.23:0/3174649317 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f180c0a2b70 0x7f180c0a5030 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f18241047f0 tx=0x7f181800a2d0 comp rx=0 tx=0).stop 2024-03-28T05:34:15.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.242+0000 7f1829f77640 1 -- 172.21.15.23:0/3174649317 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18240ffb40 msgr2=0x7f18240721c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:15.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.242+0000 7f1829f77640 1 --2- 172.21.15.23:0/3174649317 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18240ffb40 0x7f18240721c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f181c03d7f0 tx=0x7f181c0045e0 comp rx=0 tx=0).stop 2024-03-28T05:34:15.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.242+0000 7f1829f77640 1 -- 172.21.15.23:0/3174649317 shutdown_connections 2024-03-28T05:34:15.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.242+0000 7f1829f77640 1 --2- 172.21.15.23:0/3174649317 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f180c0a2b70 0x7f180c0a5030 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:15.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.242+0000 7f1829f77640 1 --2- 172.21.15.23:0/3174649317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18241065a0 0x7f18241a2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:15.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.242+0000 7f1829f77640 1 --2- 172.21.15.23:0/3174649317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18241004e0 0x7f1824072700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:15.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.242+0000 7f1829f77640 1 --2- 172.21.15.23:0/3174649317 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18240ffb40 0x7f18240721c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:15.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.242+0000 7f1829f77640 1 -- 172.21.15.23:0/3174649317 >> 172.21.15.23:0/3174649317 conn(0x7f1824076ac0 msgr2=0x7f18240fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:15.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.243+0000 7f1829f77640 1 -- 172.21.15.23:0/3174649317 shutdown_connections 2024-03-28T05:34:15.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.243+0000 7f1829f77640 1 -- 172.21.15.23:0/3174649317 wait complete. 2024-03-28T05:34:15.262 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T05:34:15.368 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:15 smithi023 ceph-mon[31954]: pgmap v332: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-03-28T05:34:15.368 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:15 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/4043736528' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:34:15.368 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:15 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3174649317' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:34:15.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.461+0000 7f24659df640 1 -- 172.21.15.23:0/4009425553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2460100f00 msgr2=0x7f246010d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:15.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.461+0000 7f24659df640 1 --2- 172.21.15.23:0/4009425553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2460100f00 0x7f246010d410 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f245800bc00 tx=0x7f2458009260 comp rx=0 tx=0).stop 2024-03-28T05:34:15.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.465+0000 7f24659df640 1 -- 172.21.15.23:0/4009425553 shutdown_connections 2024-03-28T05:34:15.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.465+0000 7f24659df640 1 --2- 172.21.15.23:0/4009425553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2460100f00 0x7f246010d410 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:15.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.465+0000 7f24659df640 1 --2- 172.21.15.23:0/4009425553 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2460100540 0x7f24601009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:15.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.465+0000 7f24659df640 1 --2- 172.21.15.23:0/4009425553 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2460105560 0x7f2460105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:15.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.465+0000 7f24659df640 1 -- 172.21.15.23:0/4009425553 >> 172.21.15.23:0/4009425553 conn(0x7f2460077630 msgr2=0x7f2460077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:15.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.468+0000 7f24659df640 1 -- 172.21.15.23:0/4009425553 shutdown_connections 2024-03-28T05:34:15.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.471+0000 7f24659df640 1 -- 172.21.15.23:0/4009425553 wait complete. 2024-03-28T05:34:15.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.472+0000 7f24659df640 1 Processor -- start 2024-03-28T05:34:15.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.474+0000 7f24659df640 1 -- start start 2024-03-28T05:34:15.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.474+0000 7f24659df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2460100540 0x7f2460073360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:15.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.474+0000 7f24659df640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2460100f00 0x7f246006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:15.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.474+0000 7f24659df640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2460105560 0x7f246006db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:15.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.474+0000 7f24659df640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246006bf80 con 0x7f2460100f00 2024-03-28T05:34:15.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.474+0000 7f24659df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246006be00 con 0x7f2460100540 2024-03-28T05:34:15.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.474+0000 7f24659df640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246006c100 con 0x7f2460105560 2024-03-28T05:34:15.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.475+0000 7f245effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2460100540 0x7f2460073360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:15.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.475+0000 7f245effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2460100540 0x7f2460073360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.23:42144/0 (socket says 172.21.15.23:42144) 2024-03-28T05:34:15.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.475+0000 7f245effd640 1 -- 172.21.15.23:0/3751056282 learned_addr learned my addr 172.21.15.23:0/3751056282 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:34:15.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.477+0000 7f245f7fe640 1 --2- 172.21.15.23:0/3751056282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2460105560 0x7f246006db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:15.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.479+0000 7f245e7fc640 1 --2- 172.21.15.23:0/3751056282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2460100f00 0x7f246006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:15.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.479+0000 7f245e7fc640 1 -- 172.21.15.23:0/3751056282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2460105560 msgr2=0x7f246006db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:15.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.479+0000 7f245e7fc640 1 --2- 172.21.15.23:0/3751056282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2460105560 0x7f246006db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:15.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.479+0000 7f245e7fc640 1 -- 172.21.15.23:0/3751056282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2460100540 msgr2=0x7f2460073360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:15.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.479+0000 7f245e7fc640 1 --2- 172.21.15.23:0/3751056282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2460100540 0x7f2460073360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:15.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.479+0000 7f245e7fc640 1 -- 172.21.15.23:0/3751056282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24601aa6f0 con 0x7f2460100f00 2024-03-28T05:34:15.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.480+0000 7f245e7fc640 1 --2- 172.21.15.23:0/3751056282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2460100f00 0x7f246006d3d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f2450009800 tx=0x7f2450009cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:15.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.482+0000 7f24649dd640 1 -- 172.21.15.23:0/3751056282 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245001c070 con 0x7f2460100f00 2024-03-28T05:34:15.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.482+0000 7f24649dd640 1 -- 172.21.15.23:0/3751056282 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2450017410 con 0x7f2460100f00 2024-03-28T05:34:15.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.482+0000 7f24649dd640 1 -- 172.21.15.23:0/3751056282 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2450015570 con 0x7f2460100f00 2024-03-28T05:34:15.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.482+0000 7f24659df640 1 -- 172.21.15.23:0/3751056282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24601aa9e0 con 0x7f2460100f00 2024-03-28T05:34:15.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.482+0000 7f24659df640 1 -- 172.21.15.23:0/3751056282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24601aaee0 con 0x7f2460100f00 2024-03-28T05:34:15.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.486+0000 7f24659df640 1 -- 172.21.15.23:0/3751056282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2460101c50 con 0x7f2460100f00 2024-03-28T05:34:15.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.489+0000 7f24649dd640 1 -- 172.21.15.23:0/3751056282 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f24500175c0 con 0x7f2460100f00 2024-03-28T05:34:15.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.489+0000 7f24649dd640 1 --2- 172.21.15.23:0/3751056282 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f24480a2b80 0x7f24480a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:15.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.490+0000 7f24649dd640 1 -- 172.21.15.23:0/3751056282 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9613+0+0 (secure 0 0 0) 0x7f24500cef00 con 0x7f2460100f00 2024-03-28T05:34:15.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.494+0000 7f245effd640 1 --2- 172.21.15.23:0/3751056282 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f24480a2b80 0x7f24480a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:15.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.499+0000 7f24649dd640 1 -- 172.21.15.23:0/3751056282 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f245009b850 con 0x7f2460100f00 2024-03-28T05:34:15.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.503+0000 7f245effd640 1 --2- 172.21.15.23:0/3751056282 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f24480a2b80 0x7f24480a5040 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f245400ab50 tx=0x7f245400a380 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:15.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:15 smithi098 ceph-mon[36087]: pgmap v332: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-03-28T05:34:15.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:15 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/4043736528' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:34:15.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:15 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3174649317' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:34:15.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:15 smithi113 ceph-mon[37164]: pgmap v332: 129 pgs: 129 active+clean; 236 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-03-28T05:34:15.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:15 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/4043736528' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:34:15.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:15 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3174649317' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T05:34:15.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.854+0000 7f24659df640 1 -- 172.21.15.23:0/3751056282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f24601ab200 con 0x7f2460100f00 2024-03-28T05:34:15.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.856+0000 7f24649dd640 1 -- 172.21.15.23:0/3751056282 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v103) v1 ==== 104+0+11 (secure 0 0 0) 0x7f245009b150 con 0x7f2460100f00 2024-03-28T05:34:15.915 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T05:34:15.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.860+0000 7f24659df640 1 -- 172.21.15.23:0/3751056282 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f24480a2b80 msgr2=0x7f24480a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:15.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.860+0000 7f24659df640 1 --2- 172.21.15.23:0/3751056282 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f24480a2b80 0x7f24480a5040 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f245400ab50 tx=0x7f245400a380 comp rx=0 tx=0).stop 2024-03-28T05:34:15.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.860+0000 7f24659df640 1 -- 172.21.15.23:0/3751056282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2460100f00 msgr2=0x7f246006d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:15.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.860+0000 7f24659df640 1 --2- 172.21.15.23:0/3751056282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2460100f00 0x7f246006d3d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f2450009800 tx=0x7f2450009cc0 comp rx=0 tx=0).stop 2024-03-28T05:34:15.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.860+0000 7f24659df640 1 -- 172.21.15.23:0/3751056282 shutdown_connections 2024-03-28T05:34:15.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.860+0000 7f24659df640 1 --2- 172.21.15.23:0/3751056282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2460105560 0x7f246006db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:15.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.860+0000 7f24659df640 1 --2- 172.21.15.23:0/3751056282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2460100f00 0x7f246006d3d0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:15.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.860+0000 7f24659df640 1 --2- 172.21.15.23:0/3751056282 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f24480a2b80 0x7f24480a5040 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f245400ab50 tx=0x7f245400a380 comp rx=0 tx=0).stop 2024-03-28T05:34:15.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.860+0000 7f24659df640 1 --2- 172.21.15.23:0/3751056282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2460100540 0x7f2460073360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:15.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.860+0000 7f24659df640 1 -- 172.21.15.23:0/3751056282 >> 172.21.15.23:0/3751056282 conn(0x7f2460077630 msgr2=0x7f2460102fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:15.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.861+0000 7f24659df640 1 -- 172.21.15.23:0/3751056282 shutdown_connections 2024-03-28T05:34:15.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:15.861+0000 7f24659df640 1 -- 172.21.15.23:0/3751056282 wait complete. 2024-03-28T05:34:15.918 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T05:34:16.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.106+0000 7fb980d22640 1 -- 172.21.15.23:0/2456038742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c10fec0 msgr2=0x7fb97c10d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:16.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.106+0000 7fb980d22640 1 --2- 172.21.15.23:0/2456038742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c10fec0 0x7fb97c10d9b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb96c009900 tx=0x7fb96c02bc40 comp rx=0 tx=0).stop 2024-03-28T05:34:16.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.108+0000 7fb980d22640 1 -- 172.21.15.23:0/2456038742 shutdown_connections 2024-03-28T05:34:16.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.108+0000 7fb980d22640 1 --2- 172.21.15.23:0/2456038742 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb97c11ae80 0x7fb97c11d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:16.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.108+0000 7fb980d22640 1 --2- 172.21.15.23:0/2456038742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb97c10def0 0x7fb97c10e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:16.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.108+0000 7fb980d22640 1 --2- 172.21.15.23:0/2456038742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c10fec0 0x7fb97c10d9b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:16.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.108+0000 7fb980d22640 1 -- 172.21.15.23:0/2456038742 >> 172.21.15.23:0/2456038742 conn(0x7fb97c06c780 msgr2=0x7fb97c06cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:16.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.111+0000 7fb980d22640 1 -- 172.21.15.23:0/2456038742 shutdown_connections 2024-03-28T05:34:16.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.111+0000 7fb980d22640 1 -- 172.21.15.23:0/2456038742 wait complete. 2024-03-28T05:34:16.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.112+0000 7fb980d22640 1 Processor -- start 2024-03-28T05:34:16.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.113+0000 7fb980d22640 1 -- start start 2024-03-28T05:34:16.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.114+0000 7fb980d22640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb97c10def0 0x7fb97c11a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:16.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.114+0000 7fb980d22640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c10fec0 0x7fb97c1146c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:16.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.114+0000 7fb980d22640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb97c11ae80 0x7fb97c114c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:16.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.114+0000 7fb980d22640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c1110b0 con 0x7fb97c11ae80 2024-03-28T05:34:16.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.114+0000 7fb980d22640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c110f30 con 0x7fb97c10fec0 2024-03-28T05:34:16.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.114+0000 7fb980d22640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c111230 con 0x7fb97c10def0 2024-03-28T05:34:16.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.115+0000 7fb97a59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb97c10def0 0x7fb97c11a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:16.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.115+0000 7fb97ad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb97c11ae80 0x7fb97c114c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:16.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.115+0000 7fb97ad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb97c11ae80 0x7fb97c114c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41578/0 (socket says 172.21.15.23:41578) 2024-03-28T05:34:16.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.115+0000 7fb97ad9d640 1 -- 172.21.15.23:0/3518253074 learned_addr learned my addr 172.21.15.23:0/3518253074 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:34:16.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.115+0000 7fb979d9b640 1 --2- 172.21.15.23:0/3518253074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c10fec0 0x7fb97c1146c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:16.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.115+0000 7fb97a59c640 1 -- 172.21.15.23:0/3518253074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c10fec0 msgr2=0x7fb97c1146c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:16.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.115+0000 7fb97a59c640 1 --2- 172.21.15.23:0/3518253074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c10fec0 0x7fb97c1146c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:16.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.115+0000 7fb97a59c640 1 -- 172.21.15.23:0/3518253074 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb97c11ae80 msgr2=0x7fb97c114c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:16.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.115+0000 7fb97a59c640 1 --2- 172.21.15.23:0/3518253074 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb97c11ae80 0x7fb97c114c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:16.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.115+0000 7fb97a59c640 1 -- 172.21.15.23:0/3518253074 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97c115380 con 0x7fb97c10def0 2024-03-28T05:34:16.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.119+0000 7fb97a59c640 1 --2- 172.21.15.23:0/3518253074 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb97c10def0 0x7fb97c11a600 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb96c009850 tx=0x7fb96c0043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:16.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.123+0000 7fb96b7fe640 1 -- 172.21.15.23:0/3518253074 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96c041070 con 0x7fb97c10def0 2024-03-28T05:34:16.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.123+0000 7fb96b7fe640 1 -- 172.21.15.23:0/3518253074 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb96c02cca0 con 0x7fb97c10def0 2024-03-28T05:34:16.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.123+0000 7fb96b7fe640 1 -- 172.21.15.23:0/3518253074 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96c003d80 con 0x7fb97c10def0 2024-03-28T05:34:16.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.125+0000 7fb980d22640 1 -- 172.21.15.23:0/3518253074 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb97c1bbb70 con 0x7fb97c10def0 2024-03-28T05:34:16.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.125+0000 7fb980d22640 1 -- 172.21.15.23:0/3518253074 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb97c1bbf20 con 0x7fb97c10def0 2024-03-28T05:34:16.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.126+0000 7fb980d22640 1 -- 172.21.15.23:0/3518253074 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb97c06da20 con 0x7fb97c10def0 2024-03-28T05:34:16.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.130+0000 7fb96b7fe640 1 -- 172.21.15.23:0/3518253074 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fb96c044070 con 0x7fb97c10def0 2024-03-28T05:34:16.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.130+0000 7fb96b7fe640 1 --2- 172.21.15.23:0/3518253074 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fb9640a2b70 0x7fb9640a5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:16.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.131+0000 7fb96b7fe640 1 -- 172.21.15.23:0/3518253074 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9613+0+0 (secure 0 0 0) 0x7fb96c0f1560 con 0x7fb97c10def0 2024-03-28T05:34:16.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.131+0000 7fb979d9b640 1 --2- 172.21.15.23:0/3518253074 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fb9640a2b70 0x7fb9640a5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:16.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.136+0000 7fb96b7fe640 1 -- 172.21.15.23:0/3518253074 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb96c0bdfb0 con 0x7fb97c10def0 2024-03-28T05:34:16.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.137+0000 7fb979d9b640 1 --2- 172.21.15.23:0/3518253074 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fb9640a2b70 0x7fb9640a5030 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb97c115e30 tx=0x7fb970005f90 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:16.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.493+0000 7fb980d22640 1 -- 172.21.15.23:0/3518253074 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb97c068490 con 0x7fb97c10def0 2024-03-28T05:34:16.495 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:16 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3751056282' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:34:16.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.495+0000 7fb96b7fe640 1 -- 172.21.15.23:0/3518253074 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v103) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb96c0bd8b0 con 0x7fb97c10def0 2024-03-28T05:34:16.497 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-28T05:34:16.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.501+0000 7fb980d22640 1 -- 172.21.15.23:0/3518253074 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fb9640a2b70 msgr2=0x7fb9640a5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:16.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.501+0000 7fb980d22640 1 --2- 172.21.15.23:0/3518253074 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fb9640a2b70 0x7fb9640a5030 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb97c115e30 tx=0x7fb970005f90 comp rx=0 tx=0).stop 2024-03-28T05:34:16.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.501+0000 7fb980d22640 1 -- 172.21.15.23:0/3518253074 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb97c10def0 msgr2=0x7fb97c11a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:16.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.501+0000 7fb980d22640 1 --2- 172.21.15.23:0/3518253074 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb97c10def0 0x7fb97c11a600 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb96c009850 tx=0x7fb96c0043b0 comp rx=0 tx=0).stop 2024-03-28T05:34:16.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.502+0000 7fb980d22640 1 -- 172.21.15.23:0/3518253074 shutdown_connections 2024-03-28T05:34:16.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.502+0000 7fb980d22640 1 --2- 172.21.15.23:0/3518253074 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb97c11ae80 0x7fb97c114c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:16.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.502+0000 7fb980d22640 1 --2- 172.21.15.23:0/3518253074 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fb9640a2b70 0x7fb9640a5030 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:16.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.502+0000 7fb980d22640 1 --2- 172.21.15.23:0/3518253074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c10fec0 0x7fb97c1146c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:16.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.502+0000 7fb980d22640 1 --2- 172.21.15.23:0/3518253074 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb97c10def0 0x7fb97c11a600 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:16.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.502+0000 7fb980d22640 1 -- 172.21.15.23:0/3518253074 >> 172.21.15.23:0/3518253074 conn(0x7fb97c06c780 msgr2=0x7fb97c11c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:16.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.502+0000 7fb980d22640 1 -- 172.21.15.23:0/3518253074 shutdown_connections 2024-03-28T05:34:16.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.504+0000 7fb980d22640 1 -- 172.21.15.23:0/3518253074 wait complete. 2024-03-28T05:34:16.521 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T05:34:16.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:16 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3751056282' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:34:16.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:16 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3751056282' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T05:34:16.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.709+0000 7f09f159a640 1 -- 172.21.15.23:0/3686347712 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e403b660 con 0x7f09ec099fb0 2024-03-28T05:34:16.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.710+0000 7f09f359e640 1 -- 172.21.15.23:0/3686347712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09ec099fb0 msgr2=0x7f09ec0a1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:16.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.710+0000 7f09f359e640 1 --2- 172.21.15.23:0/3686347712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09ec099fb0 0x7f09ec0a1b30 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f09e4010910 tx=0x7f09e402fb40 comp rx=0 tx=0).stop 2024-03-28T05:34:16.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.713+0000 7f09f359e640 1 -- 172.21.15.23:0/3686347712 shutdown_connections 2024-03-28T05:34:16.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.713+0000 7f09f359e640 1 --2- 172.21.15.23:0/3686347712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09ec099fb0 0x7f09ec0a1b30 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:16.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.713+0000 7f09f359e640 1 --2- 172.21.15.23:0/3686347712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09ec095a90 0x7f09ec099880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:16.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.713+0000 7f09f359e640 1 --2- 172.21.15.23:0/3686347712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09ec0950e0 0x7f09ec0954c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:16.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.713+0000 7f09f359e640 1 -- 172.21.15.23:0/3686347712 >> 172.21.15.23:0/3686347712 conn(0x7f09ec00a7f0 msgr2=0x7f09ec00ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:16.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.718+0000 7f09f359e640 1 -- 172.21.15.23:0/3686347712 shutdown_connections 2024-03-28T05:34:16.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.718+0000 7f09f359e640 1 -- 172.21.15.23:0/3686347712 wait complete. 2024-03-28T05:34:16.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.719+0000 7f09f359e640 1 Processor -- start 2024-03-28T05:34:16.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.719+0000 7f09f359e640 1 -- start start 2024-03-28T05:34:16.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.719+0000 7f09f359e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09ec0950e0 0x7f09ec129fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:16.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.719+0000 7f09f359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09ec099fb0 0x7f09ec12a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:16.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.719+0000 7f09f359e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09ec12ab50 0x7f09ec12eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:16.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.720+0000 7f09f359e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec0a3f70 con 0x7f09ec0950e0 2024-03-28T05:34:16.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.720+0000 7f09f359e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec0a3df0 con 0x7f09ec099fb0 2024-03-28T05:34:16.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.720+0000 7f09f359e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec0a40f0 con 0x7f09ec12ab50 2024-03-28T05:34:16.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.720+0000 7f09f1d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09ec099fb0 0x7f09ec12a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:16.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.720+0000 7f09f1d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09ec099fb0 0x7f09ec12a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.23:42162/0 (socket says 172.21.15.23:42162) 2024-03-28T05:34:16.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.720+0000 7f09f1d9b640 1 -- 172.21.15.23:0/2919388708 learned_addr learned my addr 172.21.15.23:0/2919388708 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:34:16.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.720+0000 7f09f2d9d640 1 --2- 172.21.15.23:0/2919388708 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09ec12ab50 0x7f09ec12eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:16.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.720+0000 7f09f1d9b640 1 -- 172.21.15.23:0/2919388708 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09ec12ab50 msgr2=0x7f09ec12eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:16.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.720+0000 7f09f1d9b640 1 --2- 172.21.15.23:0/2919388708 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09ec12ab50 0x7f09ec12eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:16.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.721+0000 7f09f1d9b640 1 -- 172.21.15.23:0/2919388708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09ec0950e0 msgr2=0x7f09ec129fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:16.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.721+0000 7f09f1d9b640 1 --2- 172.21.15.23:0/2919388708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09ec0950e0 0x7f09ec129fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:16.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.721+0000 7f09f1d9b640 1 -- 172.21.15.23:0/2919388708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ec12f360 con 0x7f09ec099fb0 2024-03-28T05:34:16.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.721+0000 7f09f1d9b640 1 --2- 172.21.15.23:0/2919388708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09ec099fb0 0x7f09ec12a500 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f09f402d600 tx=0x7f09f4079950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:16.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.722+0000 7f09e37fe640 1 -- 172.21.15.23:0/2919388708 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f4061ae0 con 0x7f09ec099fb0 2024-03-28T05:34:16.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.722+0000 7f09f359e640 1 -- 172.21.15.23:0/2919388708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09ec12f650 con 0x7f09ec099fb0 2024-03-28T05:34:16.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.722+0000 7f09f359e640 1 -- 172.21.15.23:0/2919388708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ec137150 con 0x7f09ec099fb0 2024-03-28T05:34:16.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.723+0000 7f09e37fe640 1 -- 172.21.15.23:0/2919388708 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09f4061c80 con 0x7f09ec099fb0 2024-03-28T05:34:16.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.723+0000 7f09e37fe640 1 -- 172.21.15.23:0/2919388708 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f407e870 con 0x7f09ec099fb0 2024-03-28T05:34:16.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.724+0000 7f09f359e640 1 -- 172.21.15.23:0/2919388708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d4005290 con 0x7f09ec099fb0 2024-03-28T05:34:16.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.726+0000 7f09e37fe640 1 -- 172.21.15.23:0/2919388708 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f09f4086030 con 0x7f09ec099fb0 2024-03-28T05:34:16.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.727+0000 7f09e37fe640 1 --2- 172.21.15.23:0/2919388708 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f09dc0a2b40 0x7f09dc0a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:16.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.727+0000 7f09e37fe640 1 -- 172.21.15.23:0/2919388708 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9613+0+0 (secure 0 0 0) 0x7f09f4139d70 con 0x7f09ec099fb0 2024-03-28T05:34:16.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.728+0000 7f09f259c640 1 --2- 172.21.15.23:0/2919388708 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f09dc0a2b40 0x7f09dc0a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:16.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.729+0000 7f09f259c640 1 --2- 172.21.15.23:0/2919388708 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f09dc0a2b40 0x7f09dc0a5000 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f09e80096f0 tx=0x7f09e8006c40 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:16.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:16.736+0000 7f09e37fe640 1 -- 172.21.15.23:0/2919388708 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f09f413e070 con 0x7f09ec099fb0 2024-03-28T05:34:17.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.094+0000 7f09f359e640 1 -- 172.21.15.23:0/2919388708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f09d4005850 con 0x7f09ec099fb0 2024-03-28T05:34:17.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.096+0000 7f09e37fe640 1 -- 172.21.15.23:0/2919388708 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v103) v1 ==== 103+0+11 (secure 0 0 0) 0x7f09f408d020 con 0x7f09ec099fb0 2024-03-28T05:34:17.096 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T05:34:17.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.099+0000 7f09f359e640 1 -- 172.21.15.23:0/2919388708 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f09dc0a2b40 msgr2=0x7f09dc0a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:17.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.099+0000 7f09f359e640 1 --2- 172.21.15.23:0/2919388708 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f09dc0a2b40 0x7f09dc0a5000 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f09e80096f0 tx=0x7f09e8006c40 comp rx=0 tx=0).stop 2024-03-28T05:34:17.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.099+0000 7f09f359e640 1 -- 172.21.15.23:0/2919388708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09ec099fb0 msgr2=0x7f09ec12a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:17.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.099+0000 7f09f359e640 1 --2- 172.21.15.23:0/2919388708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09ec099fb0 0x7f09ec12a500 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f09f402d600 tx=0x7f09f4079950 comp rx=0 tx=0).stop 2024-03-28T05:34:17.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.100+0000 7f09f359e640 1 -- 172.21.15.23:0/2919388708 shutdown_connections 2024-03-28T05:34:17.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.100+0000 7f09f359e640 1 --2- 172.21.15.23:0/2919388708 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09ec12ab50 0x7f09ec12eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:17.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.100+0000 7f09f359e640 1 --2- 172.21.15.23:0/2919388708 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f09dc0a2b40 0x7f09dc0a5000 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:17.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.100+0000 7f09f359e640 1 --2- 172.21.15.23:0/2919388708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09ec099fb0 0x7f09ec12a500 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:17.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.100+0000 7f09f359e640 1 --2- 172.21.15.23:0/2919388708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09ec0950e0 0x7f09ec129fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:17.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.100+0000 7f09f359e640 1 -- 172.21.15.23:0/2919388708 >> 172.21.15.23:0/2919388708 conn(0x7f09ec00a7f0 msgr2=0x7f09ec092ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:17.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.100+0000 7f09f359e640 1 -- 172.21.15.23:0/2919388708 shutdown_connections 2024-03-28T05:34:17.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.100+0000 7f09f359e640 1 -- 172.21.15.23:0/2919388708 wait complete. 2024-03-28T05:34:17.117 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T05:34:17.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.344+0000 7f08274f8640 1 -- 172.21.15.23:0/2790898526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0820100ef0 msgr2=0x7f082010d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:17.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.344+0000 7f08274f8640 1 --2- 172.21.15.23:0/2790898526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0820100ef0 0x7f082010d3b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f0818009be0 tx=0x7f0818030090 comp rx=0 tx=0).stop 2024-03-28T05:34:17.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.344+0000 7f08274f8640 1 -- 172.21.15.23:0/2790898526 shutdown_connections 2024-03-28T05:34:17.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.344+0000 7f08274f8640 1 --2- 172.21.15.23:0/2790898526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0820100ef0 0x7f082010d3b0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:17.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.344+0000 7f08274f8640 1 --2- 172.21.15.23:0/2790898526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0820100530 0x7f08201009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:17.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.344+0000 7f08274f8640 1 --2- 172.21.15.23:0/2790898526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08201065f0 0x7f08201069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:17.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.344+0000 7f08274f8640 1 -- 172.21.15.23:0/2790898526 >> 172.21.15.23:0/2790898526 conn(0x7f08200776a0 msgr2=0x7f0820077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:17.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.352+0000 7f08274f8640 1 -- 172.21.15.23:0/2790898526 shutdown_connections 2024-03-28T05:34:17.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.353+0000 7f08274f8640 1 -- 172.21.15.23:0/2790898526 wait complete. 2024-03-28T05:34:17.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.354+0000 7f08274f8640 1 Processor -- start 2024-03-28T05:34:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.355+0000 7f08274f8640 1 -- start start 2024-03-28T05:34:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.356+0000 7f08274f8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0820100530 0x7f082019e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.356+0000 7f08274f8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0820100ef0 0x7f082019ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.356+0000 7f08274f8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08201065f0 0x7f08201a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.356+0000 7f08274f8640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082010f850 con 0x7f0820100530 2024-03-28T05:34:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.356+0000 7f08274f8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082010f6d0 con 0x7f08201065f0 2024-03-28T05:34:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.356+0000 7f08274f8640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082010f9d0 con 0x7f0820100ef0 2024-03-28T05:34:17.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.356+0000 7f0825a95640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08201065f0 0x7f08201a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:17.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.356+0000 7f0825a95640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08201065f0 0x7f08201a2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.23:42180/0 (socket says 172.21.15.23:42180) 2024-03-28T05:34:17.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.356+0000 7f0825a95640 1 -- 172.21.15.23:0/714204179 learned_addr learned my addr 172.21.15.23:0/714204179 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:34:17.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.357+0000 7f0825294640 1 --2- 172.21.15.23:0/714204179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0820100530 0x7f082019e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:17.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.357+0000 7f0825a95640 1 -- 172.21.15.23:0/714204179 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0820100ef0 msgr2=0x7f082019ea40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T05:34:17.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.357+0000 7f0825a95640 1 --2- 172.21.15.23:0/714204179 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0820100ef0 0x7f082019ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:17.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.357+0000 7f0825a95640 1 -- 172.21.15.23:0/714204179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0820100530 msgr2=0x7f082019e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:17.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.357+0000 7f0825a95640 1 --2- 172.21.15.23:0/714204179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0820100530 0x7f082019e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:17.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.357+0000 7f0825a95640 1 -- 172.21.15.23:0/714204179 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08201a34f0 con 0x7f08201065f0 2024-03-28T05:34:17.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.357+0000 7f0825a95640 1 --2- 172.21.15.23:0/714204179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08201065f0 0x7f08201a2dc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f081803a040 tx=0x7f0818037eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:17.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.360+0000 7f08167fc640 1 -- 172.21.15.23:0/714204179 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0818045070 con 0x7f08201065f0 2024-03-28T05:34:17.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.360+0000 7f08274f8640 1 -- 172.21.15.23:0/714204179 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08201a3780 con 0x7f08201065f0 2024-03-28T05:34:17.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.360+0000 7f08274f8640 1 -- 172.21.15.23:0/714204179 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08201ab070 con 0x7f08201065f0 2024-03-28T05:34:17.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.361+0000 7f08167fc640 1 -- 172.21.15.23:0/714204179 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0818003b70 con 0x7f08201065f0 2024-03-28T05:34:17.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.361+0000 7f08167fc640 1 -- 172.21.15.23:0/714204179 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0818038830 con 0x7f08201065f0 2024-03-28T05:34:17.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.366+0000 7f08274f8640 1 -- 172.21.15.23:0/714204179 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0804005290 con 0x7f08201065f0 2024-03-28T05:34:17.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.368+0000 7f08167fc640 1 -- 172.21.15.23:0/714204179 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0818048070 con 0x7f08201065f0 2024-03-28T05:34:17.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.368+0000 7f08167fc640 1 --2- 172.21.15.23:0/714204179 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f080c0a2b70 0x7f080c0a5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:17.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.369+0000 7f08167fc640 1 -- 172.21.15.23:0/714204179 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9613+0+0 (secure 0 0 0) 0x7f08180f0570 con 0x7f08201065f0 2024-03-28T05:34:17.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.370+0000 7f0825294640 1 --2- 172.21.15.23:0/714204179 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f080c0a2b70 0x7f080c0a5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:17.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.371+0000 7f0825294640 1 --2- 172.21.15.23:0/714204179 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f080c0a2b70 0x7f080c0a5030 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f081c005fd0 tx=0x7f081c005850 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:17.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.378+0000 7f08167fc640 1 -- 172.21.15.23:0/714204179 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0818030da0 con 0x7f08201065f0 2024-03-28T05:34:17.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:17 smithi023 ceph-mon[31954]: pgmap v333: 129 pgs: 22 active+clean+snaptrim, 66 active+clean+snaptrim_wait, 41 active+clean; 234 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:34:17.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:17 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3518253074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:34:17.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:17 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2919388708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:34:17.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:17 smithi098 ceph-mon[36087]: pgmap v333: 129 pgs: 22 active+clean+snaptrim, 66 active+clean+snaptrim_wait, 41 active+clean; 234 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:34:17.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:17 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3518253074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:34:17.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:17 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2919388708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:34:17.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:17 smithi113 ceph-mon[37164]: pgmap v333: 129 pgs: 22 active+clean+snaptrim, 66 active+clean+snaptrim_wait, 41 active+clean; 234 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:34:17.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:17 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3518253074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T05:34:17.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:17 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2919388708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T05:34:17.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.759+0000 7f08274f8640 1 -- 172.21.15.23:0/714204179 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0804005850 con 0x7f08201065f0 2024-03-28T05:34:17.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.761+0000 7f08167fc640 1 -- 172.21.15.23:0/714204179 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v103) v1 ==== 81+0+941 (secure 0 0 0) 0x7f08180bcf40 con 0x7f08201065f0 2024-03-28T05:34:17.761 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2772113246 2024-03-29T05:23:33.950262+0000 2024-03-28T05:34:17.761 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/84692814 2024-03-29T05:23:33.950262+0000 2024-03-28T05:34:17.761 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/222178108 2024-03-29T05:23:33.950262+0000 2024-03-28T05:34:17.762 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/1530849123 2024-03-29T05:14:08.640340+0000 2024-03-28T05:34:17.762 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/222178108 2024-03-29T05:23:33.950262+0000 2024-03-28T05:34:17.762 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/1530849123 2024-03-29T05:14:08.640340+0000 2024-03-28T05:34:17.762 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1876451381 2024-03-29T05:14:08.640340+0000 2024-03-28T05:34:17.762 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2253559862 2024-03-29T05:14:08.640340+0000 2024-03-28T05:34:17.762 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2653562762 2024-03-29T05:13:42.579607+0000 2024-03-28T05:34:17.762 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/3337463830 2024-03-29T05:13:42.579607+0000 2024-03-28T05:34:17.762 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/944304909 2024-03-29T05:23:33.950262+0000 2024-03-28T05:34:17.762 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/1735221490 2024-03-29T05:13:42.579607+0000 2024-03-28T05:34:17.763 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2784771635 2024-03-29T05:14:08.640340+0000 2024-03-28T05:34:17.763 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1135235867 2024-03-29T05:23:33.950262+0000 2024-03-28T05:34:17.763 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2389902490 2024-03-29T05:13:42.579607+0000 2024-03-28T05:34:17.763 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/1735221490 2024-03-29T05:13:42.579607+0000 2024-03-28T05:34:17.763 INFO:teuthology.orchestra.run.smithi023.stderr:listed 16 entries 2024-03-28T05:34:17.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.766+0000 7f08274f8640 1 -- 172.21.15.23:0/714204179 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f080c0a2b70 msgr2=0x7f080c0a5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:17.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.766+0000 7f08274f8640 1 --2- 172.21.15.23:0/714204179 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f080c0a2b70 0x7f080c0a5030 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f081c005fd0 tx=0x7f081c005850 comp rx=0 tx=0).stop 2024-03-28T05:34:17.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.766+0000 7f08274f8640 1 -- 172.21.15.23:0/714204179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08201065f0 msgr2=0x7f08201a2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:17.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.766+0000 7f08274f8640 1 --2- 172.21.15.23:0/714204179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08201065f0 0x7f08201a2dc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f081803a040 tx=0x7f0818037eb0 comp rx=0 tx=0).stop 2024-03-28T05:34:17.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.766+0000 7f08274f8640 1 -- 172.21.15.23:0/714204179 shutdown_connections 2024-03-28T05:34:17.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.766+0000 7f08274f8640 1 --2- 172.21.15.23:0/714204179 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f080c0a2b70 0x7f080c0a5030 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f081c005fd0 tx=0x7f081c005850 comp rx=0 tx=0).stop 2024-03-28T05:34:17.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.766+0000 7f08274f8640 1 --2- 172.21.15.23:0/714204179 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08201065f0 0x7f08201a2dc0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:17.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.766+0000 7f08274f8640 1 --2- 172.21.15.23:0/714204179 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0820100ef0 0x7f082019ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:17.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.766+0000 7f08274f8640 1 --2- 172.21.15.23:0/714204179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0820100530 0x7f082019e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:17.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.766+0000 7f08274f8640 1 -- 172.21.15.23:0/714204179 >> 172.21.15.23:0/714204179 conn(0x7f08200776a0 msgr2=0x7f08200ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:17.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.767+0000 7f08274f8640 1 -- 172.21.15.23:0/714204179 shutdown_connections 2024-03-28T05:34:17.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:17.767+0000 7f08274f8640 1 -- 172.21.15.23:0/714204179 wait complete. 2024-03-28T05:34:17.786 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-28T05:34:17.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:34:17.786 DEBUG:teuthology.orchestra.run.smithi023:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-28T05:34:17.862 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi023.front.sepia.ceph.com 2024-03-28T05:34:17.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:34:17.863 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-28T05:34:17.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:34:17.901 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2024-03-28T05:34:17.972 INFO:teuthology.orchestra.run.smithi023.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-28T05:34:17.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:34:17.973 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T05:34:18.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T05:34:18.063 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link delete ceph-brx 2024-03-28T05:34:18.152 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-28T05:34:18.163 INFO:tasks.cephadm:Teardown begin 2024-03-28T05:34:18.164 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T05:34:18.164 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:34:18.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.343+0000 7fab16c54640 1 -- 172.21.15.23:0/2032117912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab10101020 msgr2=0x7fab101054e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:18.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.343+0000 7fab16c54640 1 --2- 172.21.15.23:0/2032117912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab10101020 0x7fab101054e0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fab08009cd0 tx=0x7fab0802c120 comp rx=0 tx=0).stop 2024-03-28T05:34:18.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.343+0000 7fab16c54640 1 -- 172.21.15.23:0/2032117912 shutdown_connections 2024-03-28T05:34:18.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.343+0000 7fab16c54640 1 --2- 172.21.15.23:0/2032117912 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab10105b00 0x7fab10105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:18.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.343+0000 7fab16c54640 1 --2- 172.21.15.23:0/2032117912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab10101020 0x7fab101054e0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:18.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.343+0000 7fab16c54640 1 --2- 172.21.15.23:0/2032117912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab10100700 0x7fab10100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:18.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.344+0000 7fab16c54640 1 -- 172.21.15.23:0/2032117912 >> 172.21.15.23:0/2032117912 conn(0x7fab100776a0 msgr2=0x7fab10077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:18.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.345+0000 7fab16c54640 1 -- 172.21.15.23:0/2032117912 shutdown_connections 2024-03-28T05:34:18.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.345+0000 7fab16c54640 1 -- 172.21.15.23:0/2032117912 wait complete. 2024-03-28T05:34:18.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.346+0000 7fab16c54640 1 Processor -- start 2024-03-28T05:34:18.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.346+0000 7fab16c54640 1 -- start start 2024-03-28T05:34:18.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.347+0000 7fab16c54640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab10100700 0x7fab10072eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:18.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.347+0000 7fab16c54640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab10101020 0x7fab1006bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:18.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.347+0000 7fab16c54640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab10105b00 0x7fab1006c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:18.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.347+0000 7fab16c54640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab10118240 con 0x7fab10101020 2024-03-28T05:34:18.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.347+0000 7fab16c54640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab101180c0 con 0x7fab10100700 2024-03-28T05:34:18.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.347+0000 7fab16c54640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab101183c0 con 0x7fab10105b00 2024-03-28T05:34:18.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.347+0000 7fab151f1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab10105b00 0x7fab1006c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:18.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.347+0000 7fab151f1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab10105b00 0x7fab1006c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:52700/0 (socket says 172.21.15.23:52700) 2024-03-28T05:34:18.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.347+0000 7fab151f1640 1 -- 172.21.15.23:0/2268996158 learned_addr learned my addr 172.21.15.23:0/2268996158 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:34:18.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.347+0000 7fab149f0640 1 --2- 172.21.15.23:0/2268996158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab10100700 0x7fab10072eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:18.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.347+0000 7fab0ffff640 1 --2- 172.21.15.23:0/2268996158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab10101020 0x7fab1006bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:18.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.348+0000 7fab151f1640 1 -- 172.21.15.23:0/2268996158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab10100700 msgr2=0x7fab10072eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:18.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.348+0000 7fab151f1640 1 --2- 172.21.15.23:0/2268996158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab10100700 0x7fab10072eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:18.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.348+0000 7fab151f1640 1 -- 172.21.15.23:0/2268996158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab10101020 msgr2=0x7fab1006bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:18.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.348+0000 7fab151f1640 1 --2- 172.21.15.23:0/2268996158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab10101020 0x7fab1006bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:18.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.348+0000 7fab151f1640 1 -- 172.21.15.23:0/2268996158 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab1006ca30 con 0x7fab10105b00 2024-03-28T05:34:18.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.348+0000 7fab0ffff640 1 --2- 172.21.15.23:0/2268996158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab10101020 0x7fab1006bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T05:34:18.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.348+0000 7fab151f1640 1 --2- 172.21.15.23:0/2268996158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab10105b00 0x7fab1006c470 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fab0000b3f0 tx=0x7fab0000b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:18.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.350+0000 7fab0dffb640 1 -- 172.21.15.23:0/2268996158 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab00015020 con 0x7fab10105b00 2024-03-28T05:34:18.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.350+0000 7fab0dffb640 1 -- 172.21.15.23:0/2268996158 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab00011d10 con 0x7fab10105b00 2024-03-28T05:34:18.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.350+0000 7fab0dffb640 1 -- 172.21.15.23:0/2268996158 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0000f2e0 con 0x7fab10105b00 2024-03-28T05:34:18.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.351+0000 7fab16c54640 1 -- 172.21.15.23:0/2268996158 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab1006cd20 con 0x7fab10105b00 2024-03-28T05:34:18.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.352+0000 7fab16c54640 1 -- 172.21.15.23:0/2268996158 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab10070d00 con 0x7fab10105b00 2024-03-28T05:34:18.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.353+0000 7fab16c54640 1 -- 172.21.15.23:0/2268996158 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab10069040 con 0x7fab10105b00 2024-03-28T05:34:18.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.358+0000 7fab0dffb640 1 -- 172.21.15.23:0/2268996158 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fab00018020 con 0x7fab10105b00 2024-03-28T05:34:18.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.359+0000 7fab0dffb640 1 --2- 172.21.15.23:0/2268996158 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7faafc0a2b70 0x7faafc0a5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:18.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.359+0000 7fab0dffb640 1 -- 172.21.15.23:0/2268996158 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9613+0+0 (secure 0 0 0) 0x7fab000cf140 con 0x7fab10105b00 2024-03-28T05:34:18.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.362+0000 7fab149f0640 1 --2- 172.21.15.23:0/2268996158 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7faafc0a2b70 0x7faafc0a5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:18.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.366+0000 7fab0dffb640 1 -- 172.21.15.23:0/2268996158 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fab0009bb90 con 0x7fab10105b00 2024-03-28T05:34:18.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.369+0000 7fab149f0640 1 --2- 172.21.15.23:0/2268996158 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7faafc0a2b70 0x7faafc0a5030 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faaf8006fd0 tx=0x7faaf8008040 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:18.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:18 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/714204179' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T05:34:18.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:18 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/714204179' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T05:34:18.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:18 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/714204179' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T05:34:18.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.813+0000 7fab16c54640 1 -- 172.21.15.23:0/2268996158 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab10071020 con 0x7fab10105b00 2024-03-28T05:34:18.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.814+0000 7fab0dffb640 1 -- 172.21.15.23:0/2268996158 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11324 (secure 0 0 0) 0x7fab0009b490 con 0x7fab10105b00 2024-03-28T05:34:18.815 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:34:18.815 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/733913400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":733913400},{"type":"v1","addr":"172.21.15.23:6835","nonce":733913400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1972901387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1972901387},{"type":"v1","addr":"172.21.15.23:6837","nonce":1972901387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/2390673788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2390673788},{"type":"v1","addr":"172.21.15.23:6839","nonce":2390673788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/289263156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":289263156},{"type":"v1","addr":"172.21.15.23:6841","nonce":289263156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3313932732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3313932732},{"type":"v1","addr":"172.21.15.98:6835","nonce":3313932732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2695340024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2695340024},{"type":"v1","addr":"172.21.15.113:6835","nonce":2695340024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2750367469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2750367469},{"type":"v1","addr":"172.21.15.98:6837","nonce":2750367469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/3222726274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":3222726274},{"type":"v1","addr":"172.21.15.113:6837","nonce":3222726274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T05:21:07.616553+0000","modified":"2024-03-28T05:21:32.732132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/144113007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":144113007},{"type":"v1","addr":"172.21.15.98:6839","nonce":144113007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.113:6839/212060489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":212060489},{"type":"v1","addr":"172.21.15.113:6839","nonce":212060489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24487,"qdb_cluster":[24487]},"id":1}]} 2024-03-28T05:34:18.815 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 20 2024-03-28T05:34:18.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.818+0000 7fab16c54640 1 -- 172.21.15.23:0/2268996158 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7faafc0a2b70 msgr2=0x7faafc0a5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:18.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.818+0000 7fab16c54640 1 --2- 172.21.15.23:0/2268996158 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7faafc0a2b70 0x7faafc0a5030 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faaf8006fd0 tx=0x7faaf8008040 comp rx=0 tx=0).stop 2024-03-28T05:34:18.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.818+0000 7fab16c54640 1 -- 172.21.15.23:0/2268996158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab10105b00 msgr2=0x7fab1006c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:18.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.818+0000 7fab16c54640 1 --2- 172.21.15.23:0/2268996158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab10105b00 0x7fab1006c470 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fab0000b3f0 tx=0x7fab0000b8b0 comp rx=0 tx=0).stop 2024-03-28T05:34:18.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.818+0000 7fab16c54640 1 -- 172.21.15.23:0/2268996158 shutdown_connections 2024-03-28T05:34:18.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.818+0000 7fab16c54640 1 --2- 172.21.15.23:0/2268996158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab10105b00 0x7fab1006c470 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:18.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.818+0000 7fab16c54640 1 --2- 172.21.15.23:0/2268996158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab10101020 0x7fab1006bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:18.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.818+0000 7fab16c54640 1 --2- 172.21.15.23:0/2268996158 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7faafc0a2b70 0x7faafc0a5030 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:18.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.818+0000 7fab16c54640 1 --2- 172.21.15.23:0/2268996158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab10100700 0x7fab10072eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:18.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.818+0000 7fab16c54640 1 -- 172.21.15.23:0/2268996158 >> 172.21.15.23:0/2268996158 conn(0x7fab100776a0 msgr2=0x7fab101034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:18.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.819+0000 7fab16c54640 1 -- 172.21.15.23:0/2268996158 shutdown_connections 2024-03-28T05:34:18.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:18.819+0000 7fab16c54640 1 -- 172.21.15.23:0/2268996158 wait complete. 2024-03-28T05:34:18.838 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T05:34:19.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.017+0000 7f0cc6edc640 1 -- 172.21.15.23:0/3435341683 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cc011ae80 msgr2=0x7f0cc011d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:19.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.017+0000 7f0cc6edc640 1 --2- 172.21.15.23:0/3435341683 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cc011ae80 0x7f0cc011d270 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8009cd0 tx=0x7f0cb802c100 comp rx=0 tx=0).stop 2024-03-28T05:34:19.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.018+0000 7f0cc6edc640 1 -- 172.21.15.23:0/3435341683 shutdown_connections 2024-03-28T05:34:19.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.018+0000 7f0cc6edc640 1 --2- 172.21.15.23:0/3435341683 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cc011ae80 0x7f0cc011d270 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:19.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.018+0000 7f0cc6edc640 1 --2- 172.21.15.23:0/3435341683 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cc010df00 0x7f0cc010e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:19.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.018+0000 7f0cc6edc640 1 --2- 172.21.15.23:0/3435341683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cc010fed0 0x7f0cc010d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:19.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.018+0000 7f0cc6edc640 1 -- 172.21.15.23:0/3435341683 >> 172.21.15.23:0/3435341683 conn(0x7f0cc006c790 msgr2=0x7f0cc006cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:19.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.018+0000 7f0cc6edc640 1 -- 172.21.15.23:0/3435341683 shutdown_connections 2024-03-28T05:34:19.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.018+0000 7f0cc6edc640 1 -- 172.21.15.23:0/3435341683 wait complete. 2024-03-28T05:34:19.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.019+0000 7f0cc6edc640 1 Processor -- start 2024-03-28T05:34:19.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.019+0000 7f0cc6edc640 1 -- start start 2024-03-28T05:34:19.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.020+0000 7f0cc6edc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cc010df00 0x7f0cc01a6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:19.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.020+0000 7f0cc6edc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cc010fed0 0x7f0cc01a7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:19.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.020+0000 7f0cc6edc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cc011ae80 0x7f0cc01ab6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:19.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.020+0000 7f0cc6edc640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc0120840 con 0x7f0cc010df00 2024-03-28T05:34:19.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.020+0000 7f0cc4c78640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cc010df00 0x7f0cc01a6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:19.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.020+0000 7f0cbffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cc010fed0 0x7f0cc01a7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:19.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.020+0000 7f0cbffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cc010fed0 0x7f0cc01a7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:52718/0 (socket says 172.21.15.23:52718) 2024-03-28T05:34:19.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.020+0000 7f0cc4c78640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cc010df00 0x7f0cc01a6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41656/0 (socket says 172.21.15.23:41656) 2024-03-28T05:34:19.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.020+0000 7f0cbffff640 1 -- 172.21.15.23:0/1454253334 learned_addr learned my addr 172.21.15.23:0/1454253334 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:34:19.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.020+0000 7f0cc6edc640 1 -- 172.21.15.23:0/1454253334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc01206c0 con 0x7f0cc011ae80 2024-03-28T05:34:19.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.020+0000 7f0cc6edc640 1 -- 172.21.15.23:0/1454253334 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc01209c0 con 0x7f0cc010fed0 2024-03-28T05:34:19.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.020+0000 7f0cc5479640 1 --2- 172.21.15.23:0/1454253334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cc011ae80 0x7f0cc01ab6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:19.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.021+0000 7f0cc4c78640 1 -- 172.21.15.23:0/1454253334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cc010fed0 msgr2=0x7f0cc01a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:19.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.021+0000 7f0cc4c78640 1 --2- 172.21.15.23:0/1454253334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cc010fed0 0x7f0cc01a7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:19.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.021+0000 7f0cc4c78640 1 -- 172.21.15.23:0/1454253334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cc011ae80 msgr2=0x7f0cc01ab6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:19.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.021+0000 7f0cc4c78640 1 --2- 172.21.15.23:0/1454253334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cc011ae80 0x7f0cc01ab6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:19.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.021+0000 7f0cc4c78640 1 -- 172.21.15.23:0/1454253334 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc01abdd0 con 0x7f0cc010df00 2024-03-28T05:34:19.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.021+0000 7f0cc4c78640 1 --2- 172.21.15.23:0/1454253334 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cc010df00 0x7f0cc01a6de0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0005fd0 tx=0x7f0cb000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:19.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.022+0000 7f0cbdffb640 1 -- 172.21.15.23:0/1454253334 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0009c30 con 0x7f0cc010df00 2024-03-28T05:34:19.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.022+0000 7f0cbdffb640 1 -- 172.21.15.23:0/1454253334 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0cb0009dd0 con 0x7f0cc010df00 2024-03-28T05:34:19.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.022+0000 7f0cbffff640 1 --2- 172.21.15.23:0/1454253334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cc010fed0 0x7f0cc01a7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:34:19.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.022+0000 7f0cbdffb640 1 -- 172.21.15.23:0/1454253334 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0010460 con 0x7f0cc010df00 2024-03-28T05:34:19.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.022+0000 7f0cc6edc640 1 -- 172.21.15.23:0/1454253334 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cc01ac0c0 con 0x7f0cc010df00 2024-03-28T05:34:19.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.022+0000 7f0cc6edc640 1 -- 172.21.15.23:0/1454253334 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cc01b3950 con 0x7f0cc010df00 2024-03-28T05:34:19.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.025+0000 7f0ca77fe640 1 -- 172.21.15.23:0/1454253334 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cc0068490 con 0x7f0cc010df00 2024-03-28T05:34:19.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.028+0000 7f0cbdffb640 1 -- 172.21.15.23:0/1454253334 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0cb0003f60 con 0x7f0cc010df00 2024-03-28T05:34:19.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.029+0000 7f0cbdffb640 1 --2- 172.21.15.23:0/1454253334 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f0ca80a2b70 0x7f0ca80a5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:19.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.029+0000 7f0cbdffb640 1 -- 172.21.15.23:0/1454253334 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9613+0+0 (secure 0 0 0) 0x7f0cb00cedb0 con 0x7f0cc010df00 2024-03-28T05:34:19.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.033+0000 7f0cbffff640 1 --2- 172.21.15.23:0/1454253334 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f0ca80a2b70 0x7f0ca80a5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:19.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.037+0000 7f0cbffff640 1 --2- 172.21.15.23:0/1454253334 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f0ca80a2b70 0x7f0ca80a5030 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0cc011d2c0 tx=0x7f0cb4008040 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:19.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.038+0000 7f0cbdffb640 1 -- 172.21.15.23:0/1454253334 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0cb0017020 con 0x7f0cc010df00 2024-03-28T05:34:19.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.479+0000 7f0ca77fe640 1 -- 172.21.15.23:0/1454253334 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cc00622a0 con 0x7f0cc010df00 2024-03-28T05:34:19.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.482+0000 7f0cbdffb640 1 -- 172.21.15.23:0/1454253334 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11324 (secure 0 0 0) 0x7f0cb009b670 con 0x7f0cc010df00 2024-03-28T05:34:19.483 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:34:19.483 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/733913400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":733913400},{"type":"v1","addr":"172.21.15.23:6835","nonce":733913400}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1972901387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1972901387},{"type":"v1","addr":"172.21.15.23:6837","nonce":1972901387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/2390673788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2390673788},{"type":"v1","addr":"172.21.15.23:6839","nonce":2390673788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/289263156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":289263156},{"type":"v1","addr":"172.21.15.23:6841","nonce":289263156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6833/1563379945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":1563379945},{"type":"v1","addr":"172.21.15.98:6833","nonce":1563379945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/3436936808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":3436936808},{"type":"v1","addr":"172.21.15.113:6833","nonce":3436936808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24413,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/3313932732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3313932732},{"type":"v1","addr":"172.21.15.98:6835","nonce":3313932732}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/2695340024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":2695340024},{"type":"v1","addr":"172.21.15.113:6835","nonce":2695340024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2750367469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":2750367469},{"type":"v1","addr":"172.21.15.98:6837","nonce":2750367469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/3222726274","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":3222726274},{"type":"v1","addr":"172.21.15.113:6837","nonce":3222726274}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T05:21:07.616553+0000","modified":"2024-03-28T05:21:32.732132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24487},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/144113007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":144113007},{"type":"v1","addr":"172.21.15.98:6839","nonce":144113007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.113:6839/212060489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":212060489},{"type":"v1","addr":"172.21.15.113:6839","nonce":212060489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24487,"qdb_cluster":[24487]},"id":1}]} 2024-03-28T05:34:19.483 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 20 2024-03-28T05:34:19.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.486+0000 7f0ca77fe640 1 -- 172.21.15.23:0/1454253334 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f0ca80a2b70 msgr2=0x7f0ca80a5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:19.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.486+0000 7f0ca77fe640 1 --2- 172.21.15.23:0/1454253334 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f0ca80a2b70 0x7f0ca80a5030 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0cc011d2c0 tx=0x7f0cb4008040 comp rx=0 tx=0).stop 2024-03-28T05:34:19.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.486+0000 7f0ca77fe640 1 -- 172.21.15.23:0/1454253334 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cc010df00 msgr2=0x7f0cc01a6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:19.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.486+0000 7f0ca77fe640 1 --2- 172.21.15.23:0/1454253334 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cc010df00 0x7f0cc01a6de0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0005fd0 tx=0x7f0cb000dde0 comp rx=0 tx=0).stop 2024-03-28T05:34:19.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.486+0000 7f0ca77fe640 1 -- 172.21.15.23:0/1454253334 shutdown_connections 2024-03-28T05:34:19.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.486+0000 7f0ca77fe640 1 --2- 172.21.15.23:0/1454253334 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f0ca80a2b70 0x7f0ca80a5030 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:19.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.486+0000 7f0ca77fe640 1 --2- 172.21.15.23:0/1454253334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cc011ae80 0x7f0cc01ab6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:19.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.487+0000 7f0ca77fe640 1 --2- 172.21.15.23:0/1454253334 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cc010fed0 0x7f0cc01a7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:19.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.487+0000 7f0ca77fe640 1 --2- 172.21.15.23:0/1454253334 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cc010df00 0x7f0cc01a6de0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:19.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.487+0000 7f0ca77fe640 1 -- 172.21.15.23:0/1454253334 >> 172.21.15.23:0/1454253334 conn(0x7f0cc006c790 msgr2=0x7f0cc0072180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:19.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.487+0000 7f0ca77fe640 1 -- 172.21.15.23:0/1454253334 shutdown_connections 2024-03-28T05:34:19.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.487+0000 7f0ca77fe640 1 -- 172.21.15.23:0/1454253334 wait complete. 2024-03-28T05:34:19.504 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T05:34:19.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:19 smithi023 ceph-mon[31954]: pgmap v334: 129 pgs: 22 active+clean+snaptrim, 66 active+clean+snaptrim_wait, 41 active+clean; 234 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:34:19.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:19 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2268996158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:34:19.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:19 smithi098 ceph-mon[36087]: pgmap v334: 129 pgs: 22 active+clean+snaptrim, 66 active+clean+snaptrim_wait, 41 active+clean; 234 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:34:19.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:19 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2268996158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:34:19.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.687+0000 7f339def8640 1 -- 172.21.15.23:0/517591258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f339810df00 msgr2=0x7f339810e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:19.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.687+0000 7f339def8640 1 --2- 172.21.15.23:0/517591258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f339810df00 0x7f339810e380 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f3388009930 tx=0x7f338802bc60 comp rx=0 tx=0).stop 2024-03-28T05:34:19.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.688+0000 7f339def8640 1 -- 172.21.15.23:0/517591258 shutdown_connections 2024-03-28T05:34:19.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.688+0000 7f339def8640 1 --2- 172.21.15.23:0/517591258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f339811ae80 0x7f339811d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:19.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.688+0000 7f339def8640 1 --2- 172.21.15.23:0/517591258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f339810df00 0x7f339810e380 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:19.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.688+0000 7f339def8640 1 --2- 172.21.15.23:0/517591258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f339810fed0 0x7f339810d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:19.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.688+0000 7f339def8640 1 -- 172.21.15.23:0/517591258 >> 172.21.15.23:0/517591258 conn(0x7f339806c790 msgr2=0x7f339806cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:19.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.690+0000 7f339def8640 1 -- 172.21.15.23:0/517591258 shutdown_connections 2024-03-28T05:34:19.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.692+0000 7f339def8640 1 -- 172.21.15.23:0/517591258 wait complete. 2024-03-28T05:34:19.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.693+0000 7f339def8640 1 Processor -- start 2024-03-28T05:34:19.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.694+0000 7f339def8640 1 -- start start 2024-03-28T05:34:19.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.695+0000 7f339def8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f339810df00 0x7f3398118c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:19.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.695+0000 7f339def8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f339810fed0 0x7f3398119140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:19.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.695+0000 7f339def8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f339811ae80 0x7f3398111eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:19.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.695+0000 7f339def8640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3398120950 con 0x7f339810df00 2024-03-28T05:34:19.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.695+0000 7f339def8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33981207d0 con 0x7f339811ae80 2024-03-28T05:34:19.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.695+0000 7f339def8640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3398120ad0 con 0x7f339810fed0 2024-03-28T05:34:19.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.696+0000 7f3396ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f339810fed0 0x7f3398119140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:19.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.696+0000 7f3396ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f339810fed0 0x7f3398119140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:52732/0 (socket says 172.21.15.23:52732) 2024-03-28T05:34:19.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.696+0000 7f3396ffd640 1 -- 172.21.15.23:0/3244455344 learned_addr learned my addr 172.21.15.23:0/3244455344 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:34:19.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.697+0000 7f3396ffd640 1 -- 172.21.15.23:0/3244455344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f339811ae80 msgr2=0x7f3398111eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T05:34:19.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.697+0000 7f3396ffd640 1 --2- 172.21.15.23:0/3244455344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f339811ae80 0x7f3398111eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:19.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.697+0000 7f3396ffd640 1 -- 172.21.15.23:0/3244455344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f339810df00 msgr2=0x7f3398118c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:19.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.697+0000 7f3396ffd640 1 --2- 172.21.15.23:0/3244455344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f339810df00 0x7f3398118c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:19.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.697+0000 7f3396ffd640 1 -- 172.21.15.23:0/3244455344 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3398112770 con 0x7f339810fed0 2024-03-28T05:34:19.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.699+0000 7f3396ffd640 1 --2- 172.21.15.23:0/3244455344 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f339810fed0 0x7f3398119140 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f338802efd0 tx=0x7f3388004430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:19.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.699+0000 7f3394ff9640 1 -- 172.21.15.23:0/3244455344 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3388041070 con 0x7f339810fed0 2024-03-28T05:34:19.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.699+0000 7f3394ff9640 1 -- 172.21.15.23:0/3244455344 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f338802ccc0 con 0x7f339810fed0 2024-03-28T05:34:19.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.699+0000 7f3394ff9640 1 -- 172.21.15.23:0/3244455344 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3388034440 con 0x7f339810fed0 2024-03-28T05:34:19.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.702+0000 7f339def8640 1 -- 172.21.15.23:0/3244455344 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3398112a00 con 0x7f339810fed0 2024-03-28T05:34:19.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.702+0000 7f339def8640 1 -- 172.21.15.23:0/3244455344 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3398116900 con 0x7f339810fed0 2024-03-28T05:34:19.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.705+0000 7f339def8640 1 -- 172.21.15.23:0/3244455344 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f339806da30 con 0x7f339810fed0 2024-03-28T05:34:19.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.708+0000 7f3394ff9640 1 -- 172.21.15.23:0/3244455344 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f3388044070 con 0x7f339810fed0 2024-03-28T05:34:19.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.709+0000 7f3394ff9640 1 --2- 172.21.15.23:0/3244455344 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f33800a29f0 0x7f33800a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:19.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.709+0000 7f33977fe640 1 --2- 172.21.15.23:0/3244455344 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f33800a29f0 0x7f33800a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:19.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.710+0000 7f3394ff9640 1 -- 172.21.15.23:0/3244455344 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9613+0+0 (secure 0 0 0) 0x7f33880bab90 con 0x7f339810fed0 2024-03-28T05:34:19.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.718+0000 7f33977fe640 1 --2- 172.21.15.23:0/3244455344 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f33800a29f0 0x7f33800a4eb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f338c005fd0 tx=0x7f338c005850 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:19.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:19.719+0000 7f3394ff9640 1 -- 172.21.15.23:0/3244455344 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f33880bd1f0 con 0x7f339810fed0 2024-03-28T05:34:19.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:19 smithi113 ceph-mon[37164]: pgmap v334: 129 pgs: 22 active+clean+snaptrim, 66 active+clean+snaptrim_wait, 41 active+clean; 234 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-28T05:34:19.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:19 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2268996158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:34:20.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.082+0000 7f339def8640 1 -- 172.21.15.23:0/3244455344 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3398116c70 con 0x7f339810fed0 2024-03-28T05:34:20.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.085+0000 7f3394ff9640 1 -- 172.21.15.23:0/3244455344 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+25389 (secure 0 0 0) 0x7f33880bcaf0 con 0x7f339810fed0 2024-03-28T05:34:20.087 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T05:34:20.087 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":103,"fsid":"bf37ebc4-ecc1-11ee-95d0-87774f69a715","created":"2024-03-28T05:13:13.453653+0000","modified":"2024-03-28T05:34:11.228836+0000","last_up_change":"2024-03-28T05:20:20.221527+0000","last_in_change":"2024-03-28T05:20:00.365478+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T05:16:32.986974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T05:21:06.103894+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T05:21:06.827190+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T05:21:10.507418+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"18150942-59a4-4c3b-83d4-c1fc3ccf1c09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6803","nonce":1219956881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6805","nonce":1219956881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6809","nonce":1219956881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1219956881},{"type":"v1","addr":"172.21.15.23:6807","nonce":1219956881}]},"public_addr":"172.21.15.23:6803/1219956881","cluster_addr":"172.21.15.23:6805/1219956881","heartbeat_back_addr":"172.21.15.23:6809/1219956881","heartbeat_front_addr":"172.21.15.23:6807/1219956881","state":["exists","up"]},{"osd":1,"uuid":"ba6d1a95-5135-4670-95f9-7878c0e7bd64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6801","nonce":555491234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6803","nonce":555491234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6807","nonce":555491234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":555491234},{"type":"v1","addr":"172.21.15.98:6805","nonce":555491234}]},"public_addr":"172.21.15.98:6801/555491234","cluster_addr":"172.21.15.98:6803/555491234","heartbeat_back_addr":"172.21.15.98:6807/555491234","heartbeat_front_addr":"172.21.15.98:6805/555491234","state":["exists","up"]},{"osd":2,"uuid":"873bdd60-2a47-403b-b1f4-ee9226a95067","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6801","nonce":1535856914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6803","nonce":1535856914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6807","nonce":1535856914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":1535856914},{"type":"v1","addr":"172.21.15.113:6805","nonce":1535856914}]},"public_addr":"172.21.15.113:6801/1535856914","cluster_addr":"172.21.15.113:6803/1535856914","heartbeat_back_addr":"172.21.15.113:6807/1535856914","heartbeat_front_addr":"172.21.15.113:6805/1535856914","state":["exists","up"]},{"osd":3,"uuid":"83cd72ef-9338-4bd0-8a17-75c9cb2e7bd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6811","nonce":2949415020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6813","nonce":2949415020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6817","nonce":2949415020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2949415020},{"type":"v1","addr":"172.21.15.23:6815","nonce":2949415020}]},"public_addr":"172.21.15.23:6811/2949415020","cluster_addr":"172.21.15.23:6813/2949415020","heartbeat_back_addr":"172.21.15.23:6817/2949415020","heartbeat_front_addr":"172.21.15.23:6815/2949415020","state":["exists","up"]},{"osd":4,"uuid":"76a90bac-b221-40a3-8238-40e8049cec36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6809","nonce":2615902819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6811","nonce":2615902819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6815","nonce":2615902819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2615902819},{"type":"v1","addr":"172.21.15.98:6813","nonce":2615902819}]},"public_addr":"172.21.15.98:6809/2615902819","cluster_addr":"172.21.15.98:6811/2615902819","heartbeat_back_addr":"172.21.15.98:6815/2615902819","heartbeat_front_addr":"172.21.15.98:6813/2615902819","state":["exists","up"]},{"osd":5,"uuid":"b53fdb55-a8df-40a8-bd1b-8bcef634e26a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6809","nonce":625271528}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6811","nonce":625271528}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6815","nonce":625271528}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":625271528},{"type":"v1","addr":"172.21.15.113:6813","nonce":625271528}]},"public_addr":"172.21.15.113:6809/625271528","cluster_addr":"172.21.15.113:6811/625271528","heartbeat_back_addr":"172.21.15.113:6815/625271528","heartbeat_front_addr":"172.21.15.113:6813/625271528","state":["exists","up"]},{"osd":6,"uuid":"f00283d2-8d67-4e97-a575-9a9d3b55d5cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6819","nonce":4188464176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6821","nonce":4188464176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6825","nonce":4188464176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4188464176},{"type":"v1","addr":"172.21.15.23:6823","nonce":4188464176}]},"public_addr":"172.21.15.23:6819/4188464176","cluster_addr":"172.21.15.23:6821/4188464176","heartbeat_back_addr":"172.21.15.23:6825/4188464176","heartbeat_front_addr":"172.21.15.23:6823/4188464176","state":["exists","up"]},{"osd":7,"uuid":"edc008a7-b07b-40a8-88b7-1614ba38a775","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6817","nonce":1985546028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6819","nonce":1985546028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6823","nonce":1985546028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":1985546028},{"type":"v1","addr":"172.21.15.98:6821","nonce":1985546028}]},"public_addr":"172.21.15.98:6817/1985546028","cluster_addr":"172.21.15.98:6819/1985546028","heartbeat_back_addr":"172.21.15.98:6823/1985546028","heartbeat_front_addr":"172.21.15.98:6821/1985546028","state":["exists","up"]},{"osd":8,"uuid":"56e7b2d1-81f0-47b2-8638-f29832bed8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6817","nonce":3534876081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6819","nonce":3534876081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6823","nonce":3534876081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":3534876081},{"type":"v1","addr":"172.21.15.113:6821","nonce":3534876081}]},"public_addr":"172.21.15.113:6817/3534876081","cluster_addr":"172.21.15.113:6819/3534876081","heartbeat_back_addr":"172.21.15.113:6823/3534876081","heartbeat_front_addr":"172.21.15.113:6821/3534876081","state":["exists","up"]},{"osd":9,"uuid":"a2e90caa-6e9c-42bf-9f18-81f7f47880cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6827","nonce":672264207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6829","nonce":672264207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6833","nonce":672264207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":672264207},{"type":"v1","addr":"172.21.15.23:6831","nonce":672264207}]},"public_addr":"172.21.15.23:6827/672264207","cluster_addr":"172.21.15.23:6829/672264207","heartbeat_back_addr":"172.21.15.23:6833/672264207","heartbeat_front_addr":"172.21.15.23:6831/672264207","state":["exists","up"]},{"osd":10,"uuid":"0eda917f-020c-437f-95ae-b26203f6c68e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6825","nonce":2699920595}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6827","nonce":2699920595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6831","nonce":2699920595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2699920595},{"type":"v1","addr":"172.21.15.98:6829","nonce":2699920595}]},"public_addr":"172.21.15.98:6825/2699920595","cluster_addr":"172.21.15.98:6827/2699920595","heartbeat_back_addr":"172.21.15.98:6831/2699920595","heartbeat_front_addr":"172.21.15.98:6829/2699920595","state":["exists","up"]},{"osd":11,"uuid":"aae78501-78a8-469a-be20-3006f5ec5037","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6825","nonce":2113644020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6827","nonce":2113644020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6831","nonce":2113644020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":2113644020},{"type":"v1","addr":"172.21.15.113:6829","nonce":2113644020}]},"public_addr":"172.21.15.113:6825/2113644020","cluster_addr":"172.21.15.113:6827/2113644020","heartbeat_back_addr":"172.21.15.113:6831/2113644020","heartbeat_front_addr":"172.21.15.113:6829/2113644020","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:15:43.527475+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:07.006538+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:30.594176+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:16:55.810999+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:20.595811+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:17:45.474138+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:10.500409+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:18:35.787638+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:01.001188+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:26.761115+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:19:52.952752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T05:20:18.559832+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":4}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":7}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2772113246":"2024-03-29T05:23:33.950262+0000","172.21.15.23:0/84692814":"2024-03-29T05:23:33.950262+0000","172.21.15.23:6801/222178108":"2024-03-29T05:23:33.950262+0000","172.21.15.23:6801/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:6800/222178108":"2024-03-29T05:23:33.950262+0000","172.21.15.23:6800/1530849123":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1876451381":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2253559862":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/2653562762":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/3337463830":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/944304909":"2024-03-29T05:23:33.950262+0000","172.21.15.23:6800/1735221490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:0/2784771635":"2024-03-29T05:14:08.640340+0000","172.21.15.23:0/1135235867":"2024-03-29T05:23:33.950262+0000","172.21.15.23:0/2389902490":"2024-03-29T05:13:42.579607+0000","172.21.15.23:6801/1735221490":"2024-03-29T05:13:42.579607+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T05:34:20.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.089+0000 7f339def8640 1 -- 172.21.15.23:0/3244455344 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f33800a29f0 msgr2=0x7f33800a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:20.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.089+0000 7f339def8640 1 --2- 172.21.15.23:0/3244455344 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f33800a29f0 0x7f33800a4eb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f338c005fd0 tx=0x7f338c005850 comp rx=0 tx=0).stop 2024-03-28T05:34:20.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.089+0000 7f339def8640 1 -- 172.21.15.23:0/3244455344 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f339810fed0 msgr2=0x7f3398119140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:20.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.089+0000 7f339def8640 1 --2- 172.21.15.23:0/3244455344 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f339810fed0 0x7f3398119140 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f338802efd0 tx=0x7f3388004430 comp rx=0 tx=0).stop 2024-03-28T05:34:20.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.089+0000 7f339def8640 1 -- 172.21.15.23:0/3244455344 shutdown_connections 2024-03-28T05:34:20.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.089+0000 7f339def8640 1 --2- 172.21.15.23:0/3244455344 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f33800a29f0 0x7f33800a4eb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:20.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.089+0000 7f339def8640 1 --2- 172.21.15.23:0/3244455344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f339811ae80 0x7f3398111eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:20.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.089+0000 7f339def8640 1 --2- 172.21.15.23:0/3244455344 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f339810fed0 0x7f3398119140 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:20.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.089+0000 7f339def8640 1 --2- 172.21.15.23:0/3244455344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f339810df00 0x7f3398118c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:20.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.089+0000 7f339def8640 1 -- 172.21.15.23:0/3244455344 >> 172.21.15.23:0/3244455344 conn(0x7f339806c790 msgr2=0x7f3398072180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:20.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.090+0000 7f339def8640 1 -- 172.21.15.23:0/3244455344 shutdown_connections 2024-03-28T05:34:20.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.090+0000 7f339def8640 1 -- 172.21.15.23:0/3244455344 wait complete. 2024-03-28T05:34:20.108 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-28T05:34:20.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.304+0000 7fd6e5e97640 1 -- 172.21.15.23:0/3757195067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6e0100ef0 msgr2=0x7fd6e0068800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:20.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.304+0000 7fd6e5e97640 1 --2- 172.21.15.23:0/3757195067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6e0100ef0 0x7fd6e0068800 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8009ca0 tx=0x7fd6d802c140 comp rx=0 tx=0).stop 2024-03-28T05:34:20.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.308+0000 7fd6e5e97640 1 -- 172.21.15.23:0/3757195067 shutdown_connections 2024-03-28T05:34:20.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.308+0000 7fd6e5e97640 1 --2- 172.21.15.23:0/3757195067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6e0100ef0 0x7fd6e0068800 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:20.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.308+0000 7fd6e5e97640 1 --2- 172.21.15.23:0/3757195067 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6e0100530 0x7fd6e01009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:20.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.308+0000 7fd6e5e97640 1 --2- 172.21.15.23:0/3757195067 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6e01065f0 0x7fd6e01069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:20.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.308+0000 7fd6e5e97640 1 -- 172.21.15.23:0/3757195067 >> 172.21.15.23:0/3757195067 conn(0x7fd6e00776a0 msgr2=0x7fd6e0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:20.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.315+0000 7fd6e5e97640 1 -- 172.21.15.23:0/3757195067 shutdown_connections 2024-03-28T05:34:20.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.317+0000 7fd6e5e97640 1 -- 172.21.15.23:0/3757195067 wait complete. 2024-03-28T05:34:20.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.317+0000 7fd6e5e97640 1 Processor -- start 2024-03-28T05:34:20.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.318+0000 7fd6e5e97640 1 -- start start 2024-03-28T05:34:20.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.318+0000 7fd6e5e97640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6e0100530 0x7fd6e0115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:20.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.318+0000 7fd6e5e97640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6e0100ef0 0x7fd6e0116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:20.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.318+0000 7fd6e5e97640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6e01065f0 0x7fd6e011b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:20.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.318+0000 7fd6e5e97640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e010f890 con 0x7fd6e01065f0 2024-03-28T05:34:20.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.318+0000 7fd6e5e97640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e010f710 con 0x7fd6e0100530 2024-03-28T05:34:20.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.318+0000 7fd6e5e97640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e010fa10 con 0x7fd6e0100ef0 2024-03-28T05:34:20.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.320+0000 7fd6dffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6e01065f0 0x7fd6e011b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:20.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.320+0000 7fd6dffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6e01065f0 0x7fd6e011b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41688/0 (socket says 172.21.15.23:41688) 2024-03-28T05:34:20.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.320+0000 7fd6dffff640 1 -- 172.21.15.23:0/274240458 learned_addr learned my addr 172.21.15.23:0/274240458 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:34:20.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.320+0000 7fd6df7fe640 1 --2- 172.21.15.23:0/274240458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6e0100530 0x7fd6e0115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:20.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.321+0000 7fd6dffff640 1 -- 172.21.15.23:0/274240458 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6e0100ef0 msgr2=0x7fd6e0116220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T05:34:20.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.321+0000 7fd6dffff640 1 --2- 172.21.15.23:0/274240458 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6e0100ef0 0x7fd6e0116220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:20.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.321+0000 7fd6dffff640 1 -- 172.21.15.23:0/274240458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6e0100530 msgr2=0x7fd6e0115ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:20.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.321+0000 7fd6dffff640 1 --2- 172.21.15.23:0/274240458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6e0100530 0x7fd6e0115ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:20.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.321+0000 7fd6dffff640 1 -- 172.21.15.23:0/274240458 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e0110f10 con 0x7fd6e01065f0 2024-03-28T05:34:20.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.322+0000 7fd6dffff640 1 --2- 172.21.15.23:0/274240458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6e01065f0 0x7fd6e011b9f0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd6d802c6d0 tx=0x7fd6d8032550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:20.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.322+0000 7fd6dcff9640 1 -- 172.21.15.23:0/274240458 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8039040 con 0x7fd6e01065f0 2024-03-28T05:34:20.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.322+0000 7fd6dcff9640 1 -- 172.21.15.23:0/274240458 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6d8044070 con 0x7fd6e01065f0 2024-03-28T05:34:20.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.322+0000 7fd6dcff9640 1 -- 172.21.15.23:0/274240458 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8004800 con 0x7fd6e01065f0 2024-03-28T05:34:20.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.323+0000 7fd6e5e97640 1 -- 172.21.15.23:0/274240458 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e01111a0 con 0x7fd6e01065f0 2024-03-28T05:34:20.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.323+0000 7fd6e5e97640 1 -- 172.21.15.23:0/274240458 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e0111640 con 0x7fd6e01065f0 2024-03-28T05:34:20.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.327+0000 7fd6dcff9640 1 -- 172.21.15.23:0/274240458 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fd6d8003970 con 0x7fd6e01065f0 2024-03-28T05:34:20.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.328+0000 7fd6e5e97640 1 -- 172.21.15.23:0/274240458 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e0101c50 con 0x7fd6e01065f0 2024-03-28T05:34:20.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.328+0000 7fd6dcff9640 1 --2- 172.21.15.23:0/274240458 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fd6c80a2b80 0x7fd6c80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:20.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.328+0000 7fd6df7fe640 1 --2- 172.21.15.23:0/274240458 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fd6c80a2b80 0x7fd6c80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:20.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.329+0000 7fd6dcff9640 1 -- 172.21.15.23:0/274240458 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9613+0+0 (secure 0 0 0) 0x7fd6d80f1650 con 0x7fd6e01065f0 2024-03-28T05:34:20.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.329+0000 7fd6df7fe640 1 --2- 172.21.15.23:0/274240458 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fd6c80a2b80 0x7fd6c80a5040 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4006fd0 tx=0x7fd6d40090e0 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:20.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.338+0000 7fd6dcff9640 1 -- 172.21.15.23:0/274240458 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd6d80be020 con 0x7fd6e01065f0 2024-03-28T05:34:20.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:20 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/1454253334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:34:20.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:20 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3244455344' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:34:20.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:20 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/1454253334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:34:20.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:20 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3244455344' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:34:20.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:20 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/1454253334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T05:34:20.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:20 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3244455344' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T05:34:20.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.782+0000 7fd6e5e97640 1 -- 172.21.15.23:0/274240458 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd6e01118a0 con 0x7fd6e01065f0 2024-03-28T05:34:20.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.791+0000 7fd6dcff9640 1 -- 172.21.15.23:0/274240458 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v21) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd6d80bd920 con 0x7fd6e01065f0 2024-03-28T05:34:20.791 INFO:teuthology.orchestra.run.smithi023.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T05:34:20.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.794+0000 7fd6e5e97640 1 -- 172.21.15.23:0/274240458 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fd6c80a2b80 msgr2=0x7fd6c80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:20.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.794+0000 7fd6e5e97640 1 --2- 172.21.15.23:0/274240458 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fd6c80a2b80 0x7fd6c80a5040 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4006fd0 tx=0x7fd6d40090e0 comp rx=0 tx=0).stop 2024-03-28T05:34:20.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.794+0000 7fd6e5e97640 1 -- 172.21.15.23:0/274240458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6e01065f0 msgr2=0x7fd6e011b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:20.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.794+0000 7fd6e5e97640 1 --2- 172.21.15.23:0/274240458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6e01065f0 0x7fd6e011b9f0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd6d802c6d0 tx=0x7fd6d8032550 comp rx=0 tx=0).stop 2024-03-28T05:34:20.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.794+0000 7fd6e5e97640 1 -- 172.21.15.23:0/274240458 shutdown_connections 2024-03-28T05:34:20.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.794+0000 7fd6e5e97640 1 --2- 172.21.15.23:0/274240458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6e01065f0 0x7fd6e011b9f0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:20.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.794+0000 7fd6e5e97640 1 --2- 172.21.15.23:0/274240458 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6e0100ef0 0x7fd6e0116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:20.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.794+0000 7fd6e5e97640 1 --2- 172.21.15.23:0/274240458 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7fd6c80a2b80 0x7fd6c80a5040 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:20.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.794+0000 7fd6e5e97640 1 --2- 172.21.15.23:0/274240458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd6e0100530 0x7fd6e0115ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:20.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.794+0000 7fd6e5e97640 1 -- 172.21.15.23:0/274240458 >> 172.21.15.23:0/274240458 conn(0x7fd6e00776a0 msgr2=0x7fd6e01040e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:20.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.794+0000 7fd6e5e97640 1 -- 172.21.15.23:0/274240458 shutdown_connections 2024-03-28T05:34:20.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:20.794+0000 7fd6e5e97640 1 -- 172.21.15.23:0/274240458 wait complete. 2024-03-28T05:34:20.812 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-28T05:34:21.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.002+0000 7f9d37737640 1 -- 172.21.15.23:0/1700834099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d30100f00 msgr2=0x7f9d3010d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:21.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.002+0000 7f9d37737640 1 --2- 172.21.15.23:0/1700834099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d30100f00 0x7f9d3010d410 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f9d28009cd0 tx=0x7f9d2802c120 comp rx=0 tx=0).stop 2024-03-28T05:34:21.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.005+0000 7f9d37737640 1 -- 172.21.15.23:0/1700834099 shutdown_connections 2024-03-28T05:34:21.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.005+0000 7f9d37737640 1 --2- 172.21.15.23:0/1700834099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d30100f00 0x7f9d3010d410 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:21.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.005+0000 7f9d37737640 1 --2- 172.21.15.23:0/1700834099 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d30100540 0x7f9d301009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:21.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.005+0000 7f9d37737640 1 --2- 172.21.15.23:0/1700834099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d30105560 0x7f9d30105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:21.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.005+0000 7f9d37737640 1 -- 172.21.15.23:0/1700834099 >> 172.21.15.23:0/1700834099 conn(0x7f9d30077630 msgr2=0x7f9d30077a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:21.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.008+0000 7f9d37737640 1 -- 172.21.15.23:0/1700834099 shutdown_connections 2024-03-28T05:34:21.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.009+0000 7f9d37737640 1 -- 172.21.15.23:0/1700834099 wait complete. 2024-03-28T05:34:21.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.010+0000 7f9d37737640 1 Processor -- start 2024-03-28T05:34:21.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.010+0000 7f9d37737640 1 -- start start 2024-03-28T05:34:21.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.010+0000 7f9d37737640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d30100540 0x7f9d3019cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:21.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.010+0000 7f9d37737640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d30100f00 0x7f9d30194ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:21.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.010+0000 7f9d37737640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d30105560 0x7f9d30195410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:21.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.010+0000 7f9d37737640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3010f930 con 0x7f9d30105560 2024-03-28T05:34:21.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.010+0000 7f9d37737640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3010f7b0 con 0x7f9d30100540 2024-03-28T05:34:21.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.010+0000 7f9d37737640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3010fab0 con 0x7f9d30100f00 2024-03-28T05:34:21.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.011+0000 7f9d35cd4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d30105560 0x7f9d30195410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:21.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.011+0000 7f9d35cd4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d30105560 0x7f9d30195410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41702/0 (socket says 172.21.15.23:41702) 2024-03-28T05:34:21.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.011+0000 7f9d35cd4640 1 -- 172.21.15.23:0/4002873875 learned_addr learned my addr 172.21.15.23:0/4002873875 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:34:21.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.011+0000 7f9d35cd4640 1 -- 172.21.15.23:0/4002873875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d30100f00 msgr2=0x7f9d30194ed0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-28T05:34:21.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.011+0000 7f9d35cd4640 1 --2- 172.21.15.23:0/4002873875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d30100f00 0x7f9d30194ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:21.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.011+0000 7f9d35cd4640 1 -- 172.21.15.23:0/4002873875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d30100540 msgr2=0x7f9d3019cf10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T05:34:21.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.011+0000 7f9d35cd4640 1 --2- 172.21.15.23:0/4002873875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d30100540 0x7f9d3019cf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:21.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.011+0000 7f9d35cd4640 1 -- 172.21.15.23:0/4002873875 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d30195c30 con 0x7f9d30105560 2024-03-28T05:34:21.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.012+0000 7f9d35cd4640 1 --2- 172.21.15.23:0/4002873875 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d30105560 0x7f9d30195410 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f9d2802c660 tx=0x7f9d28035d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:21.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.012+0000 7f9d267fc640 1 -- 172.21.15.23:0/4002873875 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d28039040 con 0x7f9d30105560 2024-03-28T05:34:21.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.012+0000 7f9d37737640 1 -- 172.21.15.23:0/4002873875 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d301b3420 con 0x7f9d30105560 2024-03-28T05:34:21.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.012+0000 7f9d37737640 1 -- 172.21.15.23:0/4002873875 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d301b38a0 con 0x7f9d30105560 2024-03-28T05:34:21.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.016+0000 7f9d267fc640 1 -- 172.21.15.23:0/4002873875 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d28044070 con 0x7f9d30105560 2024-03-28T05:34:21.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.016+0000 7f9d267fc640 1 -- 172.21.15.23:0/4002873875 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d280043c0 con 0x7f9d30105560 2024-03-28T05:34:21.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.016+0000 7f9d267fc640 1 -- 172.21.15.23:0/4002873875 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f9d28004620 con 0x7f9d30105560 2024-03-28T05:34:21.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.019+0000 7f9d267fc640 1 --2- 172.21.15.23:0/4002873875 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f9d1c0a2a10 0x7f9d1c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:21.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.019+0000 7f9d267fc640 1 -- 172.21.15.23:0/4002873875 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9611+0+0 (secure 0 0 0) 0x7f9d280f2a70 con 0x7f9d30105560 2024-03-28T05:34:21.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.021+0000 7f9d37737640 1 -- 172.21.15.23:0/4002873875 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d14005290 con 0x7f9d30105560 2024-03-28T05:34:21.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.027+0000 7f9d354d3640 1 --2- 172.21.15.23:0/4002873875 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f9d1c0a2a10 0x7f9d1c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:21.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.033+0000 7f9d267fc640 1 -- 172.21.15.23:0/4002873875 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9d280bf520 con 0x7f9d30105560 2024-03-28T05:34:21.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.033+0000 7f9d354d3640 1 --2- 172.21.15.23:0/4002873875 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f9d1c0a2a10 0x7f9d1c0a4ed0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9d30068350 tx=0x7f9d2c006bd0 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:21.117 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 28 05:34:20 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-b[65800]: ignoring --setuser ceph since I am not root 2024-03-28T05:34:21.117 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 28 05:34:20 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-b[65800]: ignoring --setgroup ceph since I am not root 2024-03-28T05:34:21.117 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 28 05:34:20 smithi098 ceph-mds[65804]: -- 172.21.15.98:0/230066116 <== mon.2 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561cd82cfa40 con 0x561cd82a2c00 2024-03-28T05:34:21.117 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 28 05:34:20 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-b[65800]: starting mds.b at 2024-03-28T05:34:21.244 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 28 05:34:20 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-i[68132]: ignoring --setuser ceph since I am not root 2024-03-28T05:34:21.244 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 28 05:34:20 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-i[68132]: ignoring --setgroup ceph since I am not root 2024-03-28T05:34:21.244 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 28 05:34:20 smithi113 ceph-mds[68136]: -- 172.21.15.113:0/2367721766 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x563e58d464e0 con 0x563e58d1cc00 2024-03-28T05:34:21.244 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 28 05:34:20 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-i[68132]: starting mds.i at 2024-03-28T05:34:21.282 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:20 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-a[31950]: 2024-03-28T05:34:20.784+0000 7f33d5361700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T05:34:21.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.486+0000 7f9d37737640 1 -- 172.21.15.23:0/4002873875 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9d14005850 con 0x7f9d30105560 2024-03-28T05:34:21.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:21 smithi098 ceph-mon[36087]: pgmap v335: 129 pgs: 17 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 58 active+clean; 234 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 307 B/s wr, 2 op/s 2024-03-28T05:34:21.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:21 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/274240458' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T05:34:21.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:21 smithi098 ceph-mon[36087]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T05:34:21.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:21 smithi098 ceph-mon[36087]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T05:34:21.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:21 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/274240458' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T05:34:21.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:21 smithi098 ceph-mon[36087]: osdmap e104: 12 total, 12 up, 12 in 2024-03-28T05:34:21.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:21 smithi098 ceph-mon[36087]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-03-28T05:34:21.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:21 smithi113 ceph-mon[37164]: pgmap v335: 129 pgs: 17 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 58 active+clean; 234 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 307 B/s wr, 2 op/s 2024-03-28T05:34:21.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:21 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/274240458' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T05:34:21.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:21 smithi113 ceph-mon[37164]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T05:34:21.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:21 smithi113 ceph-mon[37164]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T05:34:21.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:21 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/274240458' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T05:34:21.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:21 smithi113 ceph-mon[37164]: osdmap e104: 12 total, 12 up, 12 in 2024-03-28T05:34:21.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:21 smithi113 ceph-mon[37164]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-03-28T05:34:21.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:21 smithi023 ceph-mon[31954]: pgmap v335: 129 pgs: 17 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 58 active+clean; 234 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 307 B/s wr, 2 op/s 2024-03-28T05:34:21.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:21 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/274240458' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T05:34:21.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:21 smithi023 ceph-mon[31954]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T05:34:21.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:21 smithi023 ceph-mon[31954]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T05:34:21.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:21 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/274240458' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T05:34:21.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:21 smithi023 ceph-mon[31954]: osdmap e104: 12 total, 12 up, 12 in 2024-03-28T05:34:21.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:21 smithi023 ceph-mon[31954]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-03-28T05:34:21.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.794+0000 7f9d267fc640 1 -- 172.21.15.23:0/4002873875 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v22) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9d280bee20 con 0x7f9d30105560 2024-03-28T05:34:21.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.798+0000 7f9d37737640 1 -- 172.21.15.23:0/4002873875 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f9d1c0a2a10 msgr2=0x7f9d1c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:21.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.798+0000 7f9d37737640 1 --2- 172.21.15.23:0/4002873875 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f9d1c0a2a10 0x7f9d1c0a4ed0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9d30068350 tx=0x7f9d2c006bd0 comp rx=0 tx=0).stop 2024-03-28T05:34:21.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.798+0000 7f9d37737640 1 -- 172.21.15.23:0/4002873875 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d30105560 msgr2=0x7f9d30195410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:21.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.798+0000 7f9d37737640 1 --2- 172.21.15.23:0/4002873875 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d30105560 0x7f9d30195410 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f9d2802c660 tx=0x7f9d28035d80 comp rx=0 tx=0).stop 2024-03-28T05:34:21.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.798+0000 7f9d37737640 1 -- 172.21.15.23:0/4002873875 shutdown_connections 2024-03-28T05:34:21.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.798+0000 7f9d37737640 1 --2- 172.21.15.23:0/4002873875 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d30105560 0x7f9d30195410 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:21.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.798+0000 7f9d37737640 1 --2- 172.21.15.23:0/4002873875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d30100f00 0x7f9d30194ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:21.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.798+0000 7f9d37737640 1 --2- 172.21.15.23:0/4002873875 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f9d1c0a2a10 0x7f9d1c0a4ed0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:21.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.798+0000 7f9d37737640 1 --2- 172.21.15.23:0/4002873875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d30100540 0x7f9d3019cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:21.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.798+0000 7f9d37737640 1 -- 172.21.15.23:0/4002873875 >> 172.21.15.23:0/4002873875 conn(0x7f9d30077630 msgr2=0x7f9d300fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:21.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.799+0000 7f9d37737640 1 -- 172.21.15.23:0/4002873875 shutdown_connections 2024-03-28T05:34:21.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:21.799+0000 7f9d37737640 1 -- 172.21.15.23:0/4002873875 wait complete. 2024-03-28T05:34:21.817 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-28T05:34:22.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.008+0000 7f08126a1640 1 -- 172.21.15.23:0/2200038652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f080c1059d0 msgr2=0x7f080c105e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:22.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.008+0000 7f08126a1640 1 --2- 172.21.15.23:0/2200038652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f080c1059d0 0x7f080c105e60 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0804009be0 tx=0x7f080402bfa0 comp rx=0 tx=0).stop 2024-03-28T05:34:22.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.009+0000 7f08126a1640 1 -- 172.21.15.23:0/2200038652 shutdown_connections 2024-03-28T05:34:22.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.009+0000 7f08126a1640 1 --2- 172.21.15.23:0/2200038652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f080c1059d0 0x7f080c105e60 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:22.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.009+0000 7f08126a1640 1 --2- 172.21.15.23:0/2200038652 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f080c100fd0 0x7f080c105490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:22.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.009+0000 7f08126a1640 1 --2- 172.21.15.23:0/2200038652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f080c1006b0 0x7f080c100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:22.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.009+0000 7f08126a1640 1 -- 172.21.15.23:0/2200038652 >> 172.21.15.23:0/2200038652 conn(0x7f080c077750 msgr2=0x7f080c077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:22.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.011+0000 7f08126a1640 1 -- 172.21.15.23:0/2200038652 shutdown_connections 2024-03-28T05:34:22.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.012+0000 7f08126a1640 1 -- 172.21.15.23:0/2200038652 wait complete. 2024-03-28T05:34:22.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.012+0000 7f08126a1640 1 Processor -- start 2024-03-28T05:34:22.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.013+0000 7f08126a1640 1 -- start start 2024-03-28T05:34:22.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.014+0000 7f08126a1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f080c1006b0 0x7f080c19e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:22.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.014+0000 7f08126a1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f080c100fd0 0x7f080c19ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:22.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.014+0000 7f08126a1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f080c1059d0 0x7f080c1a2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:22.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.014+0000 7f08126a1640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c1180b0 con 0x7f080c1006b0 2024-03-28T05:34:22.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.014+0000 7f08126a1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c117f30 con 0x7f080c100fd0 2024-03-28T05:34:22.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.014+0000 7f08126a1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c118230 con 0x7f080c1059d0 2024-03-28T05:34:22.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.015+0000 7f080b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f080c100fd0 0x7f080c19ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:22.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.015+0000 7f080b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f080c100fd0 0x7f080c19ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.23:42266/0 (socket says 172.21.15.23:42266) 2024-03-28T05:34:22.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.015+0000 7f080b7fe640 1 -- 172.21.15.23:0/2569999624 learned_addr learned my addr 172.21.15.23:0/2569999624 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:34:22.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.015+0000 7f0810c3e640 1 --2- 172.21.15.23:0/2569999624 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f080c1059d0 0x7f080c1a2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:22.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.015+0000 7f080bfff640 1 --2- 172.21.15.23:0/2569999624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f080c1006b0 0x7f080c19e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:22.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.016+0000 7f0810c3e640 1 -- 172.21.15.23:0/2569999624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f080c100fd0 msgr2=0x7f080c19ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:22.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.016+0000 7f0810c3e640 1 --2- 172.21.15.23:0/2569999624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f080c100fd0 0x7f080c19ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:22.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.016+0000 7f0810c3e640 1 -- 172.21.15.23:0/2569999624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f080c1006b0 msgr2=0x7f080c19e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:22.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.016+0000 7f0810c3e640 1 --2- 172.21.15.23:0/2569999624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f080c1006b0 0x7f080c19e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:22.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.016+0000 7f0810c3e640 1 -- 172.21.15.23:0/2569999624 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080c1a34d0 con 0x7f080c1059d0 2024-03-28T05:34:22.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.016+0000 7f080b7fe640 1 --2- 172.21.15.23:0/2569999624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f080c100fd0 0x7f080c19ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:34:22.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.018+0000 7f0810c3e640 1 --2- 172.21.15.23:0/2569999624 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f080c1059d0 0x7f080c1a2dd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f080402c4e0 tx=0x7f0804004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:22.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.019+0000 7f08097fa640 1 -- 172.21.15.23:0/2569999624 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0804043020 con 0x7f080c1059d0 2024-03-28T05:34:22.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.019+0000 7f08097fa640 1 -- 172.21.15.23:0/2569999624 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f08040045f0 con 0x7f080c1059d0 2024-03-28T05:34:22.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.019+0000 7f08097fa640 1 -- 172.21.15.23:0/2569999624 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080403e690 con 0x7f080c1059d0 2024-03-28T05:34:22.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.020+0000 7f08126a1640 1 -- 172.21.15.23:0/2569999624 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f080c1a3760 con 0x7f080c1059d0 2024-03-28T05:34:22.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.020+0000 7f08126a1640 1 -- 172.21.15.23:0/2569999624 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f080c106a60 con 0x7f080c1059d0 2024-03-28T05:34:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.025+0000 7f08097fa640 1 -- 172.21.15.23:0/2569999624 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0804046020 con 0x7f080c1059d0 2024-03-28T05:34:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.025+0000 7f08097fa640 1 --2- 172.21.15.23:0/2569999624 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f07f40a2aa0 0x7f07f40a4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.026+0000 7f08097fa640 1 -- 172.21.15.23:0/2569999624 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9611+0+0 (secure 0 0 0) 0x7f08040f14a0 con 0x7f080c1059d0 2024-03-28T05:34:22.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.026+0000 7f08126a1640 1 -- 172.21.15.23:0/2569999624 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f080c069010 con 0x7f080c1059d0 2024-03-28T05:34:22.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.028+0000 7f080bfff640 1 --2- 172.21.15.23:0/2569999624 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f07f40a2aa0 0x7f07f40a4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:22.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.031+0000 7f080bfff640 1 --2- 172.21.15.23:0/2569999624 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f07f40a2aa0 0x7f07f40a4f60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f07fc005fd0 tx=0x7f07fc009260 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:22.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.040+0000 7f08097fa640 1 -- 172.21.15.23:0/2569999624 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f08040047a0 con 0x7f080c1059d0 2024-03-28T05:34:22.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:22.408+0000 7f08126a1640 1 -- 172.21.15.23:0/2569999624 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f080c19f860 con 0x7f080c1059d0 2024-03-28T05:34:22.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:22 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/4002873875' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T05:34:22.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:22 smithi098 ceph-mon[36087]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T05:34:22.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:22 smithi098 ceph-mon[36087]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T05:34:22.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:22 smithi098 ceph-mon[36087]: Cluster is now healthy 2024-03-28T05:34:22.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:22 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/4002873875' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T05:34:22.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:22 smithi098 ceph-mon[36087]: mds.? [v2:172.21.15.98:6838/322738162,v1:172.21.15.98:6839/322738162] up:boot 2024-03-28T05:34:22.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:22 smithi098 ceph-mon[36087]: mds.? [v2:172.21.15.113:6838/618794319,v1:172.21.15.113:6839/618794319] up:boot 2024-03-28T05:34:22.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:22 smithi098 ceph-mon[36087]: fsmap 12 up:standby 2024-03-28T05:34:22.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:22 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T05:34:22.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:22 smithi098 ceph-mon[36087]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T05:34:22.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:22 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/4002873875' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T05:34:22.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:22 smithi113 ceph-mon[37164]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T05:34:22.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:22 smithi113 ceph-mon[37164]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T05:34:22.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:22 smithi113 ceph-mon[37164]: Cluster is now healthy 2024-03-28T05:34:22.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:22 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/4002873875' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T05:34:22.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:22 smithi113 ceph-mon[37164]: mds.? [v2:172.21.15.98:6838/322738162,v1:172.21.15.98:6839/322738162] up:boot 2024-03-28T05:34:22.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:22 smithi113 ceph-mon[37164]: mds.? [v2:172.21.15.113:6838/618794319,v1:172.21.15.113:6839/618794319] up:boot 2024-03-28T05:34:22.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:22 smithi113 ceph-mon[37164]: fsmap 12 up:standby 2024-03-28T05:34:22.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:22 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T05:34:22.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:22 smithi113 ceph-mon[37164]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T05:34:22.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:22 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/4002873875' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T05:34:22.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:22 smithi023 ceph-mon[31954]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T05:34:22.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:22 smithi023 ceph-mon[31954]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T05:34:22.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:22 smithi023 ceph-mon[31954]: Cluster is now healthy 2024-03-28T05:34:22.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:22 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/4002873875' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T05:34:22.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:22 smithi023 ceph-mon[31954]: mds.? [v2:172.21.15.98:6838/322738162,v1:172.21.15.98:6839/322738162] up:boot 2024-03-28T05:34:22.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:22 smithi023 ceph-mon[31954]: mds.? [v2:172.21.15.113:6838/618794319,v1:172.21.15.113:6839/618794319] up:boot 2024-03-28T05:34:22.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:22 smithi023 ceph-mon[31954]: fsmap 12 up:standby 2024-03-28T05:34:22.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:22 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T05:34:22.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:22 smithi023 ceph-mon[31954]: from='mgr.15402 172.21.15.23:0/3902536487' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T05:34:23.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.317+0000 7f08097fa640 1 -- 172.21.15.23:0/2569999624 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v105) v1 ==== 181+0+0 (secure 0 0 0) 0x7f08040bde70 con 0x7f080c1059d0 2024-03-28T05:34:23.317 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_metadata' removed 2024-03-28T05:34:23.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.320+0000 7f08126a1640 1 -- 172.21.15.23:0/2569999624 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f07f40a2aa0 msgr2=0x7f07f40a4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:23.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.320+0000 7f08126a1640 1 --2- 172.21.15.23:0/2569999624 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f07f40a2aa0 0x7f07f40a4f60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f07fc005fd0 tx=0x7f07fc009260 comp rx=0 tx=0).stop 2024-03-28T05:34:23.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.320+0000 7f08126a1640 1 -- 172.21.15.23:0/2569999624 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f080c1059d0 msgr2=0x7f080c1a2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:23.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.320+0000 7f08126a1640 1 --2- 172.21.15.23:0/2569999624 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f080c1059d0 0x7f080c1a2dd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f080402c4e0 tx=0x7f0804004270 comp rx=0 tx=0).stop 2024-03-28T05:34:23.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.321+0000 7f08126a1640 1 -- 172.21.15.23:0/2569999624 shutdown_connections 2024-03-28T05:34:23.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.321+0000 7f08126a1640 1 --2- 172.21.15.23:0/2569999624 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f080c1059d0 0x7f080c1a2dd0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:23.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.321+0000 7f08126a1640 1 --2- 172.21.15.23:0/2569999624 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f07f40a2aa0 0x7f07f40a4f60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:23.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.321+0000 7f08126a1640 1 --2- 172.21.15.23:0/2569999624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f080c100fd0 0x7f080c19ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:23.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.321+0000 7f08126a1640 1 --2- 172.21.15.23:0/2569999624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f080c1006b0 0x7f080c19e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:23.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.321+0000 7f08126a1640 1 -- 172.21.15.23:0/2569999624 >> 172.21.15.23:0/2569999624 conn(0x7f080c077750 msgr2=0x7f080c103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:23.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.321+0000 7f08126a1640 1 -- 172.21.15.23:0/2569999624 shutdown_connections 2024-03-28T05:34:23.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.321+0000 7f08126a1640 1 -- 172.21.15.23:0/2569999624 wait complete. 2024-03-28T05:34:23.341 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-28T05:34:23.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.528+0000 7f5e4c481640 1 -- 172.21.15.23:0/2658148322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e4411ae80 msgr2=0x7f5e4411d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:23.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.528+0000 7f5e4c481640 1 --2- 172.21.15.23:0/2658148322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e4411ae80 0x7f5e4411d270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c009be0 tx=0x7f5e3c02bef0 comp rx=0 tx=0).stop 2024-03-28T05:34:23.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.528+0000 7f5e4c481640 1 -- 172.21.15.23:0/2658148322 shutdown_connections 2024-03-28T05:34:23.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.528+0000 7f5e4c481640 1 --2- 172.21.15.23:0/2658148322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e4411ae80 0x7f5e4411d270 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:23.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.528+0000 7f5e4c481640 1 --2- 172.21.15.23:0/2658148322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e4410def0 0x7f5e4410e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:23.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.528+0000 7f5e4c481640 1 --2- 172.21.15.23:0/2658148322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e4410fec0 0x7f5e4410d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:23.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.528+0000 7f5e4c481640 1 -- 172.21.15.23:0/2658148322 >> 172.21.15.23:0/2658148322 conn(0x7f5e4406c780 msgr2=0x7f5e4406cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:23.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.530+0000 7f5e4c481640 1 -- 172.21.15.23:0/2658148322 shutdown_connections 2024-03-28T05:34:23.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.530+0000 7f5e4c481640 1 -- 172.21.15.23:0/2658148322 wait complete. 2024-03-28T05:34:23.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.530+0000 7f5e4c481640 1 Processor -- start 2024-03-28T05:34:23.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.531+0000 7f5e4c481640 1 -- start start 2024-03-28T05:34:23.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.531+0000 7f5e4c481640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e4410def0 0x7f5e4411a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:23.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.531+0000 7f5e4c481640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e4410fec0 0x7f5e44116550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:23.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.531+0000 7f5e4c481640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e4411ae80 0x7f5e44116a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:23.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.531+0000 7f5e4c481640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e44113040 con 0x7f5e4410fec0 2024-03-28T05:34:23.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.531+0000 7f5e4c481640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e44112ec0 con 0x7f5e4410def0 2024-03-28T05:34:23.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.531+0000 7f5e4c481640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e441131c0 con 0x7f5e4411ae80 2024-03-28T05:34:23.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.532+0000 7f5e4a21d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e4410def0 0x7f5e4411a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:23.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.532+0000 7f5e4a21d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e4410def0 0x7f5e4411a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.23:45168/0 (socket says 172.21.15.23:45168) 2024-03-28T05:34:23.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.532+0000 7f5e4a21d640 1 -- 172.21.15.23:0/3669023962 learned_addr learned my addr 172.21.15.23:0/3669023962 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:34:23.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.532+0000 7f5e4aa1e640 1 --2- 172.21.15.23:0/3669023962 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e4411ae80 0x7f5e44116a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:23.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.533+0000 7f5e49a1c640 1 --2- 172.21.15.23:0/3669023962 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e4410fec0 0x7f5e44116550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:23.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.533+0000 7f5e4a21d640 1 -- 172.21.15.23:0/3669023962 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e4411ae80 msgr2=0x7f5e44116a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:23.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.533+0000 7f5e4a21d640 1 --2- 172.21.15.23:0/3669023962 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e4411ae80 0x7f5e44116a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:23.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.533+0000 7f5e4a21d640 1 -- 172.21.15.23:0/3669023962 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e4410fec0 msgr2=0x7f5e44116550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:23.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.533+0000 7f5e4a21d640 1 --2- 172.21.15.23:0/3669023962 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e4410fec0 0x7f5e44116550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:23.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.533+0000 7f5e4a21d640 1 -- 172.21.15.23:0/3669023962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e44117050 con 0x7f5e4410def0 2024-03-28T05:34:23.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.534+0000 7f5e4a21d640 1 --2- 172.21.15.23:0/3669023962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e4410def0 0x7f5e4411a4c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5e4000e9f0 tx=0x7f5e4000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:23.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.534+0000 7f5e3b7fe640 1 -- 172.21.15.23:0/3669023962 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4000ccc0 con 0x7f5e4410def0 2024-03-28T05:34:23.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.534+0000 7f5e3b7fe640 1 -- 172.21.15.23:0/3669023962 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e40010910 con 0x7f5e4410def0 2024-03-28T05:34:23.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.534+0000 7f5e3b7fe640 1 -- 172.21.15.23:0/3669023962 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e40018960 con 0x7f5e4410def0 2024-03-28T05:34:23.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.534+0000 7f5e4c481640 1 -- 172.21.15.23:0/3669023962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e44117340 con 0x7f5e4410def0 2024-03-28T05:34:23.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.534+0000 7f5e4c481640 1 -- 172.21.15.23:0/3669023962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e441aaef0 con 0x7f5e4410def0 2024-03-28T05:34:23.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.539+0000 7f5e4c481640 1 -- 172.21.15.23:0/3669023962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e4406da20 con 0x7f5e4410def0 2024-03-28T05:34:23.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.539+0000 7f5e3b7fe640 1 -- 172.21.15.23:0/3669023962 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f5e40010ac0 con 0x7f5e4410def0 2024-03-28T05:34:23.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.540+0000 7f5e3b7fe640 1 --2- 172.21.15.23:0/3669023962 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f5e300a2a70 0x7f5e300a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:23.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.541+0000 7f5e3b7fe640 1 -- 172.21.15.23:0/3669023962 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9086+0+0 (secure 0 0 0) 0x7f5e400cf660 con 0x7f5e4410def0 2024-03-28T05:34:23.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.541+0000 7f5e49a1c640 1 --2- 172.21.15.23:0/3669023962 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f5e300a2a70 0x7f5e300a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:23.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.542+0000 7f5e49a1c640 1 --2- 172.21.15.23:0/3669023962 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f5e300a2a70 0x7f5e300a4f30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5e44117d10 tx=0x7f5e3401a270 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:23.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.544+0000 7f5e3b7fe640 1 -- 172.21.15.23:0/3669023962 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5e4001f340 con 0x7f5e4410def0 2024-03-28T05:34:23.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:23 smithi098 ceph-mon[36087]: pgmap v337: 129 pgs: 17 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 58 active+clean; 231 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-28T05:34:23.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:23 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/2569999624' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T05:34:23.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:23 smithi098 ceph-mon[36087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T05:34:23.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:23 smithi113 ceph-mon[37164]: pgmap v337: 129 pgs: 17 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 58 active+clean; 231 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-28T05:34:23.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:23 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/2569999624' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T05:34:23.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:23 smithi113 ceph-mon[37164]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T05:34:23.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:23 smithi023 ceph-mon[31954]: pgmap v337: 129 pgs: 17 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 58 active+clean; 231 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-28T05:34:23.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:23 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/2569999624' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T05:34:23.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:23 smithi023 ceph-mon[31954]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T05:34:23.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:23.898+0000 7f5e4c481640 1 -- 172.21.15.23:0/3669023962 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5e4410afa0 con 0x7f5e4410def0 2024-03-28T05:34:24.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.340+0000 7f5e3b7fe640 1 -- 172.21.15.23:0/3669023962 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v106) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5e4009c130 con 0x7f5e4410def0 2024-03-28T05:34:24.340 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data' removed 2024-03-28T05:34:24.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.343+0000 7f5e4c481640 1 -- 172.21.15.23:0/3669023962 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f5e300a2a70 msgr2=0x7f5e300a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:24.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.343+0000 7f5e4c481640 1 --2- 172.21.15.23:0/3669023962 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f5e300a2a70 0x7f5e300a4f30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5e44117d10 tx=0x7f5e3401a270 comp rx=0 tx=0).stop 2024-03-28T05:34:24.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.343+0000 7f5e4c481640 1 -- 172.21.15.23:0/3669023962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e4410def0 msgr2=0x7f5e4411a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:24.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.343+0000 7f5e4c481640 1 --2- 172.21.15.23:0/3669023962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e4410def0 0x7f5e4411a4c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5e4000e9f0 tx=0x7f5e4000edf0 comp rx=0 tx=0).stop 2024-03-28T05:34:24.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.343+0000 7f5e4c481640 1 -- 172.21.15.23:0/3669023962 shutdown_connections 2024-03-28T05:34:24.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.343+0000 7f5e4c481640 1 --2- 172.21.15.23:0/3669023962 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e4411ae80 0x7f5e44116a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:24.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.343+0000 7f5e4c481640 1 --2- 172.21.15.23:0/3669023962 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e4410fec0 0x7f5e44116550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:24.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.343+0000 7f5e4c481640 1 --2- 172.21.15.23:0/3669023962 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f5e300a2a70 0x7f5e300a4f30 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5e44117d10 tx=0x7f5e3401a270 comp rx=0 tx=0).stop 2024-03-28T05:34:24.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.343+0000 7f5e4c481640 1 --2- 172.21.15.23:0/3669023962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e4410def0 0x7f5e4411a4c0 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5e4000e9f0 tx=0x7f5e4000edf0 comp rx=0 tx=0).stop 2024-03-28T05:34:24.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.343+0000 7f5e4c481640 1 -- 172.21.15.23:0/3669023962 >> 172.21.15.23:0/3669023962 conn(0x7f5e4406c780 msgr2=0x7f5e4411c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:24.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.344+0000 7f5e4c481640 1 -- 172.21.15.23:0/3669023962 shutdown_connections 2024-03-28T05:34:24.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.344+0000 7f5e4c481640 1 -- 172.21.15.23:0/3669023962 wait complete. 2024-03-28T05:34:24.362 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-28T05:34:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.535+0000 7f36a3fff640 1 -- 172.21.15.23:0/3174292292 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a802ccd0 con 0x7f36ac100440 2024-03-28T05:34:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.536+0000 7f36b350c640 1 -- 172.21.15.23:0/3174292292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36ac100440 msgr2=0x7f36ac100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.536+0000 7f36b350c640 1 --2- 172.21.15.23:0/3174292292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36ac100440 0x7f36ac100820 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f36a8009900 tx=0x7f36a802bc80 comp rx=0 tx=0).stop 2024-03-28T05:34:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.537+0000 7f36b350c640 1 -- 172.21.15.23:0/3174292292 shutdown_connections 2024-03-28T05:34:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.537+0000 7f36b350c640 1 --2- 172.21.15.23:0/3174292292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36ac10dff0 0x7f36ac1103e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:24.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.537+0000 7f36b350c640 1 --2- 172.21.15.23:0/3174292292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36ac100d60 0x7f36ac10dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:24.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.537+0000 7f36b350c640 1 --2- 172.21.15.23:0/3174292292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36ac100440 0x7f36ac100820 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:24.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.537+0000 7f36b350c640 1 -- 172.21.15.23:0/3174292292 >> 172.21.15.23:0/3174292292 conn(0x7f36ac077750 msgr2=0x7f36ac077b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:24.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.538+0000 7f36b350c640 1 -- 172.21.15.23:0/3174292292 shutdown_connections 2024-03-28T05:34:24.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.538+0000 7f36b350c640 1 -- 172.21.15.23:0/3174292292 wait complete. 2024-03-28T05:34:24.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.539+0000 7f36b350c640 1 Processor -- start 2024-03-28T05:34:24.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.539+0000 7f36b350c640 1 -- start start 2024-03-28T05:34:24.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.539+0000 7f36b350c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36ac100440 0x7f36ac19a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:24.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.540+0000 7f36b350c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36ac100d60 0x7f36ac19a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:24.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.540+0000 7f36b350c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36ac10dff0 0x7f36ac19e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:24.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.540+0000 7f36b350c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ac113d70 con 0x7f36ac10dff0 2024-03-28T05:34:24.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.540+0000 7f36b12a8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36ac100440 0x7f36ac19a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:24.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.540+0000 7f36b12a8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36ac100440 0x7f36ac19a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.23:35464/0 (socket says 172.21.15.23:35464) 2024-03-28T05:34:24.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.540+0000 7f36b12a8640 1 -- 172.21.15.23:0/3707011944 learned_addr learned my addr 172.21.15.23:0/3707011944 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T05:34:24.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.540+0000 7f36b0aa7640 1 --2- 172.21.15.23:0/3707011944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36ac100d60 0x7f36ac19a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:24.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.540+0000 7f36b1aa9640 1 --2- 172.21.15.23:0/3707011944 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36ac10dff0 0x7f36ac19e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:24.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.540+0000 7f36b350c640 1 -- 172.21.15.23:0/3707011944 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ac113bf0 con 0x7f36ac100d60 2024-03-28T05:34:24.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.540+0000 7f36b350c640 1 -- 172.21.15.23:0/3707011944 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ac113ef0 con 0x7f36ac100440 2024-03-28T05:34:24.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.541+0000 7f36b0aa7640 1 -- 172.21.15.23:0/3707011944 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36ac100440 msgr2=0x7f36ac19a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:24.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.541+0000 7f36b0aa7640 1 --2- 172.21.15.23:0/3707011944 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36ac100440 0x7f36ac19a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:24.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.541+0000 7f36b0aa7640 1 -- 172.21.15.23:0/3707011944 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36ac10dff0 msgr2=0x7f36ac19e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:24.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.541+0000 7f36b0aa7640 1 --2- 172.21.15.23:0/3707011944 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36ac10dff0 0x7f36ac19e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:24.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.541+0000 7f36b0aa7640 1 -- 172.21.15.23:0/3707011944 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ac19f120 con 0x7f36ac100d60 2024-03-28T05:34:24.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.541+0000 7f36b12a8640 1 --2- 172.21.15.23:0/3707011944 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36ac100440 0x7f36ac19a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T05:34:24.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.541+0000 7f36b0aa7640 1 --2- 172.21.15.23:0/3707011944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36ac100d60 0x7f36ac19a670 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f36a400bd50 tx=0x7f36a400be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:24.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.541+0000 7f36a27fc640 1 -- 172.21.15.23:0/3707011944 <== mon.2 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a4019070 con 0x7f36ac100d60 2024-03-28T05:34:24.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.542+0000 7f36a27fc640 1 -- 172.21.15.23:0/3707011944 <== mon.2 v2:172.21.15.98:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36a4014920 con 0x7f36ac100d60 2024-03-28T05:34:24.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.542+0000 7f36a27fc640 1 -- 172.21.15.23:0/3707011944 <== mon.2 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a4012990 con 0x7f36ac100d60 2024-03-28T05:34:24.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.542+0000 7f36b350c640 1 -- 172.21.15.23:0/3707011944 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36ac19f410 con 0x7f36ac100d60 2024-03-28T05:34:24.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.542+0000 7f36b350c640 1 -- 172.21.15.23:0/3707011944 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36ac1a6ca0 con 0x7f36ac100d60 2024-03-28T05:34:24.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.544+0000 7f36b350c640 1 -- 172.21.15.23:0/3707011944 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36ac068490 con 0x7f36ac100d60 2024-03-28T05:34:24.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.545+0000 7f36a27fc640 1 -- 172.21.15.23:0/3707011944 <== mon.2 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f36a401c070 con 0x7f36ac100d60 2024-03-28T05:34:24.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.546+0000 7f36a27fc640 1 --2- 172.21.15.23:0/3707011944 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f36980a2a70 0x7f36980a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T05:34:24.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.546+0000 7f36a27fc640 1 -- 172.21.15.23:0/3707011944 <== mon.2 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8498+0+0 (secure 0 0 0) 0x7f36a40ce700 con 0x7f36ac100d60 2024-03-28T05:34:24.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.546+0000 7f36b12a8640 1 --2- 172.21.15.23:0/3707011944 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f36980a2a70 0x7f36980a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T05:34:24.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.547+0000 7f36b12a8640 1 --2- 172.21.15.23:0/3707011944 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f36980a2a70 0x7f36980a4f30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f36a8036040 tx=0x7f36a8000bc0 comp rx=0 tx=0).ready entity=mgr.15402 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T05:34:24.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.554+0000 7f36a27fc640 1 -- 172.21.15.23:0/3707011944 <== mon.2 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f36a409b520 con 0x7f36ac100d60 2024-03-28T05:34:24.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:24 smithi098 ceph-mon[36087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T05:34:24.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:24 smithi098 ceph-mon[36087]: osdmap e105: 12 total, 12 up, 12 in 2024-03-28T05:34:24.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:24 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3669023962' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T05:34:24.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:24 smithi098 ceph-mon[36087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T05:34:24.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:24 smithi113 ceph-mon[37164]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T05:34:24.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:24 smithi113 ceph-mon[37164]: osdmap e105: 12 total, 12 up, 12 in 2024-03-28T05:34:24.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:24 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3669023962' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T05:34:24.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:24 smithi113 ceph-mon[37164]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T05:34:24.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:24 smithi023 ceph-mon[31954]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T05:34:24.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:24 smithi023 ceph-mon[31954]: osdmap e105: 12 total, 12 up, 12 in 2024-03-28T05:34:24.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:24 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3669023962' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T05:34:24.782 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:24 smithi023 ceph-mon[31954]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T05:34:24.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:24.905+0000 7f36b350c640 1 -- 172.21.15.23:0/3707011944 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f36ac1a6ed0 con 0x7f36ac100d60 2024-03-28T05:34:25.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:25.349+0000 7f36a27fc640 1 -- 172.21.15.23:0/3707011944 <== mon.2 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v107) v1 ==== 178+0+0 (secure 0 0 0) 0x7f36a409ae20 con 0x7f36ac100d60 2024-03-28T05:34:25.349 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data_ec' removed 2024-03-28T05:34:25.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:25.352+0000 7f36b350c640 1 -- 172.21.15.23:0/3707011944 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f36980a2a70 msgr2=0x7f36980a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:25.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:25.352+0000 7f36b350c640 1 --2- 172.21.15.23:0/3707011944 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f36980a2a70 0x7f36980a4f30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f36a8036040 tx=0x7f36a8000bc0 comp rx=0 tx=0).stop 2024-03-28T05:34:25.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:25.353+0000 7f36b350c640 1 -- 172.21.15.23:0/3707011944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36ac100d60 msgr2=0x7f36ac19a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T05:34:25.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:25.353+0000 7f36b350c640 1 --2- 172.21.15.23:0/3707011944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36ac100d60 0x7f36ac19a670 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f36a400bd50 tx=0x7f36a400be40 comp rx=0 tx=0).stop 2024-03-28T05:34:25.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:25.353+0000 7f36b350c640 1 -- 172.21.15.23:0/3707011944 shutdown_connections 2024-03-28T05:34:25.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:25.353+0000 7f36b350c640 1 --2- 172.21.15.23:0/3707011944 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36ac10dff0 0x7f36ac19e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:25.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:25.353+0000 7f36b350c640 1 --2- 172.21.15.23:0/3707011944 >> [v2:172.21.15.23:6800/1385031613,v1:172.21.15.23:6801/1385031613] conn(0x7f36980a2a70 0x7f36980a4f30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:25.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:25.353+0000 7f36b350c640 1 --2- 172.21.15.23:0/3707011944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36ac100d60 0x7f36ac19a670 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:25.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:25.353+0000 7f36b350c640 1 --2- 172.21.15.23:0/3707011944 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36ac100440 0x7f36ac19a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T05:34:25.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:25.353+0000 7f36b350c640 1 -- 172.21.15.23:0/3707011944 >> 172.21.15.23:0/3707011944 conn(0x7f36ac077750 msgr2=0x7f36ac10f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T05:34:25.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:25.353+0000 7f36b350c640 1 -- 172.21.15.23:0/3707011944 shutdown_connections 2024-03-28T05:34:25.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T05:34:25.353+0000 7f36b350c640 1 -- 172.21.15.23:0/3707011944 wait complete. 2024-03-28T05:34:25.371 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T05:34:25.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T05:34:25.459 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T05:34:25.508 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-28T05:34:25.508 DEBUG:teuthology.orchestra.run.smithi023:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T05:34:25.526 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T05:34:25.546 DEBUG:teuthology.orchestra.run.smithi113:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T05:34:25.567 INFO:tasks.cephadm:Stopping all daemons... 2024-03-28T05:34:25.567 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-28T05:34:25.568 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mon.a 2024-03-28T05:34:25.596 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:25 smithi023 ceph-mon[31954]: pgmap v339: 97 pgs: 17 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 26 active+clean; 212 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:34:25.596 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:25 smithi023 ceph-mon[31954]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T05:34:25.596 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:25 smithi023 ceph-mon[31954]: osdmap e106: 12 total, 12 up, 12 in 2024-03-28T05:34:25.596 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:25 smithi023 ceph-mon[31954]: from='client.? 172.21.15.23:0/3707011944' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T05:34:25.596 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:25 smithi023 ceph-mon[31954]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T05:34:25.616 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:25 smithi098 ceph-mon[36087]: pgmap v339: 97 pgs: 17 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 26 active+clean; 212 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:34:25.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:25 smithi098 ceph-mon[36087]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T05:34:25.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:25 smithi098 ceph-mon[36087]: osdmap e106: 12 total, 12 up, 12 in 2024-03-28T05:34:25.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:25 smithi098 ceph-mon[36087]: from='client.? 172.21.15.23:0/3707011944' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T05:34:25.617 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:25 smithi098 ceph-mon[36087]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T05:34:25.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:25 smithi113 ceph-mon[37164]: pgmap v339: 97 pgs: 17 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 26 active+clean; 212 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T05:34:25.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:25 smithi113 ceph-mon[37164]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T05:34:25.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:25 smithi113 ceph-mon[37164]: osdmap e106: 12 total, 12 up, 12 in 2024-03-28T05:34:25.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:25 smithi113 ceph-mon[37164]: from='client.? 172.21.15.23:0/3707011944' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T05:34:25.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:25 smithi113 ceph-mon[37164]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T05:34:25.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:25 smithi023 systemd[1]: Stopping Ceph mon.a for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:34:26.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:25 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-a[31950]: 2024-03-28T05:34:25.926+0000 7f33dab6c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:34:26.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:25 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-a[31950]: 2024-03-28T05:34:25.926+0000 7f33dab6c700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-28T05:34:26.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:25 smithi023 podman[95794]: 2024-03-28 05:34:25.98690832 +0000 UTC m=+0.186924163 container died 6ba187531ba0f3ed1e1e055f2c7ee8cb929c8df2c5d692aa1be4443803d55012 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-03-28T05:34:26.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:26 smithi023 podman[95794]: 2024-03-28 05:34:26.326792055 +0000 UTC m=+0.526807941 container cleanup 6ba187531ba0f3ed1e1e055f2c7ee8cb929c8df2c5d692aa1be4443803d55012 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-a, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-28T05:34:26.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 05:34:26 smithi023 bash[95794]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-a 2024-03-28T05:34:26.772 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mon.a.service' 2024-03-28T05:34:26.865 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:34:26.866 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-28T05:34:26.866 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-28T05:34:26.866 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mon.b 2024-03-28T05:34:27.116 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:26 smithi098 systemd[1]: Stopping Ceph mon.b for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:34:27.590 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:27 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-b[36083]: 2024-03-28T05:34:27.188+0000 7f43ce275700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:34:27.591 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:27 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-b[36083]: 2024-03-28T05:34:27.188+0000 7f43ce275700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-28T05:34:27.591 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:27 smithi098 podman[71478]: 2024-03-28 05:34:27.281690675 +0000 UTC m=+0.229388221 container died 0b0ed0d41fb2ad098a5567d430dce3fa8aa76917852a763888579c5bf36e7f41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-28T05:34:27.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:27 smithi098 podman[71478]: 2024-03-28 05:34:27.690595628 +0000 UTC m=+0.638293174 container cleanup 0b0ed0d41fb2ad098a5567d430dce3fa8aa76917852a763888579c5bf36e7f41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-28T05:34:27.867 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:27 smithi098 bash[71478]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-b 2024-03-28T05:34:28.242 INFO:journalctl@ceph.mon.b.smithi098.stdout:Mar 28 05:34:27 smithi098 podman[71492]: 2024-03-28 05:34:27.959347019 +0000 UTC m=+0.680309104 container remove 0b0ed0d41fb2ad098a5567d430dce3fa8aa76917852a763888579c5bf36e7f41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, ceph=True) 2024-03-28T05:34:28.253 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mon.b.service' 2024-03-28T05:34:28.307 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:34:28.308 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-28T05:34:28.308 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-28T05:34:28.308 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mon.c 2024-03-28T05:34:28.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:28 smithi113 systemd[1]: Stopping Ceph mon.c for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:34:29.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:28 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-c[37138]: 2024-03-28T05:34:28.828+0000 7f2059f6e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:34:29.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:28 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-c[37138]: 2024-03-28T05:34:28.828+0000 7f2059f6e700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-28T05:34:29.244 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:28 smithi113 podman[73378]: 2024-03-28 05:34:28.890806858 +0000 UTC m=+0.260979202 container died b14ccc7fcf6f229952bca333d8fa02584dcaaf49ef3f567b9a2de5096cce53ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T05:34:29.643 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:29 smithi113 podman[73378]: 2024-03-28 05:34:29.589940337 +0000 UTC m=+0.960112683 container cleanup b14ccc7fcf6f229952bca333d8fa02584dcaaf49ef3f567b9a2de5096cce53ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True) 2024-03-28T05:34:29.643 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:29 smithi113 bash[73378]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-c 2024-03-28T05:34:29.994 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 28 05:34:29 smithi113 podman[73392]: 2024-03-28 05:34:29.801870968 +0000 UTC m=+0.905732081 container remove b14ccc7fcf6f229952bca333d8fa02584dcaaf49ef3f567b9a2de5096cce53ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mon-c, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=) 2024-03-28T05:34:30.019 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mon.c.service' 2024-03-28T05:34:30.066 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:34:30.067 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-28T05:34:30.067 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-28T05:34:30.067 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mgr.x 2024-03-28T05:34:30.407 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:34:30 smithi023 systemd[1]: Stopping Ceph mgr.x for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:34:30.749 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:34:30 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x[32187]: 2024-03-28T05:34:30.406+0000 7fb17fb8e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:34:30.749 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:34:30 smithi023 podman[95915]: 2024-03-28 05:34:30.448435806 +0000 UTC m=+0.177866606 container died bb34103dde93405b5399ab26b8a68dde45e876abfc9f8e0a4498baeec3ac4686 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T05:34:31.032 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:34:30 smithi023 podman[95915]: 2024-03-28 05:34:30.849392428 +0000 UTC m=+0.578823217 container cleanup bb34103dde93405b5399ab26b8a68dde45e876abfc9f8e0a4498baeec3ac4686 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-28T05:34:31.032 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 05:34:30 smithi023 bash[95915]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-x 2024-03-28T05:34:31.282 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-28T05:34:31.320 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:34:31.321 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-28T05:34:31.321 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-28T05:34:31.321 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mgr.y 2024-03-28T05:34:31.616 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:34:31 smithi098 systemd[1]: Stopping Ceph mgr.y for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:34:32.001 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:34:31 smithi098 podman[71600]: 2024-03-28 05:34:31.67498286 +0000 UTC m=+0.151724748 container died 61998b1d7dd0ea89166036176592634b692e444bed05ef6a4d7e0cbf2f8012f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-28T05:34:32.331 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:34:32 smithi098 podman[71600]: 2024-03-28 05:34:32.113214381 +0000 UTC m=+0.589956261 container cleanup 61998b1d7dd0ea89166036176592634b692e444bed05ef6a4d7e0cbf2f8012f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-28T05:34:32.331 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:34:32 smithi098 bash[71600]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y 2024-03-28T05:34:32.588 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:34:32 smithi098 podman[71615]: 2024-03-28 05:34:32.331381806 +0000 UTC m=+0.653274034 container remove 61998b1d7dd0ea89166036176592634b692e444bed05ef6a4d7e0cbf2f8012f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-y, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-28T05:34:32.589 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 28 05:34:32 smithi098 systemd[1]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-28T05:34:32.593 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-28T05:34:32.631 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:34:32.631 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-28T05:34:32.631 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-28T05:34:32.632 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mgr.z 2024-03-28T05:34:32.976 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:34:32 smithi113 systemd[1]: Stopping Ceph mgr.z for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:34:33.244 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:34:32 smithi113 podman[73511]: 2024-03-28 05:34:32.976719749 +0000 UTC m=+0.155452227 container died a0459dda73a9ac3a3241ac82047ca0e9cf87db092f67fcfe8ea6b9e45fc231ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-28T05:34:33.583 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:34:33 smithi113 podman[73511]: 2024-03-28 05:34:33.356736303 +0000 UTC m=+0.535468733 container cleanup a0459dda73a9ac3a3241ac82047ca0e9cf87db092f67fcfe8ea6b9e45fc231ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T05:34:33.583 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:34:33 smithi113 bash[73511]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z 2024-03-28T05:34:33.810 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mgr.z.service' 2024-03-28T05:34:33.835 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:34:33 smithi113 podman[73525]: 2024-03-28 05:34:33.583619703 +0000 UTC m=+0.595548496 container remove a0459dda73a9ac3a3241ac82047ca0e9cf87db092f67fcfe8ea6b9e45fc231ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T05:34:33.835 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:34:33 smithi113 systemd[1]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-03-28T05:34:33.835 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:34:33 smithi113 systemd[1]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mgr.z.service: Failed with result 'exit-code'. 2024-03-28T05:34:33.835 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:34:33 smithi113 systemd[1]: Stopped Ceph mgr.z for bf37ebc4-ecc1-11ee-95d0-87774f69a715. 2024-03-28T05:34:33.836 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 28 05:34:33 smithi113 systemd[1]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mgr.z.service: Consumed 19.586s CPU time. 2024-03-28T05:34:33.849 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:34:33.850 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-28T05:34:33.850 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-28T05:34:33.850 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.0 2024-03-28T05:34:34.162 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:34:33 smithi023 systemd[1]: Stopping Ceph osd.0 for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:34:34.532 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:34:34 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-0[46655]: 2024-03-28T05:34:34.162+0000 7fe49baa4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:34:34.533 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:34:34 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-0[46655]: 2024-03-28T05:34:34.162+0000 7fe49baa4700 -1 osd.0 107 *** Got signal Terminated *** 2024-03-28T05:34:34.533 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:34:34 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-0[46655]: 2024-03-28T05:34:34.162+0000 7fe49baa4700 -1 osd.0 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T05:34:40.033 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:34:39 smithi023 podman[96038]: 2024-03-28 05:34:39.63567258 +0000 UTC m=+5.582848844 container died 6c59b1300d65f657a4d6c720840cdc14562eda8efdfc74a38a4d96ff6fc3ce27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-28T05:34:40.380 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:34:40 smithi023 podman[96038]: 2024-03-28 05:34:40.137739996 +0000 UTC m=+6.084916251 container cleanup 6c59b1300d65f657a4d6c720840cdc14562eda8efdfc74a38a4d96ff6fc3ce27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-28T05:34:40.380 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:34:40 smithi023 bash[96038]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-0 2024-03-28T05:34:40.713 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:34:40 smithi023 podman[96057]: 2024-03-28 05:34:40.380650511 +0000 UTC m=+0.742384854 container remove 6c59b1300d65f657a4d6c720840cdc14562eda8efdfc74a38a4d96ff6fc3ce27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-28T05:34:41.032 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:34:40 smithi023 podman[96150]: 2024-03-28 05:34:40.616642662 +0000 UTC m=+0.018224843 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:34:41.032 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:34:40 smithi023 podman[96150]: 2024-03-28 05:34:40.714930243 +0000 UTC m=+0.116512412 container create 849bb17b3fc19e078c7df285f3b132e127729b83bc312cccb7353ef7a7035ee5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T05:34:41.033 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:34:40 smithi023 podman[96150]: 2024-03-28 05:34:40.959905987 +0000 UTC m=+0.361488156 container init 849bb17b3fc19e078c7df285f3b132e127729b83bc312cccb7353ef7a7035ee5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-28T05:34:41.033 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:34:40 smithi023 podman[96150]: 2024-03-28 05:34:40.963042355 +0000 UTC m=+0.364624524 container start 849bb17b3fc19e078c7df285f3b132e127729b83bc312cccb7353ef7a7035ee5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-28T05:34:41.532 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:34:41 smithi023 podman[96150]: 2024-03-28 05:34:41.051563002 +0000 UTC m=+0.453145171 container attach 849bb17b3fc19e078c7df285f3b132e127729b83bc312cccb7353ef7a7035ee5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-28T05:34:41.532 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 05:34:41 smithi023 podman[96150]: 2024-03-28 05:34:41.257185082 +0000 UTC m=+0.658767250 container died 849bb17b3fc19e078c7df285f3b132e127729b83bc312cccb7353ef7a7035ee5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T05:34:41.875 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.0.service' 2024-03-28T05:34:41.917 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:34:41.917 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-28T05:34:41.917 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-28T05:34:41.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.1 2024-03-28T05:34:42.326 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:34:41 smithi098 systemd[1]: Stopping Ceph osd.1 for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:34:42.616 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:34:42 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-1[40683]: 2024-03-28T05:34:42.325+0000 7f88b7be2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:34:42.617 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:34:42 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-1[40683]: 2024-03-28T05:34:42.325+0000 7f88b7be2700 -1 osd.1 107 *** Got signal Terminated *** 2024-03-28T05:34:42.617 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:34:42 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-1[40683]: 2024-03-28T05:34:42.325+0000 7f88b7be2700 -1 osd.1 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T05:34:48.219 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:34:47 smithi098 podman[71724]: 2024-03-28 05:34:47.895662764 +0000 UTC m=+5.786551555 container died 0d65114419cf9eaee33bc87e3f85b45010e307ea363fdaf40fe1ab3a61913856 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T05:34:48.570 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:34:48 smithi098 podman[71724]: 2024-03-28 05:34:48.326723796 +0000 UTC m=+6.217612576 container cleanup 0d65114419cf9eaee33bc87e3f85b45010e307ea363fdaf40fe1ab3a61913856 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-28T05:34:48.570 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:34:48 smithi098 bash[71724]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-1 2024-03-28T05:34:48.867 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:34:48 smithi098 podman[71742]: 2024-03-28 05:34:48.570215784 +0000 UTC m=+0.674738103 container remove 0d65114419cf9eaee33bc87e3f85b45010e307ea363fdaf40fe1ab3a61913856 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-1, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-28T05:34:49.240 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:34:48 smithi098 podman[71833]: 2024-03-28 05:34:48.889482435 +0000 UTC m=+0.017711597 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:34:49.241 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:34:49 smithi098 podman[71833]: 2024-03-28 05:34:49.004554316 +0000 UTC m=+0.132783465 container create b4d028b70f147b635e2661f76df4181652d61f57c4fd76c06f32b9e65ef3ae7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-1-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T05:34:49.538 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:34:49 smithi098 podman[71833]: 2024-03-28 05:34:49.241026493 +0000 UTC m=+0.369255635 container init b4d028b70f147b635e2661f76df4181652d61f57c4fd76c06f32b9e65ef3ae7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T05:34:49.538 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:34:49 smithi098 podman[71833]: 2024-03-28 05:34:49.244201271 +0000 UTC m=+0.372430415 container start b4d028b70f147b635e2661f76df4181652d61f57c4fd76c06f32b9e65ef3ae7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-28T05:34:49.538 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:34:49 smithi098 podman[71833]: 2024-03-28 05:34:49.299208831 +0000 UTC m=+0.427437986 container attach b4d028b70f147b635e2661f76df4181652d61f57c4fd76c06f32b9e65ef3ae7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-28T05:34:49.867 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 28 05:34:49 smithi098 podman[71833]: 2024-03-28 05:34:49.538566853 +0000 UTC m=+0.666796005 container died b4d028b70f147b635e2661f76df4181652d61f57c4fd76c06f32b9e65ef3ae7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-1-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-28T05:34:50.308 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.1.service' 2024-03-28T05:34:50.352 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:34:50.352 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-28T05:34:50.352 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-28T05:34:50.352 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.2 2024-03-28T05:34:50.677 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:34:50 smithi113 systemd[1]: Stopping Ceph osd.2 for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:34:50.994 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:34:50 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-2[42296]: 2024-03-28T05:34:50.676+0000 7feb8282b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:34:50.994 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:34:50 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-2[42296]: 2024-03-28T05:34:50.676+0000 7feb8282b700 -1 osd.2 107 *** Got signal Terminated *** 2024-03-28T05:34:50.994 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:34:50 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-2[42296]: 2024-03-28T05:34:50.676+0000 7feb8282b700 -1 osd.2 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T05:34:56.494 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:34:56 smithi113 podman[73637]: 2024-03-28 05:34:56.194524616 +0000 UTC m=+5.638349344 container died 1a819290bcdd0dbcb6345fa28e7b9631e483b521044248f009aea1eeb091fa23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-2, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-28T05:34:56.791 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:34:56 smithi113 podman[73637]: 2024-03-28 05:34:56.518584075 +0000 UTC m=+5.962408802 container cleanup 1a819290bcdd0dbcb6345fa28e7b9631e483b521044248f009aea1eeb091fa23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-28T05:34:56.791 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:34:56 smithi113 bash[73637]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-2 2024-03-28T05:34:57.244 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:34:56 smithi113 podman[73654]: 2024-03-28 05:34:56.791295832 +0000 UTC m=+0.599302344 container remove 1a819290bcdd0dbcb6345fa28e7b9631e483b521044248f009aea1eeb091fa23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-28T05:34:57.584 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:34:57 smithi113 podman[73749]: 2024-03-28 05:34:57.176192961 +0000 UTC m=+0.017869483 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:34:57.584 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:34:57 smithi113 podman[73749]: 2024-03-28 05:34:57.281383619 +0000 UTC m=+0.123060132 container create 7a58dbf404739d3e40eb63a3579bd6adc7232862d3d509a6ecee39e07f7eef3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-28T05:34:57.585 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:34:57 smithi113 podman[73749]: 2024-03-28 05:34:57.484254585 +0000 UTC m=+0.325931106 container init 7a58dbf404739d3e40eb63a3579bd6adc7232862d3d509a6ecee39e07f7eef3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-28T05:34:57.585 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:34:57 smithi113 podman[73749]: 2024-03-28 05:34:57.488058397 +0000 UTC m=+0.329734912 container start 7a58dbf404739d3e40eb63a3579bd6adc7232862d3d509a6ecee39e07f7eef3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-28T05:34:57.994 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:34:57 smithi113 podman[73749]: 2024-03-28 05:34:57.584363993 +0000 UTC m=+0.426040518 container attach 7a58dbf404739d3e40eb63a3579bd6adc7232862d3d509a6ecee39e07f7eef3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T05:34:57.994 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 28 05:34:57 smithi113 podman[73749]: 2024-03-28 05:34:57.83456048 +0000 UTC m=+0.676237006 container died 7a58dbf404739d3e40eb63a3579bd6adc7232862d3d509a6ecee39e07f7eef3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-28T05:34:58.471 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.2.service' 2024-03-28T05:34:58.515 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:34:58.515 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-28T05:34:58.516 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-28T05:34:58.516 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.3 2024-03-28T05:34:58.822 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 05:34:58 smithi023 systemd[1]: Stopping Ceph osd.3 for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:34:59.282 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 05:34:58 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-3[53300]: 2024-03-28T05:34:58.821+0000 7f5c473e3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:34:59.282 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 05:34:58 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-3[53300]: 2024-03-28T05:34:58.821+0000 7f5c473e3700 -1 osd.3 107 *** Got signal Terminated *** 2024-03-28T05:34:59.282 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 05:34:58 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-3[53300]: 2024-03-28T05:34:58.821+0000 7f5c473e3700 -1 osd.3 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T05:35:00.532 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 05:35:00 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-3[53300]: 2024-03-28T05:35:00.099+0000 7f5c3f9cf700 -1 osd.3 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:35.848295+0000 front 2024-03-28T05:34:35.848255+0000 (oldest deadline 2024-03-28T05:34:59.948435+0000) 2024-03-28T05:35:01.532 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 05:35:01 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-3[53300]: 2024-03-28T05:35:01.053+0000 7f5c3f9cf700 -1 osd.3 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:35.848295+0000 front 2024-03-28T05:34:35.848255+0000 (oldest deadline 2024-03-28T05:34:59.948435+0000) 2024-03-28T05:35:01.533 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:01 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:01.299+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:01.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:01 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:01.439+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:02.282 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 05:35:02 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-3[53300]: 2024-03-28T05:35:02.014+0000 7f5c3f9cf700 -1 osd.3 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:35.848295+0000 front 2024-03-28T05:34:35.848255+0000 (oldest deadline 2024-03-28T05:34:59.948435+0000) 2024-03-28T05:35:02.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:02 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:02.074+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:02.462 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:02 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:02.045+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:35.431401+0000 front 2024-03-28T05:34:35.431260+0000 (oldest deadline 2024-03-28T05:35:01.331244+0000) 2024-03-28T05:35:02.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:02 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:02.462+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:02.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:02 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:02.305+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:03.282 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 05:35:03 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-3[53300]: 2024-03-28T05:35:03.010+0000 7f5c3f9cf700 -1 osd.3 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:35.848295+0000 front 2024-03-28T05:34:35.848255+0000 (oldest deadline 2024-03-28T05:34:59.948435+0000) 2024-03-28T05:35:03.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:03 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:03.091+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:03.418 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:03 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:03.032+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:35.431401+0000 front 2024-03-28T05:34:35.431260+0000 (oldest deadline 2024-03-28T05:35:01.331244+0000) 2024-03-28T05:35:03.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:03 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:03.419+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:03.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:03 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:03.312+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:04.116 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:03 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:03.720+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:04.387 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:04 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:04.277+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:04.387 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:04 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:04.046+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:04.404 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:04 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:04.057+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:35.431401+0000 front 2024-03-28T05:34:35.431260+0000 (oldest deadline 2024-03-28T05:35:01.331244+0000) 2024-03-28T05:35:04.672 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 05:35:04 smithi023 podman[96258]: 2024-03-28 05:35:04.388013229 +0000 UTC m=+5.680719940 container died 2d3ce698d6bd097fab26633d0420e34e080d5439cbb285091045cf9d4be0f9ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-3, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-28T05:35:04.743 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:04 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:04.404+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:05.010 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 05:35:04 smithi023 podman[96258]: 2024-03-28 05:35:04.772504733 +0000 UTC m=+6.065211432 container cleanup 2d3ce698d6bd097fab26633d0420e34e080d5439cbb285091045cf9d4be0f9ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-3, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-28T05:35:05.010 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 05:35:04 smithi023 bash[96258]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-3 2024-03-28T05:35:05.116 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:04 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:04.724+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:05.282 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 05:35:05 smithi023 podman[96287]: 2024-03-28 05:35:05.0758871 +0000 UTC m=+0.683142790 container remove 2d3ce698d6bd097fab26633d0420e34e080d5439cbb285091045cf9d4be0f9ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-3, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-28T05:35:05.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:05 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:05.008+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:05.282 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:05 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:05.253+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:05.418 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:05 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:05.019+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:35.431401+0000 front 2024-03-28T05:34:35.431260+0000 (oldest deadline 2024-03-28T05:35:01.331244+0000) 2024-03-28T05:35:05.693 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 05:35:05 smithi023 podman[96381]: 2024-03-28 05:35:05.337960522 +0000 UTC m=+0.017741349 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:35:05.693 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 05:35:05 smithi023 podman[96381]: 2024-03-28 05:35:05.458340208 +0000 UTC m=+0.138121078 container create d52c44072ef232668ebdeb2cdff6204e8a0a29d5944c798674ae36320b847611 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-28T05:35:05.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:05 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:05.418+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:05.991 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 05:35:05 smithi023 podman[96381]: 2024-03-28 05:35:05.694143366 +0000 UTC m=+0.373924193 container init d52c44072ef232668ebdeb2cdff6204e8a0a29d5944c798674ae36320b847611 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-28T05:35:05.991 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 05:35:05 smithi023 podman[96381]: 2024-03-28 05:35:05.697374225 +0000 UTC m=+0.377155052 container start d52c44072ef232668ebdeb2cdff6204e8a0a29d5944c798674ae36320b847611 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True) 2024-03-28T05:35:05.991 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 05:35:05 smithi023 podman[96381]: 2024-03-28 05:35:05.785859172 +0000 UTC m=+0.465640013 container attach d52c44072ef232668ebdeb2cdff6204e8a0a29d5944c798674ae36320b847611 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-3-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T05:35:06.116 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:05 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:05.769+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:06.249 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 05:35:05 smithi023 podman[96381]: 2024-03-28 05:35:05.992066867 +0000 UTC m=+0.671847705 container died d52c44072ef232668ebdeb2cdff6204e8a0a29d5944c798674ae36320b847611 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T05:35:06.250 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:06 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:06.006+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:06.430 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:06 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:06.051+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:35.431401+0000 front 2024-03-28T05:34:35.431260+0000 (oldest deadline 2024-03-28T05:35:01.331244+0000) 2024-03-28T05:35:06.522 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:06 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:06.248+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:06.549 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.3.service' 2024-03-28T05:35:06.591 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:35:06.591 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-28T05:35:06.591 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-28T05:35:06.591 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.4 2024-03-28T05:35:06.743 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:06 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:06.430+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:06.915 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 28 05:35:06 smithi098 systemd[1]: Stopping Ceph osd.4 for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:35:06.915 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:06 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:06.794+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:07.366 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 28 05:35:06 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-4[47205]: 2024-03-28T05:35:06.915+0000 7fc8f6091700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:35:07.366 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 28 05:35:06 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-4[47205]: 2024-03-28T05:35:06.915+0000 7fc8f6091700 -1 osd.4 107 *** Got signal Terminated *** 2024-03-28T05:35:07.367 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 28 05:35:06 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-4[47205]: 2024-03-28T05:35:06.915+0000 7fc8f6091700 -1 osd.4 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T05:35:07.430 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:07 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:07.030+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:35.431401+0000 front 2024-03-28T05:34:35.431260+0000 (oldest deadline 2024-03-28T05:35:01.331244+0000) 2024-03-28T05:35:07.532 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:07 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:07.270+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:07.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:07 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:07.043+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:07.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:07 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:07.430+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:08.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:07 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:07.824+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:08.410 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:08 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:08.006+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:35.431401+0000 front 2024-03-28T05:34:35.431260+0000 (oldest deadline 2024-03-28T05:35:01.331244+0000) 2024-03-28T05:35:08.532 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:08 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:08.268+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:08.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:08 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:08.072+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:08.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:08 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:08.410+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:09.367 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:08 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:08.938+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:09.367 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:08 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:08.870+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:09.382 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:09 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:09.049+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:35.431401+0000 front 2024-03-28T05:34:35.431260+0000 (oldest deadline 2024-03-28T05:35:01.331244+0000) 2024-03-28T05:35:09.382 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:09 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:09.049+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:45.833562+0000 front 2024-03-28T05:34:45.833735+0000 (oldest deadline 2024-03-28T05:35:08.133615+0000) 2024-03-28T05:35:09.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:09 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:09.108+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:09.532 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:09 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:09.272+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:09.733 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:09 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:09.627+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:09.734 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:09 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:09.383+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:10.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:09 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:09.941+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:10.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:09 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:09.845+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:10.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:09 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:09.845+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:10.400 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:10 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:10.024+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:35.431401+0000 front 2024-03-28T05:34:35.431260+0000 (oldest deadline 2024-03-28T05:35:01.331244+0000) 2024-03-28T05:35:10.400 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:10 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:10.024+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:45.833562+0000 front 2024-03-28T05:34:45.833735+0000 (oldest deadline 2024-03-28T05:35:08.133615+0000) 2024-03-28T05:35:10.532 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:10 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:10.317+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:10.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:10 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:10.094+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:10.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:10 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:10.094+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:10.744 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:10 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:10.643+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:10.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:10 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:10.400+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:11.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:10 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:10.901+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:11.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:10 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:10.812+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:11.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:10 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:10.812+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:11.366 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:11 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:11.045+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:11.366 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:11 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:11.045+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:11.423 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:11 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:11.017+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:35.431401+0000 front 2024-03-28T05:34:35.431260+0000 (oldest deadline 2024-03-28T05:35:01.331244+0000) 2024-03-28T05:35:11.423 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:11 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:11.017+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:45.833562+0000 front 2024-03-28T05:34:45.833735+0000 (oldest deadline 2024-03-28T05:35:08.133615+0000) 2024-03-28T05:35:11.674 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:11 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:11.423+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:11.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:11 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:11.365+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:11.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:11 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:11.365+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.307423+0000 front 2024-03-28T05:34:46.307386+0000 (oldest deadline 2024-03-28T05:35:10.407399+0000) 2024-03-28T05:35:11.867 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 28 05:35:11 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-4[47205]: 2024-03-28T05:35:11.569+0000 7fc8ee67d700 -1 osd.4 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:41.512665+0000 front 2024-03-28T05:34:41.512456+0000 (oldest deadline 2024-03-28T05:35:10.913428+0000) 2024-03-28T05:35:11.867 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:11 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:11.783+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:11.867 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:11 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:11.783+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:11.993 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:11 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:11.675+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:12.244 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:11 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:11.993+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:35.431401+0000 front 2024-03-28T05:34:35.431260+0000 (oldest deadline 2024-03-28T05:35:01.331244+0000) 2024-03-28T05:35:12.244 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:11 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:11.993+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:45.833562+0000 front 2024-03-28T05:34:45.833735+0000 (oldest deadline 2024-03-28T05:35:08.133615+0000) 2024-03-28T05:35:12.261 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:11 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:11.882+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:12.384 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:12 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:12.051+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:12.384 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:12 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:12.051+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:12.582 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 28 05:35:12 smithi098 podman[71961]: 2024-03-28 05:35:12.358301851 +0000 UTC m=+5.567350485 container died e18bc9ad1099b5452600042830373741170a88afbc7b6118af0b99a9b927139e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-4, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-28T05:35:12.744 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:12 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:12.712+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:12.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:12 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:12.473+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:12.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:12 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:12.383+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:12.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:12 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:12.383+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.307423+0000 front 2024-03-28T05:34:46.307386+0000 (oldest deadline 2024-03-28T05:35:10.407399+0000) 2024-03-28T05:35:12.853 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:12 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:12.768+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:12.853 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:12 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:12.768+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:12.853 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 28 05:35:12 smithi098 podman[71961]: 2024-03-28 05:35:12.656976345 +0000 UTC m=+5.866024974 container cleanup e18bc9ad1099b5452600042830373741170a88afbc7b6118af0b99a9b927139e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-4, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T05:35:12.853 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 28 05:35:12 smithi098 bash[71961]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-4 2024-03-28T05:35:13.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:12 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:12.853+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:13.117 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 28 05:35:12 smithi098 podman[71980]: 2024-03-28 05:35:12.990236628 +0000 UTC m=+0.629516497 container remove e18bc9ad1099b5452600042830373741170a88afbc7b6118af0b99a9b927139e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T05:35:13.244 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:12 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:12.950+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:35.431401+0000 front 2024-03-28T05:34:35.431260+0000 (oldest deadline 2024-03-28T05:35:01.331244+0000) 2024-03-28T05:35:13.244 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:12 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:12.950+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:45.833562+0000 front 2024-03-28T05:34:45.833735+0000 (oldest deadline 2024-03-28T05:35:08.133615+0000) 2024-03-28T05:35:13.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:13 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:13.014+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:13.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:13 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:13.014+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:13.432 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:13 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:13.431+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:13.617 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 28 05:35:13 smithi098 podman[72072]: 2024-03-28 05:35:13.217016854 +0000 UTC m=+0.017636198 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:35:13.617 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 28 05:35:13 smithi098 podman[72072]: 2024-03-28 05:35:13.318026938 +0000 UTC m=+0.118646289 container create d35d3a18926b2d238266c21e757846a6e06982dc8b55f9f6f7daaf98d8ef61b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-28T05:35:13.617 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 28 05:35:13 smithi098 podman[72072]: 2024-03-28 05:35:13.554060458 +0000 UTC m=+0.354679805 container init d35d3a18926b2d238266c21e757846a6e06982dc8b55f9f6f7daaf98d8ef61b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, ceph=True) 2024-03-28T05:35:13.617 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 28 05:35:13 smithi098 podman[72072]: 2024-03-28 05:35:13.557397109 +0000 UTC m=+0.358016448 container start d35d3a18926b2d238266c21e757846a6e06982dc8b55f9f6f7daaf98d8ef61b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-28T05:35:13.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:13 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:13.337+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:13.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:13 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:13.337+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.307423+0000 front 2024-03-28T05:34:46.307386+0000 (oldest deadline 2024-03-28T05:35:10.407399+0000) 2024-03-28T05:35:14.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:13 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:13.853+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:14.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:13 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:13.724+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:14.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:13 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:13.724+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:14.117 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 28 05:35:13 smithi098 podman[72072]: 2024-03-28 05:35:13.645989601 +0000 UTC m=+0.446608943 container attach d35d3a18926b2d238266c21e757846a6e06982dc8b55f9f6f7daaf98d8ef61b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-28T05:35:14.117 INFO:journalctl@ceph.osd.4.smithi098.stdout:Mar 28 05:35:13 smithi098 podman[72072]: 2024-03-28 05:35:13.843247742 +0000 UTC m=+0.643867077 container died d35d3a18926b2d238266c21e757846a6e06982dc8b55f9f6f7daaf98d8ef61b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-28T05:35:14.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:13 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:13.753+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:14.244 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:13 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:13.926+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:35.431401+0000 front 2024-03-28T05:34:35.431260+0000 (oldest deadline 2024-03-28T05:35:01.331244+0000) 2024-03-28T05:35:14.244 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:13 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:13.926+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:45.833562+0000 front 2024-03-28T05:34:45.833735+0000 (oldest deadline 2024-03-28T05:35:08.133615+0000) 2024-03-28T05:35:14.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:14 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:14.005+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:14.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:14 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:14.005+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:14.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.4.service' 2024-03-28T05:35:14.549 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:35:14.549 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-28T05:35:14.549 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-28T05:35:14.549 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.5 2024-03-28T05:35:14.745 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:14 smithi113 systemd[1]: Stopping Ceph osd.5 for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:35:14.745 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:14 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:14.446+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:14.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:14 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:14.371+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:14.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:14 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:14.371+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.307423+0000 front 2024-03-28T05:34:46.307386+0000 (oldest deadline 2024-03-28T05:35:10.407399+0000) 2024-03-28T05:35:14.837 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:14 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:14.685+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:14.837 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:14 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:14.685+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:15.116 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:14 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:14.837+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:15.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:14 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:14.778+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:15.244 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:14 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:14.867+0000 7f93adb21700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:35:15.244 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:14 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:14.867+0000 7f93adb21700 -1 osd.5 107 *** Got signal Terminated *** 2024-03-28T05:35:15.244 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:14 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:14.867+0000 7f93adb21700 -1 osd.5 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T05:35:15.244 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:14 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:14.935+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:35.431401+0000 front 2024-03-28T05:34:35.431260+0000 (oldest deadline 2024-03-28T05:35:01.331244+0000) 2024-03-28T05:35:15.244 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:14 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:14.935+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:45.833562+0000 front 2024-03-28T05:34:45.833735+0000 (oldest deadline 2024-03-28T05:35:08.133615+0000) 2024-03-28T05:35:15.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:14 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:14.973+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:15.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:14 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:14.973+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:15.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:15 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:15.408+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:15.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:15 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:15.356+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:15.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:15 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:15.356+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.307423+0000 front 2024-03-28T05:34:46.307386+0000 (oldest deadline 2024-03-28T05:35:10.407399+0000) 2024-03-28T05:35:16.116 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:15 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:15.790+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:16.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:15 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:15.688+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:16.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:15 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:15.688+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:16.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:15 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:15.787+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:16.244 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:15 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:15.907+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:35.431401+0000 front 2024-03-28T05:34:35.431260+0000 (oldest deadline 2024-03-28T05:35:01.331244+0000) 2024-03-28T05:35:16.244 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:15 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:15.907+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:45.833562+0000 front 2024-03-28T05:34:45.833735+0000 (oldest deadline 2024-03-28T05:35:08.133615+0000) 2024-03-28T05:35:16.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:15 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:15.997+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:16.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:15 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:15.997+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:16.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:16 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:16.424+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:16.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:16 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:16.400+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:16.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:16 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:16.400+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.307423+0000 front 2024-03-28T05:34:46.307386+0000 (oldest deadline 2024-03-28T05:35:10.407399+0000) 2024-03-28T05:35:17.116 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:16 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:16.752+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:17.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:16 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:16.732+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:17.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:16 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:16.732+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:17.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:16 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:16.824+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:17.244 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:16 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:16.935+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:35.431401+0000 front 2024-03-28T05:34:35.431260+0000 (oldest deadline 2024-03-28T05:35:01.331244+0000) 2024-03-28T05:35:17.244 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:16 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:16.935+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:45.833562+0000 front 2024-03-28T05:34:45.833735+0000 (oldest deadline 2024-03-28T05:35:08.133615+0000) 2024-03-28T05:35:17.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:16 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:16.949+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:17.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:16 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:16.949+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:17.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:17 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:17.453+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:17.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:17 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:17.433+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:17.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:17 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:17.433+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.307423+0000 front 2024-03-28T05:34:46.307386+0000 (oldest deadline 2024-03-28T05:35:10.407399+0000) 2024-03-28T05:35:18.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:17 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:17.796+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:18.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:17 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:17.698+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:18.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:17 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:17.698+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:18.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:17 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:17.831+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:18.244 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:17 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:17.950+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:35.431401+0000 front 2024-03-28T05:34:35.431260+0000 (oldest deadline 2024-03-28T05:35:01.331244+0000) 2024-03-28T05:35:18.244 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:17 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:17.950+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:45.833562+0000 front 2024-03-28T05:34:45.833735+0000 (oldest deadline 2024-03-28T05:35:08.133615+0000) 2024-03-28T05:35:18.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:17 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:17.931+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:18.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:17 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:17.931+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:18.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:18 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:18.451+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:18.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:18 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:18.451+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.307423+0000 front 2024-03-28T05:34:46.307386+0000 (oldest deadline 2024-03-28T05:35:10.407399+0000) 2024-03-28T05:35:18.852 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:18 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:18.501+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:19.116 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:18 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:18.784+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:19.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:18 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:18.725+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:19.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:18 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:18.725+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:19.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:18 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:18.725+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:19.244 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:18 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:18.972+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:35.431401+0000 front 2024-03-28T05:34:35.431260+0000 (oldest deadline 2024-03-28T05:35:01.331244+0000) 2024-03-28T05:35:19.244 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:18 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5[48989]: 2024-03-28T05:35:18.972+0000 7f93aa115700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:45.833562+0000 front 2024-03-28T05:34:45.833735+0000 (oldest deadline 2024-03-28T05:35:08.133615+0000) 2024-03-28T05:35:19.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:18 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:18.851+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:19.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:18 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:18.851+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:19.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:18 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:18.973+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:19.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:18 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:18.973+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:19.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:19 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:19.489+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:19.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:19 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:19.489+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:19.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:19 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:19.497+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:19.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:19 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:19.497+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.307423+0000 front 2024-03-28T05:34:46.307386+0000 (oldest deadline 2024-03-28T05:35:10.407399+0000) 2024-03-28T05:35:19.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:19 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:19.497+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.508789+0000 front 2024-03-28T05:34:54.508826+0000 (oldest deadline 2024-03-28T05:35:18.608645+0000) 2024-03-28T05:35:20.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:19 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:19.774+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:20.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:19 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:19.759+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:20.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:19 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:19.759+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:20.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:19 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:19.759+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:20.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:19 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:19.806+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:20.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:19 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:19.806+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:20.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:19 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:19.985+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:20.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:19 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:19.985+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:20.660 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:20 smithi113 podman[73867]: 2024-03-28 05:35:20.37154227 +0000 UTC m=+5.629818782 container died cfb21da0173698fdc08b3d2a979f7962a338c3de3315ee678faffff1057eaa30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T05:35:20.660 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:20 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:20.467+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:20.660 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:20 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:20.467+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:20.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:20 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:20.511+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:20.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:20 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:20.511+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.307423+0000 front 2024-03-28T05:34:46.307386+0000 (oldest deadline 2024-03-28T05:35:10.407399+0000) 2024-03-28T05:35:20.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:20 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:20.511+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.508789+0000 front 2024-03-28T05:34:54.508826+0000 (oldest deadline 2024-03-28T05:35:18.608645+0000) 2024-03-28T05:35:20.994 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:20 smithi113 podman[73867]: 2024-03-28 05:35:20.742658055 +0000 UTC m=+6.000934577 container cleanup cfb21da0173698fdc08b3d2a979f7962a338c3de3315ee678faffff1057eaa30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T05:35:20.994 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:20 smithi113 bash[73867]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5 2024-03-28T05:35:20.994 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:20 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:20.823+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:20.994 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:20 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:20.823+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:21.116 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:20 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:20.770+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:21.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:20 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:20.808+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:21.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:20 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:20.808+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:21.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:20 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:20.808+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:21.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:21 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:21.009+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:21.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:21 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:21.009+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:21.350 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:21 smithi113 podman[73888]: 2024-03-28 05:35:21.027639423 +0000 UTC m=+0.659568639 container remove cfb21da0173698fdc08b3d2a979f7962a338c3de3315ee678faffff1057eaa30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-28T05:35:21.608 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:21 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:21.422+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:21.608 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:21 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:21.422+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:21.608 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:21 smithi113 podman[73979]: 2024-03-28 05:35:21.254214129 +0000 UTC m=+0.017699945 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:35:21.608 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:21 smithi113 podman[73979]: 2024-03-28 05:35:21.370195025 +0000 UTC m=+0.133680833 container create 83c5335db21f41086fb90dd541af7412739bcb2d49193f9d70864b6325333ed9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T05:35:21.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:21 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:21.473+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:21.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:21 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:21.473+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.307423+0000 front 2024-03-28T05:34:46.307386+0000 (oldest deadline 2024-03-28T05:35:10.407399+0000) 2024-03-28T05:35:21.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:21 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:21.473+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.508789+0000 front 2024-03-28T05:34:54.508826+0000 (oldest deadline 2024-03-28T05:35:18.608645+0000) 2024-03-28T05:35:21.897 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:21 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:21.823+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:21.897 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:21 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:21.823+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:21.897 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:21 smithi113 podman[73979]: 2024-03-28 05:35:21.608521605 +0000 UTC m=+0.372007406 container init 83c5335db21f41086fb90dd541af7412739bcb2d49193f9d70864b6325333ed9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-28T05:35:21.897 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:21 smithi113 podman[73979]: 2024-03-28 05:35:21.611714078 +0000 UTC m=+0.375199889 container start 83c5335db21f41086fb90dd541af7412739bcb2d49193f9d70864b6325333ed9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T05:35:21.897 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:21 smithi113 podman[73979]: 2024-03-28 05:35:21.71676295 +0000 UTC m=+0.480248776 container attach 83c5335db21f41086fb90dd541af7412739bcb2d49193f9d70864b6325333ed9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T05:35:22.116 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:21 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:21.753+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:22.116 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:21 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:21.753+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.638524+0000 front 2024-03-28T05:34:55.638391+0000 (oldest deadline 2024-03-28T05:35:21.538234+0000) 2024-03-28T05:35:22.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:21 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:21.793+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:22.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:21 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:21.793+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:22.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:21 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:21.793+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:22.244 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 28 05:35:21 smithi113 podman[73979]: 2024-03-28 05:35:21.898017118 +0000 UTC m=+0.661502947 container died 83c5335db21f41086fb90dd541af7412739bcb2d49193f9d70864b6325333ed9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T05:35:22.515 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:22 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:22.045+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:22.515 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:22 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:22.045+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:22.515 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:22 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:22.045+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:22.553 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:22 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:22.390+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:22.553 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:22 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:22.390+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:22.577 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.5.service' 2024-03-28T05:35:22.617 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:35:22.618 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-28T05:35:22.618 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-28T05:35:22.618 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.6 2024-03-28T05:35:22.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:22 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:22.514+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:22.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:22 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:22.514+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.307423+0000 front 2024-03-28T05:34:46.307386+0000 (oldest deadline 2024-03-28T05:35:10.407399+0000) 2024-03-28T05:35:22.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:22 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:22.514+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.508789+0000 front 2024-03-28T05:34:54.508826+0000 (oldest deadline 2024-03-28T05:35:18.608645+0000) 2024-03-28T05:35:22.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:22 smithi023 systemd[1]: Stopping Ceph osd.6 for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:35:23.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:22 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:22.778+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:23.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:22 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:22.778+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.638524+0000 front 2024-03-28T05:34:55.638391+0000 (oldest deadline 2024-03-28T05:35:21.538234+0000) 2024-03-28T05:35:23.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:22 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:22.826+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:23.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:22 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:22.826+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:23.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:22 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:22.826+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:23.243 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:22 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:22.822+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:23.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:22 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:22.822+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:23.282 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:22 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:22.933+0000 7f5540676700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:35:23.282 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:22 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:22.933+0000 7f5540676700 -1 osd.6 107 *** Got signal Terminated *** 2024-03-28T05:35:23.282 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:22 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:22.933+0000 7f5540676700 -1 osd.6 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T05:35:23.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:23 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:23.041+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:23.283 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:23 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:23.041+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:23.283 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:23 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:23.041+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:23.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:23 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:23.411+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:23.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:23 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:23.411+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:23.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:23 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:23.526+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:23.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:23 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:23.526+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.307423+0000 front 2024-03-28T05:34:46.307386+0000 (oldest deadline 2024-03-28T05:35:10.407399+0000) 2024-03-28T05:35:23.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:23 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:23.526+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.508789+0000 front 2024-03-28T05:34:54.508826+0000 (oldest deadline 2024-03-28T05:35:18.608645+0000) 2024-03-28T05:35:24.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:23 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:23.755+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:24.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:23 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:23.755+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.638524+0000 front 2024-03-28T05:34:55.638391+0000 (oldest deadline 2024-03-28T05:35:21.538234+0000) 2024-03-28T05:35:24.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:23 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:23.856+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:24.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:23 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:23.856+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:24.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:23 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:23.856+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:24.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:23 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:23.773+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:24.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:23 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:23.773+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:24.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:24 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:24.037+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:24.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:24 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:24.037+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:24.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:24 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:24.037+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:24.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:24 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:24.417+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:24.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:24 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:24.417+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:24.991 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:24 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:24.557+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:24.991 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:24 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:24.557+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.307423+0000 front 2024-03-28T05:34:46.307386+0000 (oldest deadline 2024-03-28T05:35:10.407399+0000) 2024-03-28T05:35:24.991 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:24 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:24.557+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.508789+0000 front 2024-03-28T05:34:54.508826+0000 (oldest deadline 2024-03-28T05:35:18.608645+0000) 2024-03-28T05:35:25.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:24 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:24.803+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:25.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:24 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:24.803+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.638524+0000 front 2024-03-28T05:34:55.638391+0000 (oldest deadline 2024-03-28T05:35:21.538234+0000) 2024-03-28T05:35:25.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:24 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:24.837+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:25.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:24 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:24.837+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:25.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:24 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:24.837+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:25.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:24 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:24.787+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:25.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:24 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:24.787+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:25.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:24 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:24.990+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:25.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:24 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:24.990+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:25.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:24 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:24.990+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:25.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:25 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:25.368+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:25.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:25 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:25.368+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:25.946 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:25 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:25.550+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:25.946 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:25 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:25.550+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.307423+0000 front 2024-03-28T05:34:46.307386+0000 (oldest deadline 2024-03-28T05:35:10.407399+0000) 2024-03-28T05:35:25.946 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:25 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:25.550+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.508789+0000 front 2024-03-28T05:34:54.508826+0000 (oldest deadline 2024-03-28T05:35:18.608645+0000) 2024-03-28T05:35:26.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:25 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:25.849+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:26.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:25 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:25.849+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.638524+0000 front 2024-03-28T05:34:55.638391+0000 (oldest deadline 2024-03-28T05:35:21.538234+0000) 2024-03-28T05:35:26.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:25 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:25.868+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:26.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:25 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:25.868+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:26.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:25 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:25.868+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:26.243 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:25 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:25.830+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:26.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:25 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:25.830+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:26.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:25 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:25.946+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:26.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:25 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:25.946+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:26.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:25 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:25.946+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:26.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:26 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:26.337+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:26.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:26 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:26.337+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:26.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:26 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:26.337+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:26.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:26 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:26.529+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:26.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:26 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:26.529+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.307423+0000 front 2024-03-28T05:34:46.307386+0000 (oldest deadline 2024-03-28T05:35:10.407399+0000) 2024-03-28T05:35:26.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:26 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:26.529+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.508789+0000 front 2024-03-28T05:34:54.508826+0000 (oldest deadline 2024-03-28T05:35:18.608645+0000) 2024-03-28T05:35:27.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:26 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:26.820+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:27.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:26 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:26.820+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.638524+0000 front 2024-03-28T05:34:55.638391+0000 (oldest deadline 2024-03-28T05:35:21.538234+0000) 2024-03-28T05:35:27.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:26 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:26.914+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:27.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:26 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:26.914+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:27.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:26 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:26.914+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:27.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:26 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:26.914+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:27.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:26 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:26.783+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:27.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:26 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:26.783+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:27.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:26 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:26.993+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:27.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:26 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:26.993+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:27.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:26 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:26.993+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:27.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:27 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:27.348+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:27.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:27 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:27.348+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:27.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:27 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:27.348+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:27.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:27 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:27.505+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:37.704598+0000 front 2024-03-28T05:34:37.704627+0000 (oldest deadline 2024-03-28T05:35:00.604737+0000) 2024-03-28T05:35:27.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:27 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:27.505+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.307423+0000 front 2024-03-28T05:34:46.307386+0000 (oldest deadline 2024-03-28T05:35:10.407399+0000) 2024-03-28T05:35:27.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:27 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:27.505+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.508789+0000 front 2024-03-28T05:34:54.508826+0000 (oldest deadline 2024-03-28T05:35:18.608645+0000) 2024-03-28T05:35:27.782 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:27 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6[60305]: 2024-03-28T05:35:27.505+0000 7f553cc6a700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.509656+0000 front 2024-03-28T05:35:01.509539+0000 (oldest deadline 2024-03-28T05:35:26.809753+0000) 2024-03-28T05:35:28.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:27 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:27.814+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:28.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:27 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:27.814+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.638524+0000 front 2024-03-28T05:34:55.638391+0000 (oldest deadline 2024-03-28T05:35:21.538234+0000) 2024-03-28T05:35:28.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:27 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:27.814+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.538711+0000 front 2024-03-28T05:35:01.538680+0000 (oldest deadline 2024-03-28T05:35:27.438729+0000) 2024-03-28T05:35:28.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:27 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:27.931+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:28.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:27 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:27.931+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:28.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:27 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:27.931+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:28.118 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:27 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:27.931+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:28.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:27 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:27.817+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:28.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:27 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:27.817+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:28.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:27 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:27.817+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.525750+0000 front 2024-03-28T05:35:01.525777+0000 (oldest deadline 2024-03-28T05:35:26.825951+0000) 2024-03-28T05:35:28.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:28 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:28.016+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:28.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:28 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:28.016+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:28.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:28 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:28.016+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:28.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:28 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:28.016+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:28.585 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:28 smithi023 podman[96487]: 2024-03-28 05:35:28.376841972 +0000 UTC m=+5.563658785 container died ff62f518f7e44735f70395d7b83c432802c7fa0f6647a62d50cfac0d3ee8ef85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=) 2024-03-28T05:35:28.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:28 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:28.353+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:28.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:28 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:28.353+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:28.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:28 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:28.353+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:28.976 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:28 smithi023 podman[96487]: 2024-03-28 05:35:28.75916288 +0000 UTC m=+5.945979695 container cleanup ff62f518f7e44735f70395d7b83c432802c7fa0f6647a62d50cfac0d3ee8ef85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-28T05:35:28.976 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:28 smithi023 bash[96487]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6 2024-03-28T05:35:29.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:28 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:28.780+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:29.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:28 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:28.780+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.638524+0000 front 2024-03-28T05:34:55.638391+0000 (oldest deadline 2024-03-28T05:35:21.538234+0000) 2024-03-28T05:35:29.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:28 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:28.780+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.538711+0000 front 2024-03-28T05:35:01.538680+0000 (oldest deadline 2024-03-28T05:35:27.438729+0000) 2024-03-28T05:35:29.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:28 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:28.920+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:29.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:28 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:28.920+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:29.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:28 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:28.920+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:29.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:28 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:28.920+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:29.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:28 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:28.796+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:29.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:28 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:28.796+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:29.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:28 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:28.796+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.525750+0000 front 2024-03-28T05:35:01.525777+0000 (oldest deadline 2024-03-28T05:35:26.825951+0000) 2024-03-28T05:35:29.282 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:28 smithi023 podman[96505]: 2024-03-28 05:35:28.976788714 +0000 UTC m=+0.593859896 container remove ff62f518f7e44735f70395d7b83c432802c7fa0f6647a62d50cfac0d3ee8ef85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-28T05:35:29.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:29 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:29.062+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:29.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:29 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:29.062+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:29.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:29 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:29.062+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:29.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:29 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:29.062+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:29.517 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:29 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:29.349+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:29.517 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:29 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:29.349+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:29.517 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:29 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:29.349+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:29.572 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:29 smithi023 podman[96602]: 2024-03-28 05:35:29.196916179 +0000 UTC m=+0.017898481 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:35:29.572 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:29 smithi023 podman[96602]: 2024-03-28 05:35:29.319559607 +0000 UTC m=+0.140541917 container create 892c5ce70db2e4ae80e587033db282d04ea424127e7930155e1aca699977a1ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-28T05:35:29.572 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:29 smithi023 podman[96602]: 2024-03-28 05:35:29.5720959 +0000 UTC m=+0.393078201 container init 892c5ce70db2e4ae80e587033db282d04ea424127e7930155e1aca699977a1ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-28T05:35:29.873 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:29 smithi023 podman[96602]: 2024-03-28 05:35:29.575704155 +0000 UTC m=+0.396686459 container start 892c5ce70db2e4ae80e587033db282d04ea424127e7930155e1aca699977a1ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T05:35:29.873 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:29 smithi023 podman[96602]: 2024-03-28 05:35:29.689000588 +0000 UTC m=+0.509982900 container attach 892c5ce70db2e4ae80e587033db282d04ea424127e7930155e1aca699977a1ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-28T05:35:30.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:29 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:29.826+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:30.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:29 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:29.826+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.638524+0000 front 2024-03-28T05:34:55.638391+0000 (oldest deadline 2024-03-28T05:35:21.538234+0000) 2024-03-28T05:35:30.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:29 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:29.826+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.538711+0000 front 2024-03-28T05:35:01.538680+0000 (oldest deadline 2024-03-28T05:35:27.438729+0000) 2024-03-28T05:35:30.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:29 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:29.904+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:30.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:29 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:29.904+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:30.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:29 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:29.904+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:30.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:29 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:29.904+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:30.224 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 05:35:29 smithi023 podman[96602]: 2024-03-28 05:35:29.874164218 +0000 UTC m=+0.695146526 container died 892c5ce70db2e4ae80e587033db282d04ea424127e7930155e1aca699977a1ab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS) 2024-03-28T05:35:30.225 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:30 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:30.076+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:30.225 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:30 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:30.076+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:30.225 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:30 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:30.076+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:30.225 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:30 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:30.076+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:30.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:29 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:29.778+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:30.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:29 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:29.778+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:30.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:29 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:29.778+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.525750+0000 front 2024-03-28T05:35:01.525777+0000 (oldest deadline 2024-03-28T05:35:26.825951+0000) 2024-03-28T05:35:30.495 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.6.service' 2024-03-28T05:35:30.538 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:35:30.538 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-28T05:35:30.538 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-28T05:35:30.538 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.7 2024-03-28T05:35:30.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:30 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:30.321+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:30.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:30 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:30.321+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:30.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:30 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:30.321+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:30.836 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:30 smithi098 systemd[1]: Stopping Ceph osd.7 for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:35:31.116 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:30 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:30.890+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:31.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:30 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:30.890+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:31.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:30 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:30.890+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:31.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:30 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:30.890+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:31.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:30 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:30.836+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:31.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:30 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:30.836+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.638524+0000 front 2024-03-28T05:34:55.638391+0000 (oldest deadline 2024-03-28T05:35:21.538234+0000) 2024-03-28T05:35:31.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:30 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:30.836+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.538711+0000 front 2024-03-28T05:35:01.538680+0000 (oldest deadline 2024-03-28T05:35:27.438729+0000) 2024-03-28T05:35:31.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:30 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:30.860+0000 7fdccf759700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:35:31.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:30 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:30.860+0000 7fdccf759700 -1 osd.7 107 *** Got signal Terminated *** 2024-03-28T05:35:31.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:30 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:30.860+0000 7fdccf759700 -1 osd.7 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T05:35:31.243 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:30 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:30.767+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:31.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:30 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:30.767+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:31.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:30 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:30.767+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.525750+0000 front 2024-03-28T05:35:01.525777+0000 (oldest deadline 2024-03-28T05:35:26.825951+0000) 2024-03-28T05:35:31.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:31 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:31.054+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:31.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:31 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:31.054+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:31.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:31 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:31.054+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:31.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:31 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:31.054+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:31.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:31 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:31.355+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:31.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:31 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:31.355+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:31.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:31 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:31.355+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:32.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:31 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:31.800+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:32.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:31 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:31.800+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.638524+0000 front 2024-03-28T05:34:55.638391+0000 (oldest deadline 2024-03-28T05:35:21.538234+0000) 2024-03-28T05:35:32.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:31 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:31.800+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.538711+0000 front 2024-03-28T05:35:01.538680+0000 (oldest deadline 2024-03-28T05:35:27.438729+0000) 2024-03-28T05:35:32.118 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:31 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:31.862+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:32.118 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:31 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:31.862+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:32.118 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:31 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:31.862+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:32.118 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:31 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:31.862+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:32.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:31 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:31.793+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:32.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:31 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:31.793+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:32.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:31 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:31.793+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.525750+0000 front 2024-03-28T05:35:01.525777+0000 (oldest deadline 2024-03-28T05:35:26.825951+0000) 2024-03-28T05:35:32.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:32 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:32.054+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:32.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:32 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:32.054+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:32.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:32 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:32.054+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:32.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:32 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:32.054+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:32.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:32 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:32.306+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:32.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:32 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:32.306+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:32.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:32 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:32.306+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:33.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:32 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:32.838+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:33.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:32 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:32.838+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.638524+0000 front 2024-03-28T05:34:55.638391+0000 (oldest deadline 2024-03-28T05:35:21.538234+0000) 2024-03-28T05:35:33.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:32 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:32.838+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.538711+0000 front 2024-03-28T05:35:01.538680+0000 (oldest deadline 2024-03-28T05:35:27.438729+0000) 2024-03-28T05:35:33.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:32 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:32.818+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:33.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:32 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:32.818+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:33.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:32 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:32.818+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:33.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:32 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:32.818+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:33.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:32 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:32.831+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:33.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:32 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:32.831+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:33.244 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:32 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:32.831+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.525750+0000 front 2024-03-28T05:35:01.525777+0000 (oldest deadline 2024-03-28T05:35:26.825951+0000) 2024-03-28T05:35:33.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:33 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:33.015+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:33.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:33 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:33.015+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:33.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:33 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:33.015+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:33.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:33 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:33.015+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:33.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:33 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:33.259+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:33.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:33 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:33.259+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:33.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:33 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:33.259+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:34.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:33 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:33.787+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:34.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:33 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:33.787+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:34.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:33 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:33.787+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:34.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:33 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:33.787+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:34.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:33 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:33.824+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:34.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:33 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:33.824+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.638524+0000 front 2024-03-28T05:34:55.638391+0000 (oldest deadline 2024-03-28T05:35:21.538234+0000) 2024-03-28T05:35:34.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:33 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:33.824+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.538711+0000 front 2024-03-28T05:35:01.538680+0000 (oldest deadline 2024-03-28T05:35:27.438729+0000) 2024-03-28T05:35:34.118 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:33 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:33.824+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.539689+0000 front 2024-03-28T05:35:11.539729+0000 (oldest deadline 2024-03-28T05:35:33.239934+0000) 2024-03-28T05:35:34.225 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:33 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:33.793+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:34.225 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:33 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:33.793+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:34.225 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:33 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:33.793+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.525750+0000 front 2024-03-28T05:35:01.525777+0000 (oldest deadline 2024-03-28T05:35:26.825951+0000) 2024-03-28T05:35:34.225 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:33 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:33.793+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:10.827747+0000 front 2024-03-28T05:35:10.827706+0000 (oldest deadline 2024-03-28T05:35:33.127780+0000) 2024-03-28T05:35:34.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:34 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:34.019+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:34.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:34 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:34.019+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:34.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:34 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:34.019+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:34.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:34 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:34.019+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:34.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:34 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:34.019+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:07.246038+0000 front 2024-03-28T05:35:07.245902+0000 (oldest deadline 2024-03-28T05:35:33.145893+0000) 2024-03-28T05:35:34.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:34 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:34.225+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:34.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:34 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:34.225+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:34.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:34 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:34.225+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:35.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:34 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:34.784+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:35.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:34 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:34.784+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.638524+0000 front 2024-03-28T05:34:55.638391+0000 (oldest deadline 2024-03-28T05:35:21.538234+0000) 2024-03-28T05:35:35.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:34 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:34.784+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.538711+0000 front 2024-03-28T05:35:01.538680+0000 (oldest deadline 2024-03-28T05:35:27.438729+0000) 2024-03-28T05:35:35.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:34 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:34.784+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.539689+0000 front 2024-03-28T05:35:11.539729+0000 (oldest deadline 2024-03-28T05:35:33.239934+0000) 2024-03-28T05:35:35.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:34 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:34.759+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:35.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:34 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:34.759+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:35.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:34 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:34.759+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:35.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:34 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:34.759+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:35.118 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:34 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:34.759+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:35.180 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:34 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:34.808+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:35.180 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:34 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:34.808+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:35.180 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:34 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:34.808+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.525750+0000 front 2024-03-28T05:35:01.525777+0000 (oldest deadline 2024-03-28T05:35:26.825951+0000) 2024-03-28T05:35:35.180 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:34 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:34.808+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:10.827747+0000 front 2024-03-28T05:35:10.827706+0000 (oldest deadline 2024-03-28T05:35:33.127780+0000) 2024-03-28T05:35:35.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:35 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:35.179+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:35.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:35 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:35.179+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:35.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:35 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:35.179+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:35.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:35 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:35.179+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:35.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:35 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:35.069+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:35.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:35 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:35.069+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:35.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:35 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:35.069+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:35.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:35 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:35.069+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:35.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:35 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:35.069+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:07.246038+0000 front 2024-03-28T05:35:07.245902+0000 (oldest deadline 2024-03-28T05:35:33.145893+0000) 2024-03-28T05:35:36.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:35 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:35.815+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.336588+0000 front 2024-03-28T05:34:46.336563+0000 (oldest deadline 2024-03-28T05:35:08.636727+0000) 2024-03-28T05:35:36.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:35 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:35.815+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.638524+0000 front 2024-03-28T05:34:55.638391+0000 (oldest deadline 2024-03-28T05:35:21.538234+0000) 2024-03-28T05:35:36.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:35 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:35.815+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.538711+0000 front 2024-03-28T05:35:01.538680+0000 (oldest deadline 2024-03-28T05:35:27.438729+0000) 2024-03-28T05:35:36.117 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:35 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7[53780]: 2024-03-28T05:35:35.815+0000 7fdccbd4d700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.539689+0000 front 2024-03-28T05:35:11.539729+0000 (oldest deadline 2024-03-28T05:35:33.239934+0000) 2024-03-28T05:35:36.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:35 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:35.727+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:36.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:35 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:35.727+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:36.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:35 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:35.727+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:36.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:35 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:35.727+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:36.118 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:35 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:35.727+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:36.139 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:35 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:35.796+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:36.140 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:35 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:35.796+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:36.140 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:35 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:35.796+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.525750+0000 front 2024-03-28T05:35:01.525777+0000 (oldest deadline 2024-03-28T05:35:26.825951+0000) 2024-03-28T05:35:36.140 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:35 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:35.796+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:10.827747+0000 front 2024-03-28T05:35:10.827706+0000 (oldest deadline 2024-03-28T05:35:33.127780+0000) 2024-03-28T05:35:36.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:36 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:36.139+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:36.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:36 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:36.139+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:36.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:36 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:36.139+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:36.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:36 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:36.139+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:36.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:36 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:36.037+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:36.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:36 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:36.037+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:36.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:36 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:36.037+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:36.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:36 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:36.037+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:36.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:36 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:36.037+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:07.246038+0000 front 2024-03-28T05:35:07.245902+0000 (oldest deadline 2024-03-28T05:35:33.145893+0000) 2024-03-28T05:35:36.586 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:36 smithi098 podman[72176]: 2024-03-28 05:35:36.260430543 +0000 UTC m=+5.525376603 container died de4fe9f5cc53b14fe1c0ad2abb10b051f9de5825d7fe6775aa49c3052c5cfb5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T05:35:36.867 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:36 smithi098 podman[72176]: 2024-03-28 05:35:36.695097353 +0000 UTC m=+5.960043393 container cleanup de4fe9f5cc53b14fe1c0ad2abb10b051f9de5825d7fe6775aa49c3052c5cfb5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.build-date=20240319) 2024-03-28T05:35:36.867 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:36 smithi098 bash[72176]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7 2024-03-28T05:35:36.867 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:36 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:36.753+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:36.867 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:36 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:36.753+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:36.867 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:36 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:36.753+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:36.867 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:36 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:36.753+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:36.867 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:36 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:36.753+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:37.157 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:36 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:36.759+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:37.158 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:36 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:36.759+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:37.158 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:36 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:36.759+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.525750+0000 front 2024-03-28T05:35:01.525777+0000 (oldest deadline 2024-03-28T05:35:26.825951+0000) 2024-03-28T05:35:37.158 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:36 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:36.759+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:10.827747+0000 front 2024-03-28T05:35:10.827706+0000 (oldest deadline 2024-03-28T05:35:33.127780+0000) 2024-03-28T05:35:37.352 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:36 smithi098 podman[72205]: 2024-03-28 05:35:36.92968934 +0000 UTC m=+0.663436142 container remove de4fe9f5cc53b14fe1c0ad2abb10b051f9de5825d7fe6775aa49c3052c5cfb5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T05:35:37.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:37 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:37.157+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:37.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:37 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:37.157+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:37.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:37 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:37.157+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:37.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:37 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:37.157+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:37.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:37 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:37.042+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:37.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:37 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:37.042+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:37.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:37 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:37.042+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:37.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:37 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:37.042+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:37.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:37 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:37.042+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:07.246038+0000 front 2024-03-28T05:35:07.245902+0000 (oldest deadline 2024-03-28T05:35:33.145893+0000) 2024-03-28T05:35:37.616 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:37 smithi098 podman[72295]: 2024-03-28 05:35:37.256038036 +0000 UTC m=+0.018099331 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:35:37.617 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:37 smithi098 podman[72295]: 2024-03-28 05:35:37.381457609 +0000 UTC m=+0.143518943 container create f81b7553ebbadda922f062ebeda53feba0406727f4840b55783ba927aa7e18e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-28T05:35:37.945 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:37 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:37.791+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:37.945 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:37 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:37.791+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:37.945 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:37 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:37.791+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:37.946 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:37 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:37.791+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:37.946 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:37 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:37.791+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:37.946 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:37 smithi098 podman[72295]: 2024-03-28 05:35:37.643642359 +0000 UTC m=+0.405703661 container init f81b7553ebbadda922f062ebeda53feba0406727f4840b55783ba927aa7e18e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7-deactivate, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-28T05:35:37.946 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:37 smithi098 podman[72295]: 2024-03-28 05:35:37.646774201 +0000 UTC m=+0.408835507 container start f81b7553ebbadda922f062ebeda53feba0406727f4840b55783ba927aa7e18e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-28T05:35:37.946 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:37 smithi098 podman[72295]: 2024-03-28 05:35:37.72464436 +0000 UTC m=+0.486705665 container attach f81b7553ebbadda922f062ebeda53feba0406727f4840b55783ba927aa7e18e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-28T05:35:38.145 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:37 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:37.804+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:38.145 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:37 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:37.804+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:38.145 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:37 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:37.804+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.525750+0000 front 2024-03-28T05:35:01.525777+0000 (oldest deadline 2024-03-28T05:35:26.825951+0000) 2024-03-28T05:35:38.145 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:37 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:37.804+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:10.827747+0000 front 2024-03-28T05:35:10.827706+0000 (oldest deadline 2024-03-28T05:35:33.127780+0000) 2024-03-28T05:35:38.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:38 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:38.027+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:38.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:38 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:38.027+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:38.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:38 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:38.027+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:38.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:38 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:38.027+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:38.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:38 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:38.027+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:07.246038+0000 front 2024-03-28T05:35:07.245902+0000 (oldest deadline 2024-03-28T05:35:33.145893+0000) 2024-03-28T05:35:38.311 INFO:journalctl@ceph.osd.7.smithi098.stdout:Mar 28 05:35:37 smithi098 podman[72295]: 2024-03-28 05:35:37.945833024 +0000 UTC m=+0.707894326 container died f81b7553ebbadda922f062ebeda53feba0406727f4840b55783ba927aa7e18e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T05:35:38.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:38 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:38.145+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:38.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:38 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:38.145+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:38.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:38 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:38.145+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:38.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:38 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:38.145+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:38.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.7.service' 2024-03-28T05:35:38.694 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:35:38.694 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-28T05:35:38.694 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-28T05:35:38.694 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.8 2024-03-28T05:35:38.994 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:38 smithi113 systemd[1]: Stopping Ceph osd.8 for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:35:38.994 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:38 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:38.830+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:38.994 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:38 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:38.830+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:38.994 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:38 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:38.830+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.525750+0000 front 2024-03-28T05:35:01.525777+0000 (oldest deadline 2024-03-28T05:35:26.825951+0000) 2024-03-28T05:35:38.994 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:38 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:38.830+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:10.827747+0000 front 2024-03-28T05:35:10.827706+0000 (oldest deadline 2024-03-28T05:35:33.127780+0000) 2024-03-28T05:35:39.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:38 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:38.820+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:39.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:38 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:38.820+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:39.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:38 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:38.820+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:39.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:38 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:38.820+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:39.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:38 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:38.820+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:39.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:39 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:39.001+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:39.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:39 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:39.001+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:39.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:39 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:39.001+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:39.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:39 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:39.001+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:39.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:39 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:39.001+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:07.246038+0000 front 2024-03-28T05:35:07.245902+0000 (oldest deadline 2024-03-28T05:35:33.145893+0000) 2024-03-28T05:35:39.494 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:39 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:39.030+0000 7efef63f9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:35:39.494 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:39 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:39.030+0000 7efef63f9700 -1 osd.8 107 *** Got signal Terminated *** 2024-03-28T05:35:39.494 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:39 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:39.030+0000 7efef63f9700 -1 osd.8 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T05:35:39.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:39 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:39.121+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:39.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:39 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:39.121+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:39.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:39 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:39.121+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:39.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:39 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:39.121+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:40.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:39 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:39.853+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:40.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:39 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:39.853+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:40.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:39 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:39.853+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:40.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:39 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:39.853+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:40.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:39 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:39.853+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:40.169 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:39 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:39.857+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:40.169 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:39 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:39.857+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:40.169 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:39 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:39.857+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.525750+0000 front 2024-03-28T05:35:01.525777+0000 (oldest deadline 2024-03-28T05:35:26.825951+0000) 2024-03-28T05:35:40.169 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:39 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:39.857+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:10.827747+0000 front 2024-03-28T05:35:10.827706+0000 (oldest deadline 2024-03-28T05:35:33.127780+0000) 2024-03-28T05:35:40.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:39 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:39.958+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:40.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:39 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:39.958+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:40.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:39 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:39.958+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:40.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:39 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:39.958+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:40.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:39 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:39.958+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:07.246038+0000 front 2024-03-28T05:35:07.245902+0000 (oldest deadline 2024-03-28T05:35:33.145893+0000) 2024-03-28T05:35:40.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:40 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:40.169+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:40.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:40 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:40.169+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:40.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:40 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:40.169+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:40.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:40 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:40.169+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:41.134 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:40 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:40.812+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:41.134 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:40 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:40.812+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:41.134 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:40 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:40.812+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.525750+0000 front 2024-03-28T05:35:01.525777+0000 (oldest deadline 2024-03-28T05:35:26.825951+0000) 2024-03-28T05:35:41.134 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:40 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:40.812+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:10.827747+0000 front 2024-03-28T05:35:10.827706+0000 (oldest deadline 2024-03-28T05:35:33.127780+0000) 2024-03-28T05:35:41.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:40 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:40.920+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:41.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:40 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:40.920+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:41.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:40 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:40.920+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:41.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:40 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:40.920+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:41.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:40 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:40.920+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:07.246038+0000 front 2024-03-28T05:35:07.245902+0000 (oldest deadline 2024-03-28T05:35:33.145893+0000) 2024-03-28T05:35:41.367 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:40 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:40.868+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:41.367 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:40 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:40.868+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:41.367 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:40 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:40.868+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:41.367 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:40 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:40.868+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:41.367 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:40 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:40.868+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:41.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:41 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:41.133+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:41.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:41 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:41.133+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:41.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:41 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:41.133+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:41.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:41 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:41.133+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:42.106 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:41 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:41.774+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:42.106 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:41 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:41.774+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:42.106 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:41 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:41.774+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.525750+0000 front 2024-03-28T05:35:01.525777+0000 (oldest deadline 2024-03-28T05:35:26.825951+0000) 2024-03-28T05:35:42.106 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:41 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:41.774+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:10.827747+0000 front 2024-03-28T05:35:10.827706+0000 (oldest deadline 2024-03-28T05:35:33.127780+0000) 2024-03-28T05:35:42.116 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:41 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:41.893+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:42.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:41 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:41.893+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:42.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:41 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:41.893+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:42.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:41 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:41.893+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:42.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:41 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:41.893+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:42.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:41 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:41.969+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:42.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:41 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:41.969+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:42.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:41 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:41.969+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:42.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:41 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:41.969+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:42.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:41 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:41.969+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:07.246038+0000 front 2024-03-28T05:35:07.245902+0000 (oldest deadline 2024-03-28T05:35:33.145893+0000) 2024-03-28T05:35:42.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:41 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:41.969+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:18.947627+0000 front 2024-03-28T05:35:18.947665+0000 (oldest deadline 2024-03-28T05:35:41.247322+0000) 2024-03-28T05:35:42.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:42 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:42.105+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:42.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:42 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:42.105+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:42.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:42 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:42.105+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:42.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:42 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:42.105+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:43.090 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:42 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:42.800+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:43.090 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:42 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:42.800+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:43.090 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:42 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:42.800+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.525750+0000 front 2024-03-28T05:35:01.525777+0000 (oldest deadline 2024-03-28T05:35:26.825951+0000) 2024-03-28T05:35:43.090 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:42 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:42.800+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:10.827747+0000 front 2024-03-28T05:35:10.827706+0000 (oldest deadline 2024-03-28T05:35:33.127780+0000) 2024-03-28T05:35:43.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:42 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:42.981+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:43.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:42 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:42.981+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:43.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:42 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:42.981+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:43.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:42 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:42.981+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:43.283 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:42 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:42.981+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:07.246038+0000 front 2024-03-28T05:35:07.245902+0000 (oldest deadline 2024-03-28T05:35:33.145893+0000) 2024-03-28T05:35:43.283 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:42 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:42.981+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:18.947627+0000 front 2024-03-28T05:35:18.947665+0000 (oldest deadline 2024-03-28T05:35:41.247322+0000) 2024-03-28T05:35:43.366 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:42 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:42.905+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:43.367 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:42 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:42.905+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:43.367 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:42 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:42.905+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:43.367 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:42 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:42.905+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:43.367 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:42 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:42.905+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:43.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:43 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:43.089+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:43.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:43 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:43.089+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:43.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:43 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:43.089+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:43.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:43 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:43.089+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:44.059 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:43 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:43.802+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.522355+0000 front 2024-03-28T05:34:43.522211+0000 (oldest deadline 2024-03-28T05:35:09.422430+0000) 2024-03-28T05:35:44.059 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:43 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:43.802+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:55.223922+0000 front 2024-03-28T05:34:55.223853+0000 (oldest deadline 2024-03-28T05:35:18.724014+0000) 2024-03-28T05:35:44.059 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:43 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:43.802+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.525750+0000 front 2024-03-28T05:35:01.525777+0000 (oldest deadline 2024-03-28T05:35:26.825951+0000) 2024-03-28T05:35:44.059 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:43 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:43.802+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:10.827747+0000 front 2024-03-28T05:35:10.827706+0000 (oldest deadline 2024-03-28T05:35:33.127780+0000) 2024-03-28T05:35:44.060 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:43 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8[56000]: 2024-03-28T05:35:43.802+0000 7efeee9e5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:17.228979+0000 front 2024-03-28T05:35:17.228959+0000 (oldest deadline 2024-03-28T05:35:43.129121+0000) 2024-03-28T05:35:44.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:44 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:44.028+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:44.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:44 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:44.028+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:44.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:44 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:44.028+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:44.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:44 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:44.028+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:44.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:44 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:44.028+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:07.246038+0000 front 2024-03-28T05:35:07.245902+0000 (oldest deadline 2024-03-28T05:35:33.145893+0000) 2024-03-28T05:35:44.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:44 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:44.028+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:18.947627+0000 front 2024-03-28T05:35:18.947665+0000 (oldest deadline 2024-03-28T05:35:41.247322+0000) 2024-03-28T05:35:44.367 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:43 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:43.900+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:44.367 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:43 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:43.900+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:44.367 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:43 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:43.900+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:44.367 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:43 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:43.900+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:44.367 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:43 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:43.900+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:44.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:44 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:44.059+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:44.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:44 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:44.059+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:44.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:44 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:44.059+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:44.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:44 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:44.059+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:44.863 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:44 smithi113 podman[74096]: 2024-03-28 05:35:44.597424051 +0000 UTC m=+5.697265390 container died 7897eec27f8fa61a03623f3f7289052769922f5b62dd3af85f517413cad9b0fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T05:35:45.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:44 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:44.856+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:45.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:44 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:44.856+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:45.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:44 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:44.856+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:45.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:44 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:44.856+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:45.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:44 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:44.856+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:45.173 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:44 smithi113 podman[74096]: 2024-03-28 05:35:44.863437329 +0000 UTC m=+5.963278650 container cleanup 7897eec27f8fa61a03623f3f7289052769922f5b62dd3af85f517413cad9b0fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-28T05:35:45.173 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:44 smithi113 bash[74096]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8 2024-03-28T05:35:45.173 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:45 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:45.034+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:45.173 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:45 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:45.034+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:45.173 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:45 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:45.034+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:45.173 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:45 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:45.034+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:45.173 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:45 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:45.034+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:35:45.484 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:45 smithi113 podman[74113]: 2024-03-28 05:35:45.173249942 +0000 UTC m=+0.574984438 container remove 7897eec27f8fa61a03623f3f7289052769922f5b62dd3af85f517413cad9b0fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-03-28T05:35:45.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:45 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:45.057+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:45.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:45 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:45.057+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:45.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:45 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:45.057+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:45.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:45 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:45.057+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:45.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:45 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:45.057+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:07.246038+0000 front 2024-03-28T05:35:07.245902+0000 (oldest deadline 2024-03-28T05:35:33.145893+0000) 2024-03-28T05:35:45.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:45 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:45.057+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:18.947627+0000 front 2024-03-28T05:35:18.947665+0000 (oldest deadline 2024-03-28T05:35:41.247322+0000) 2024-03-28T05:35:45.743 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:45 smithi113 podman[74202]: 2024-03-28 05:35:45.387619241 +0000 UTC m=+0.018118935 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:35:45.743 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:45 smithi113 podman[74202]: 2024-03-28 05:35:45.507565571 +0000 UTC m=+0.138065270 container create 334b575c3e1737dd7fa28b6565fea3b854a41a2ba7949284ad18c117488d4caf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-28T05:35:45.994 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:45 smithi113 podman[74202]: 2024-03-28 05:35:45.743695336 +0000 UTC m=+0.374195021 container init 334b575c3e1737dd7fa28b6565fea3b854a41a2ba7949284ad18c117488d4caf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-28T05:35:45.994 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:45 smithi113 podman[74202]: 2024-03-28 05:35:45.746820959 +0000 UTC m=+0.377320631 container start 334b575c3e1737dd7fa28b6565fea3b854a41a2ba7949284ad18c117488d4caf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T05:35:45.994 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:45 smithi113 podman[74202]: 2024-03-28 05:35:45.843762686 +0000 UTC m=+0.474262370 container attach 334b575c3e1737dd7fa28b6565fea3b854a41a2ba7949284ad18c117488d4caf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5) 2024-03-28T05:35:46.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:45 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:45.838+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:46.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:45 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:45.838+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:46.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:45 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:45.838+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:46.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:45 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:45.838+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:46.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:45 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:45.838+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:46.287 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 28 05:35:46 smithi113 podman[74202]: 2024-03-28 05:35:46.028680715 +0000 UTC m=+0.659180402 container died 334b575c3e1737dd7fa28b6565fea3b854a41a2ba7949284ad18c117488d4caf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-28T05:35:46.288 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:46 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:46.019+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:46.288 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:46 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:46.019+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:46.288 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:46 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:46.019+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:46.288 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:46 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:46.019+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:46.288 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:46 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:46.019+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:35:46.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:46 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:46.078+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:46.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:46 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:46.078+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:46.533 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:46 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:46.078+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:46.533 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:46 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:46.078+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:46.533 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:46 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:46.078+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:07.246038+0000 front 2024-03-28T05:35:07.245902+0000 (oldest deadline 2024-03-28T05:35:33.145893+0000) 2024-03-28T05:35:46.533 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:46 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:46.078+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:18.947627+0000 front 2024-03-28T05:35:18.947665+0000 (oldest deadline 2024-03-28T05:35:41.247322+0000) 2024-03-28T05:35:46.674 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.8.service' 2024-03-28T05:35:46.715 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:35:46.715 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-28T05:35:46.715 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-28T05:35:46.715 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.9 2024-03-28T05:35:47.003 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:46 smithi023 systemd[1]: Stopping Ceph osd.9 for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:35:47.003 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:47.002+0000 7f7253728700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:35:47.003 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:47.002+0000 7f7253728700 -1 osd.9 107 *** Got signal Terminated *** 2024-03-28T05:35:47.003 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:47.002+0000 7f7253728700 -1 osd.9 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T05:35:47.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:46 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:46.828+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:47.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:46 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:46.828+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:47.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:46 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:46.828+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:47.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:46 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:46.828+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:47.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:46 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:46.828+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:47.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:46 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:46.828+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:14.067208+0000 front 2024-03-28T05:35:14.067179+0000 (oldest deadline 2024-03-28T05:35:45.867714+0000) 2024-03-28T05:35:47.243 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:46 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:46.992+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:47.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:46 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:46.992+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:47.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:46 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:46.992+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:47.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:46 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:46.992+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:47.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:46 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:46.992+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:35:47.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:47.099+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:47.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:47.099+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:47.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:47.099+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:47.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:47.099+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:47.282 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:47.099+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:07.246038+0000 front 2024-03-28T05:35:07.245902+0000 (oldest deadline 2024-03-28T05:35:33.145893+0000) 2024-03-28T05:35:47.283 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:47 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:47.099+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:18.947627+0000 front 2024-03-28T05:35:18.947665+0000 (oldest deadline 2024-03-28T05:35:41.247322+0000) 2024-03-28T05:35:48.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:47 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:47.822+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:48.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:47 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:47.822+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:48.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:47 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:47.822+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:48.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:47 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:47.822+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:48.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:47 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:47.822+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:48.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:47 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:47.822+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:14.067208+0000 front 2024-03-28T05:35:14.067179+0000 (oldest deadline 2024-03-28T05:35:45.867714+0000) 2024-03-28T05:35:48.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:48 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:48.018+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:48.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:48 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:48.018+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:48.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:48 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:48.018+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:48.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:48 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:48.018+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:48.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:48 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:48.018+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:35:48.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:48 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:48.063+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:48.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:48 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:48.063+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:48.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:48 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:48.063+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:48.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:48 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:48.063+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:48.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:48 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:48.063+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:07.246038+0000 front 2024-03-28T05:35:07.245902+0000 (oldest deadline 2024-03-28T05:35:33.145893+0000) 2024-03-28T05:35:48.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:48 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:48.063+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:18.947627+0000 front 2024-03-28T05:35:18.947665+0000 (oldest deadline 2024-03-28T05:35:41.247322+0000) 2024-03-28T05:35:49.116 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:48 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:48.785+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:49.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:48 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:48.785+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:49.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:48 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:48.785+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:49.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:48 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:48.785+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:49.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:48 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:48.785+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:49.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:48 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:48.785+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:14.067208+0000 front 2024-03-28T05:35:14.067179+0000 (oldest deadline 2024-03-28T05:35:45.867714+0000) 2024-03-28T05:35:49.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:48 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:48.970+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:49.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:48 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:48.970+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:49.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:48 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:48.970+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:49.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:48 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:48.970+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:49.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:48 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:48.970+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:35:49.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:49 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:49.039+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:49.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:49 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:49.039+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:49.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:49 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:49.039+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:49.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:49 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:49.039+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:49.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:49 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:49.039+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:07.246038+0000 front 2024-03-28T05:35:07.245902+0000 (oldest deadline 2024-03-28T05:35:33.145893+0000) 2024-03-28T05:35:49.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:49 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:49.039+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:18.947627+0000 front 2024-03-28T05:35:18.947665+0000 (oldest deadline 2024-03-28T05:35:41.247322+0000) 2024-03-28T05:35:50.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:49 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:49.765+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:50.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:49 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:49.765+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:50.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:49 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:49.765+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:50.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:49 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:49.765+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:50.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:49 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:49.765+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:50.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:49 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:49.765+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:14.067208+0000 front 2024-03-28T05:35:14.067179+0000 (oldest deadline 2024-03-28T05:35:45.867714+0000) 2024-03-28T05:35:50.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:49 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:49.977+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:50.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:49 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:49.977+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:50.245 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:49 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:49.977+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:50.245 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:49 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:49.977+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:50.245 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:49 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:49.977+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:35:50.245 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:49 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:49.977+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:25.324395+0000 front 2024-03-28T05:35:25.324419+0000 (oldest deadline 2024-03-28T05:35:49.324746+0000) 2024-03-28T05:35:50.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:50 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:50.055+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:50.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:50 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:50.055+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:50.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:50 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:50.055+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:50.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:50 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:50.055+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:50.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:50 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:50.055+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:07.246038+0000 front 2024-03-28T05:35:07.245902+0000 (oldest deadline 2024-03-28T05:35:33.145893+0000) 2024-03-28T05:35:50.533 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:50 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:50.055+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:18.947627+0000 front 2024-03-28T05:35:18.947665+0000 (oldest deadline 2024-03-28T05:35:41.247322+0000) 2024-03-28T05:35:51.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:50 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:50.745+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:51.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:50 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:50.745+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:51.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:50 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:50.745+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:51.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:50 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:50.745+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:51.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:50 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:50.745+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:51.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:50 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:50.745+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:14.067208+0000 front 2024-03-28T05:35:14.067179+0000 (oldest deadline 2024-03-28T05:35:45.867714+0000) 2024-03-28T05:35:51.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:50 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:50.990+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:51.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:50 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:50.990+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:51.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:50 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:50.990+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:51.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:50 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:50.990+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:51.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:50 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:50.990+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:35:51.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:50 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:50.990+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:25.324395+0000 front 2024-03-28T05:35:25.324419+0000 (oldest deadline 2024-03-28T05:35:49.324746+0000) 2024-03-28T05:35:51.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:51 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:51.056+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.338082+0000 front 2024-03-28T05:34:38.338110+0000 (oldest deadline 2024-03-28T05:35:01.838310+0000) 2024-03-28T05:35:51.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:51 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:51.056+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:46.841895+0000 front 2024-03-28T05:34:46.841729+0000 (oldest deadline 2024-03-28T05:35:09.741920+0000) 2024-03-28T05:35:51.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:51 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:51.056+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:53.843131+0000 front 2024-03-28T05:34:53.843066+0000 (oldest deadline 2024-03-28T05:35:21.444013+0000) 2024-03-28T05:35:51.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:51 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:51.056+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.145072+0000 front 2024-03-28T05:35:03.145174+0000 (oldest deadline 2024-03-28T05:35:27.245271+0000) 2024-03-28T05:35:51.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:51 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:51.056+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:07.246038+0000 front 2024-03-28T05:35:07.245902+0000 (oldest deadline 2024-03-28T05:35:33.145893+0000) 2024-03-28T05:35:51.532 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:51 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9[67040]: 2024-03-28T05:35:51.056+0000 7f724fd1c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:18.947627+0000 front 2024-03-28T05:35:18.947665+0000 (oldest deadline 2024-03-28T05:35:41.247322+0000) 2024-03-28T05:35:52.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:51 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:51.706+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:52.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:51 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:51.706+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:52.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:51 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:51.706+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:52.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:51 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:51.706+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:52.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:51 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:51.706+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:52.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:51 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:51.706+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:14.067208+0000 front 2024-03-28T05:35:14.067179+0000 (oldest deadline 2024-03-28T05:35:45.867714+0000) 2024-03-28T05:35:52.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:52 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:52.014+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:52.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:52 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:52.014+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:52.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:52 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:52.014+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:52.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:52 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:52.014+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:52.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:52 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:52.014+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:35:52.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:52 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:52.014+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:25.324395+0000 front 2024-03-28T05:35:25.324419+0000 (oldest deadline 2024-03-28T05:35:49.324746+0000) 2024-03-28T05:35:52.712 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:52 smithi023 podman[96707]: 2024-03-28 05:35:52.4355494 +0000 UTC m=+5.524486072 container died aa6939a2d2b88ed30cb5ca155e4cb168b6fae2063d83c915d0a09a6d6083b7dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-28T05:35:53.032 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:52 smithi023 podman[96707]: 2024-03-28 05:35:52.802901641 +0000 UTC m=+5.891838308 container cleanup aa6939a2d2b88ed30cb5ca155e4cb168b6fae2063d83c915d0a09a6d6083b7dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-28T05:35:53.032 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:52 smithi023 bash[96707]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9 2024-03-28T05:35:53.116 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:52 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:52.730+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:53.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:52 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:52.730+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:53.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:52 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:52.730+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:53.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:52 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:52.730+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:53.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:52 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:52.730+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:53.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:52 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:52.730+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:14.067208+0000 front 2024-03-28T05:35:14.067179+0000 (oldest deadline 2024-03-28T05:35:45.867714+0000) 2024-03-28T05:35:53.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:52 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:52.730+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:26.968722+0000 front 2024-03-28T05:35:26.968704+0000 (oldest deadline 2024-03-28T05:35:52.268497+0000) 2024-03-28T05:35:53.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:52 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:52.988+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:53.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:52 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:52.988+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:53.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:52 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:52.988+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:53.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:52 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:52.988+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:53.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:52 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:52.988+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:35:53.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:52 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:52.988+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:25.324395+0000 front 2024-03-28T05:35:25.324419+0000 (oldest deadline 2024-03-28T05:35:49.324746+0000) 2024-03-28T05:35:53.416 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:53 smithi023 podman[96724]: 2024-03-28 05:35:53.054043093 +0000 UTC m=+0.613144071 container remove aa6939a2d2b88ed30cb5ca155e4cb168b6fae2063d83c915d0a09a6d6083b7dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T05:35:53.682 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:53 smithi023 podman[96816]: 2024-03-28 05:35:53.31957054 +0000 UTC m=+0.017619919 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:35:53.682 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:53 smithi023 podman[96816]: 2024-03-28 05:35:53.421403115 +0000 UTC m=+0.119452473 container create db161aac9211cdc473c0129a372b7f4250c48121d9beeb018f297d42dba6bdad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T05:35:53.682 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:53 smithi023 podman[96816]: 2024-03-28 05:35:53.607001606 +0000 UTC m=+0.305050968 container init db161aac9211cdc473c0129a372b7f4250c48121d9beeb018f297d42dba6bdad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T05:35:53.682 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:53 smithi023 podman[96816]: 2024-03-28 05:35:53.610076077 +0000 UTC m=+0.308125440 container start db161aac9211cdc473c0129a372b7f4250c48121d9beeb018f297d42dba6bdad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9-deactivate, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T05:35:53.682 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:53 smithi023 podman[96816]: 2024-03-28 05:35:53.681997091 +0000 UTC m=+0.380046462 container attach db161aac9211cdc473c0129a372b7f4250c48121d9beeb018f297d42dba6bdad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9-deactivate, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, ceph=True) 2024-03-28T05:35:54.032 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:53 smithi023 podman[96816]: 2024-03-28 05:35:53.909282879 +0000 UTC m=+0.607332247 container died db161aac9211cdc473c0129a372b7f4250c48121d9beeb018f297d42dba6bdad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T05:35:54.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:53 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:53.745+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:54.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:53 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:53.745+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:54.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:53 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:53.745+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:54.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:53 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:53.745+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:54.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:53 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:53.745+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:54.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:53 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:53.745+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:14.067208+0000 front 2024-03-28T05:35:14.067179+0000 (oldest deadline 2024-03-28T05:35:45.867714+0000) 2024-03-28T05:35:54.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:53 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:53.745+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:26.968722+0000 front 2024-03-28T05:35:26.968704+0000 (oldest deadline 2024-03-28T05:35:52.268497+0000) 2024-03-28T05:35:54.449 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 05:35:54 smithi023 podman[96842]: 2024-03-28 05:35:54.427231549 +0000 UTC m=+0.512816500 container remove db161aac9211cdc473c0129a372b7f4250c48121d9beeb018f297d42dba6bdad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T05:35:54.454 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.9.service' 2024-03-28T05:35:54.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:54 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:54.038+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:54.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:54 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:54.038+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:54.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:54 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:54.038+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:54.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:54 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:54.038+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:54.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:54 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:54.038+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:35:54.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:54 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:54.038+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:25.324395+0000 front 2024-03-28T05:35:25.324419+0000 (oldest deadline 2024-03-28T05:35:49.324746+0000) 2024-03-28T05:35:54.496 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:35:54.496 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-28T05:35:54.496 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-28T05:35:54.496 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.10 2024-03-28T05:35:54.784 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:54 smithi098 systemd[1]: Stopping Ceph osd.10 for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:35:55.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:54 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:54.784+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:55.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:54 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:54.784+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:55.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:54 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:54.784+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:55.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:54 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:54.784+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:55.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:54 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:54.784+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:55.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:54 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:54.784+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:14.067208+0000 front 2024-03-28T05:35:14.067179+0000 (oldest deadline 2024-03-28T05:35:45.867714+0000) 2024-03-28T05:35:55.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:54 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:54.784+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:26.968722+0000 front 2024-03-28T05:35:26.968704+0000 (oldest deadline 2024-03-28T05:35:52.268497+0000) 2024-03-28T05:35:55.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:54 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:54.821+0000 7fa8fd5f4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:35:55.118 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:54 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:54.821+0000 7fa8fd5f4700 -1 osd.10 107 *** Got signal Terminated *** 2024-03-28T05:35:55.118 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:54 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:54.821+0000 7fa8fd5f4700 -1 osd.10 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T05:35:55.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:55 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:55.071+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:55.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:55 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:55.071+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:55.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:55 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:55.071+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:55.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:55 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:55.071+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:55.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:55 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:55.071+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:35:55.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:55 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:55.071+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:25.324395+0000 front 2024-03-28T05:35:25.324419+0000 (oldest deadline 2024-03-28T05:35:49.324746+0000) 2024-03-28T05:35:56.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:55 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:55.738+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:56.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:55 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:55.738+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:56.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:55 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:55.738+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:56.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:55 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:55.738+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:56.118 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:55 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:55.738+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:56.118 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:55 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:55.738+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:14.067208+0000 front 2024-03-28T05:35:14.067179+0000 (oldest deadline 2024-03-28T05:35:45.867714+0000) 2024-03-28T05:35:56.118 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:55 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:55.738+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:26.968722+0000 front 2024-03-28T05:35:26.968704+0000 (oldest deadline 2024-03-28T05:35:52.268497+0000) 2024-03-28T05:35:56.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:56 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:56.107+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:56.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:56 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:56.107+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:56.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:56 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:56.107+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:56.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:56 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:56.107+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:56.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:56 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:56.107+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:35:56.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:56 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:56.107+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:25.324395+0000 front 2024-03-28T05:35:25.324419+0000 (oldest deadline 2024-03-28T05:35:49.324746+0000) 2024-03-28T05:35:57.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:56 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:56.709+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:57.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:56 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:56.709+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:57.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:56 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:56.709+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:57.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:56 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:56.709+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:57.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:56 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:56.709+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:57.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:56 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:56.709+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:14.067208+0000 front 2024-03-28T05:35:14.067179+0000 (oldest deadline 2024-03-28T05:35:45.867714+0000) 2024-03-28T05:35:57.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:56 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:56.709+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:26.968722+0000 front 2024-03-28T05:35:26.968704+0000 (oldest deadline 2024-03-28T05:35:52.268497+0000) 2024-03-28T05:35:57.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:57 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:57.099+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:57.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:57 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:57.099+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:57.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:57 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:57.099+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:57.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:57 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:57.099+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:57.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:57 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:57.099+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:35:57.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:57 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:57.099+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:25.324395+0000 front 2024-03-28T05:35:25.324419+0000 (oldest deadline 2024-03-28T05:35:49.324746+0000) 2024-03-28T05:35:58.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:57 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:57.715+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:58.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:57 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:57.715+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:58.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:57 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:57.715+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:58.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:57 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:57.715+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:58.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:57 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:57.715+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:58.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:57 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:57.715+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:14.067208+0000 front 2024-03-28T05:35:14.067179+0000 (oldest deadline 2024-03-28T05:35:45.867714+0000) 2024-03-28T05:35:58.118 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:57 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:57.715+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:26.968722+0000 front 2024-03-28T05:35:26.968704+0000 (oldest deadline 2024-03-28T05:35:52.268497+0000) 2024-03-28T05:35:58.396 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:58 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:58.067+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:58.396 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:58 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:58.067+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:58.396 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:58 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:58.067+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:58.396 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:58 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:58.067+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:58.396 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:58 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:58.067+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:35:58.396 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:58 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:58.067+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:25.324395+0000 front 2024-03-28T05:35:25.324419+0000 (oldest deadline 2024-03-28T05:35:49.324746+0000) 2024-03-28T05:35:59.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:58 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:58.715+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:35:59.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:58 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:58.715+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:35:59.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:58 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:58.715+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:35:59.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:58 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:58.715+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:35:59.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:58 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:58.715+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:35:59.118 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:58 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:58.715+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:14.067208+0000 front 2024-03-28T05:35:14.067179+0000 (oldest deadline 2024-03-28T05:35:45.867714+0000) 2024-03-28T05:35:59.118 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:58 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:58.715+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:26.968722+0000 front 2024-03-28T05:35:26.968704+0000 (oldest deadline 2024-03-28T05:35:52.268497+0000) 2024-03-28T05:35:59.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:59 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:59.078+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:35:59.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:59 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:59.078+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:35:59.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:59 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:59.078+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:35:59.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:59 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:59.078+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:35:59.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:59 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:59.078+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:35:59.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:35:59 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:35:59.078+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:25.324395+0000 front 2024-03-28T05:35:25.324419+0000 (oldest deadline 2024-03-28T05:35:49.324746+0000) 2024-03-28T05:36:00.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:59 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:59.744+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:38.362675+0000 front 2024-03-28T05:34:38.362587+0000 (oldest deadline 2024-03-28T05:35:03.662685+0000) 2024-03-28T05:36:00.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:59 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:59.744+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6804 osd.1 since back 2024-03-28T05:34:43.663153+0000 front 2024-03-28T05:34:43.663178+0000 (oldest deadline 2024-03-28T05:35:08.963348+0000) 2024-03-28T05:36:00.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:59 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:59.744+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:51.864367+0000 front 2024-03-28T05:34:51.864183+0000 (oldest deadline 2024-03-28T05:35:17.764073+0000) 2024-03-28T05:36:00.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:59 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:59.744+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:03.065385+0000 front 2024-03-28T05:35:03.065217+0000 (oldest deadline 2024-03-28T05:35:26.565137+0000) 2024-03-28T05:36:00.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:59 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:59.744+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.166383+0000 front 2024-03-28T05:35:11.166335+0000 (oldest deadline 2024-03-28T05:35:34.066520+0000) 2024-03-28T05:36:00.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:59 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:59.744+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:14.067208+0000 front 2024-03-28T05:35:14.067179+0000 (oldest deadline 2024-03-28T05:35:45.867714+0000) 2024-03-28T05:36:00.118 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:35:59 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10[60405]: 2024-03-28T05:35:59.744+0000 7fa8f9be8700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:26.968722+0000 front 2024-03-28T05:35:26.968704+0000 (oldest deadline 2024-03-28T05:35:52.268497+0000) 2024-03-28T05:36:00.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:00 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:00.122+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:36:00.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:00 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:00.122+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:36:00.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:00 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:00.122+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:36:00.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:00 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:00.122+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:36:00.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:00 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:00.122+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:36:00.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:00 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:00.122+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:25.324395+0000 front 2024-03-28T05:35:25.324419+0000 (oldest deadline 2024-03-28T05:35:49.324746+0000) 2024-03-28T05:36:00.673 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:36:00 smithi098 podman[72400]: 2024-03-28 05:36:00.379778055 +0000 UTC m=+5.689013131 container died 365de499674e2776fc65ef18e29216d8e084cefe2fd984707270cbcec406058d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-28T05:36:01.023 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:36:00 smithi098 podman[72400]: 2024-03-28 05:36:00.773288098 +0000 UTC m=+6.082523133 container cleanup 365de499674e2776fc65ef18e29216d8e084cefe2fd984707270cbcec406058d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-28T05:36:01.024 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:36:00 smithi098 bash[72400]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10 2024-03-28T05:36:01.338 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:36:01 smithi098 podman[72417]: 2024-03-28 05:36:01.024029733 +0000 UTC m=+0.643408793 container remove 365de499674e2776fc65ef18e29216d8e084cefe2fd984707270cbcec406058d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-28T05:36:01.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:01 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:01.149+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:36:01.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:01 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:01.149+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:36:01.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:01 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:01.149+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:36:01.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:01 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:01.149+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:36:01.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:01 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:01.149+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:36:01.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:01 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:01.149+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:25.324395+0000 front 2024-03-28T05:35:25.324419+0000 (oldest deadline 2024-03-28T05:35:49.324746+0000) 2024-03-28T05:36:01.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:01 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:01.149+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-03-28T05:35:34.526322+0000 front 2024-03-28T05:35:34.526380+0000 (oldest deadline 2024-03-28T05:36:00.426248+0000) 2024-03-28T05:36:01.617 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:36:01 smithi098 podman[72510]: 2024-03-28 05:36:01.24211494 +0000 UTC m=+0.017651041 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:36:01.617 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:36:01 smithi098 podman[72510]: 2024-03-28 05:36:01.370703822 +0000 UTC m=+0.146239959 container create c8019f6a823ee0fa66c79f630f735a45db46980d83a3aefb5671d9390c40ed4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-28T05:36:01.617 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:36:01 smithi098 podman[72510]: 2024-03-28 05:36:01.585711442 +0000 UTC m=+0.361247545 container init c8019f6a823ee0fa66c79f630f735a45db46980d83a3aefb5671d9390c40ed4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-28T05:36:01.617 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:36:01 smithi098 podman[72510]: 2024-03-28 05:36:01.588714694 +0000 UTC m=+0.364250780 container start c8019f6a823ee0fa66c79f630f735a45db46980d83a3aefb5671d9390c40ed4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-28T05:36:02.116 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:36:01 smithi098 podman[72510]: 2024-03-28 05:36:01.677530989 +0000 UTC m=+0.453067096 container attach c8019f6a823ee0fa66c79f630f735a45db46980d83a3aefb5671d9390c40ed4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-28T05:36:02.117 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:36:01 smithi098 podman[72510]: 2024-03-28 05:36:01.883075281 +0000 UTC m=+0.658611383 container died c8019f6a823ee0fa66c79f630f735a45db46980d83a3aefb5671d9390c40ed4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-28T05:36:02.405 INFO:journalctl@ceph.osd.10.smithi098.stdout:Mar 28 05:36:02 smithi098 podman[72536]: 2024-03-28 05:36:02.389130032 +0000 UTC m=+0.500878670 container remove c8019f6a823ee0fa66c79f630f735a45db46980d83a3aefb5671d9390c40ed4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-10-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-28T05:36:02.410 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.10.service' 2024-03-28T05:36:02.450 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:36:02.450 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-28T05:36:02.451 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-28T05:36:02.451 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.11 2024-03-28T05:36:02.480 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:02 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:02.151+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:36:02.480 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:02 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:02.151+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:36:02.480 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:02 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:02.151+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:36:02.480 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:02 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:02.151+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:36:02.480 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:02 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:02.151+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:36:02.480 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:02 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:02.151+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:25.324395+0000 front 2024-03-28T05:35:25.324419+0000 (oldest deadline 2024-03-28T05:35:49.324746+0000) 2024-03-28T05:36:02.481 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:02 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:02.151+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-03-28T05:35:34.526322+0000 front 2024-03-28T05:35:34.526380+0000 (oldest deadline 2024-03-28T05:36:00.426248+0000) 2024-03-28T05:36:02.744 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:02 smithi113 systemd[1]: Stopping Ceph osd.11 for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:36:03.149 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:02 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:02.766+0000 7fc01710d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:36:03.149 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:02 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:02.766+0000 7fc01710d700 -1 osd.11 107 *** Got signal Terminated *** 2024-03-28T05:36:03.149 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:02 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:02.766+0000 7fc01710d700 -1 osd.11 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T05:36:03.149 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:03 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:03.149+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:36:03.149 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:03 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:03.149+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:36:03.149 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:03 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:03.149+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:36:03.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:03 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:03.149+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:36:03.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:03 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:03.149+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:36:03.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:03 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:03.149+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:25.324395+0000 front 2024-03-28T05:35:25.324419+0000 (oldest deadline 2024-03-28T05:35:49.324746+0000) 2024-03-28T05:36:03.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:03 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:03.149+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-03-28T05:35:34.526322+0000 front 2024-03-28T05:35:34.526380+0000 (oldest deadline 2024-03-28T05:36:00.426248+0000) 2024-03-28T05:36:04.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:04 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:04.176+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:36:04.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:04 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:04.176+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:36:04.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:04 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:04.176+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:36:04.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:04 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:04.176+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:36:04.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:04 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:04.176+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:36:04.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:04 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:04.176+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:25.324395+0000 front 2024-03-28T05:35:25.324419+0000 (oldest deadline 2024-03-28T05:35:49.324746+0000) 2024-03-28T05:36:04.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:04 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:04.176+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-03-28T05:35:34.526322+0000 front 2024-03-28T05:35:34.526380+0000 (oldest deadline 2024-03-28T05:36:00.426248+0000) 2024-03-28T05:36:05.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:05 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:05.135+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:36:05.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:05 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:05.135+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:36:05.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:05 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:05.135+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:36:05.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:05 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:05.135+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:36:05.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:05 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:05.135+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:36:05.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:05 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:05.135+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:25.324395+0000 front 2024-03-28T05:35:25.324419+0000 (oldest deadline 2024-03-28T05:35:49.324746+0000) 2024-03-28T05:36:05.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:05 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:05.135+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-03-28T05:35:34.526322+0000 front 2024-03-28T05:35:34.526380+0000 (oldest deadline 2024-03-28T05:36:00.426248+0000) 2024-03-28T05:36:06.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:06 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:06.109+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:36:06.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:06 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:06.109+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:36:06.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:06 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:06.109+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:36:06.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:06 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:06.109+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:36:06.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:06 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:06.109+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:36:06.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:06 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:06.109+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:25.324395+0000 front 2024-03-28T05:35:25.324419+0000 (oldest deadline 2024-03-28T05:35:49.324746+0000) 2024-03-28T05:36:06.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:06 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:06.109+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-03-28T05:35:34.526322+0000 front 2024-03-28T05:35:34.526380+0000 (oldest deadline 2024-03-28T05:36:00.426248+0000) 2024-03-28T05:36:06.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:06 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:06.109+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6820 osd.8 since back 2024-03-28T05:35:42.126996+0000 front 2024-03-28T05:35:42.127065+0000 (oldest deadline 2024-03-28T05:36:05.627011+0000) 2024-03-28T05:36:07.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:07 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:07.076+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T05:34:36.718030+0000 front 2024-03-28T05:34:36.718046+0000 (oldest deadline 2024-03-28T05:35:01.417985+0000) 2024-03-28T05:36:07.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:07 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:07.076+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-28T05:34:54.320117+0000 front 2024-03-28T05:34:54.319994+0000 (oldest deadline 2024-03-28T05:35:19.020213+0000) 2024-03-28T05:36:07.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:07 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:07.076+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T05:35:01.921248+0000 front 2024-03-28T05:35:01.921296+0000 (oldest deadline 2024-03-28T05:35:25.421222+0000) 2024-03-28T05:36:07.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:07 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:07.076+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6812 osd.4 since back 2024-03-28T05:35:11.322453+0000 front 2024-03-28T05:35:11.322308+0000 (oldest deadline 2024-03-28T05:35:34.822244+0000) 2024-03-28T05:36:07.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:07 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:07.076+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6812 osd.5 since back 2024-03-28T05:35:19.523443+0000 front 2024-03-28T05:35:19.523426+0000 (oldest deadline 2024-03-28T05:35:44.823483+0000) 2024-03-28T05:36:07.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:07 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:07.076+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T05:35:25.324395+0000 front 2024-03-28T05:35:25.324419+0000 (oldest deadline 2024-03-28T05:35:49.324746+0000) 2024-03-28T05:36:07.494 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:07 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:07.076+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.98:6820 osd.7 since back 2024-03-28T05:35:34.526322+0000 front 2024-03-28T05:35:34.526380+0000 (oldest deadline 2024-03-28T05:36:00.426248+0000) 2024-03-28T05:36:07.495 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:07 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11[62643]: 2024-03-28T05:36:07.076+0000 7fc00f6f9700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.113:6820 osd.8 since back 2024-03-28T05:35:42.126996+0000 front 2024-03-28T05:35:42.127065+0000 (oldest deadline 2024-03-28T05:36:05.627011+0000) 2024-03-28T05:36:08.599 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:08 smithi113 podman[74320]: 2024-03-28 05:36:08.273075561 +0000 UTC m=+5.630932694 container died 9abf1efc270c385a4a83efb322135b9d7cb0839a1257458fb3c3f466dee2a948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-28T05:36:08.891 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:08 smithi113 podman[74320]: 2024-03-28 05:36:08.697952286 +0000 UTC m=+6.055809380 container cleanup 9abf1efc270c385a4a83efb322135b9d7cb0839a1257458fb3c3f466dee2a948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-28T05:36:08.891 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:08 smithi113 bash[74320]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11 2024-03-28T05:36:09.244 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:08 smithi113 podman[74337]: 2024-03-28 05:36:08.891872575 +0000 UTC m=+0.609851581 container remove 9abf1efc270c385a4a83efb322135b9d7cb0839a1257458fb3c3f466dee2a948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T05:36:09.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:09 smithi113 podman[74428]: 2024-03-28 05:36:09.301001559 +0000 UTC m=+0.108381085 container create 015b2633e1893648e9bdcf9c76804f1d917085efd2bb7beb2d4875c80bfa0f60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T05:36:09.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:09 smithi113 podman[74428]: 2024-03-28 05:36:09.210052899 +0000 UTC m=+0.017432427 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T05:36:09.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:09 smithi113 podman[74428]: 2024-03-28 05:36:09.511728999 +0000 UTC m=+0.319108509 container init 015b2633e1893648e9bdcf9c76804f1d917085efd2bb7beb2d4875c80bfa0f60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2) 2024-03-28T05:36:09.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:09 smithi113 podman[74428]: 2024-03-28 05:36:09.51481386 +0000 UTC m=+0.322193360 container start 015b2633e1893648e9bdcf9c76804f1d917085efd2bb7beb2d4875c80bfa0f60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-28T05:36:09.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:09 smithi113 podman[74428]: 2024-03-28 05:36:09.578191553 +0000 UTC m=+0.385571077 container attach 015b2633e1893648e9bdcf9c76804f1d917085efd2bb7beb2d4875c80bfa0f60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T05:36:09.994 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 28 05:36:09 smithi113 podman[74428]: 2024-03-28 05:36:09.805827075 +0000 UTC m=+0.613206588 container died 015b2633e1893648e9bdcf9c76804f1d917085efd2bb7beb2d4875c80bfa0f60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-osd-11-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319) 2024-03-28T05:36:10.407 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@osd.11.service' 2024-03-28T05:36:10.489 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:36:10.489 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-28T05:36:10.489 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-28T05:36:10.489 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.a 2024-03-28T05:36:10.782 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 05:36:10 smithi023 systemd[1]: Stopping Ceph mds.a for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:36:11.282 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 05:36:10 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-a[73886]: 2024-03-28T05:36:10.804+0000 7f27fefa8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:36:11.282 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 05:36:10 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-a[73886]: 2024-03-28T05:36:10.804+0000 7f27fefa8700 -1 mds.a *** got signal Terminated *** 2024-03-28T05:36:13.429 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 05:36:13 smithi023 podman[96938]: 2024-03-28 05:36:13.069693913 +0000 UTC m=+2.386963824 container died 85fb08824ad8e3f29dabb037e4bd1093220c7e2eebb46dbc3a98876c6792d713 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-a, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-28T05:36:13.687 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 05:36:13 smithi023 podman[96938]: 2024-03-28 05:36:13.503956882 +0000 UTC m=+2.821226841 container cleanup 85fb08824ad8e3f29dabb037e4bd1093220c7e2eebb46dbc3a98876c6792d713 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T05:36:13.700 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 05:36:13 smithi023 bash[96938]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-a 2024-03-28T05:36:13.923 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.a.service' 2024-03-28T05:36:13.947 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 05:36:13 smithi023 podman[96952]: 2024-03-28 05:36:13.688253053 +0000 UTC m=+0.617454718 container remove 85fb08824ad8e3f29dabb037e4bd1093220c7e2eebb46dbc3a98876c6792d713 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T05:36:13.947 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 05:36:13 smithi023 systemd[1]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.a.service: Deactivated successfully. 2024-03-28T05:36:13.947 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 05:36:13 smithi023 systemd[1]: Stopped Ceph mds.a for bf37ebc4-ecc1-11ee-95d0-87774f69a715. 2024-03-28T05:36:13.947 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 05:36:13 smithi023 systemd[1]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.a.service: Consumed 1.050s CPU time. 2024-03-28T05:36:13.961 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:36:13.961 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-28T05:36:13.961 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-28T05:36:13.961 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.d 2024-03-28T05:36:14.282 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 05:36:14 smithi023 systemd[1]: Stopping Ceph mds.d for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:36:14.782 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 05:36:14 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-d[72923]: 2024-03-28T05:36:14.283+0000 7f6b3a074700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:36:14.782 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 05:36:14 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-d[72923]: 2024-03-28T05:36:14.283+0000 7f6b3a074700 -1 mds.d *** got signal Terminated *** 2024-03-28T05:36:17.608 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 05:36:17 smithi023 podman[97059]: 2024-03-28 05:36:17.336375294 +0000 UTC m=+3.155735078 container died c2781f8fc1e55dbd0626d42fa6b8fe6712e0928d8976c859d898d9f43a89dcc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-28T05:36:18.032 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 05:36:17 smithi023 podman[97059]: 2024-03-28 05:36:17.699281338 +0000 UTC m=+3.518641122 container cleanup c2781f8fc1e55dbd0626d42fa6b8fe6712e0928d8976c859d898d9f43a89dcc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-d, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-28T05:36:18.032 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 05:36:17 smithi023 bash[97059]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-d 2024-03-28T05:36:18.282 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.d.service' 2024-03-28T05:36:18.316 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:36:18.317 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-28T05:36:18.317 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-28T05:36:18.317 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.g 2024-03-28T05:36:18.636 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 05:36:18 smithi023 systemd[1]: Stopping Ceph mds.g for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:36:19.032 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 05:36:18 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-g[73521]: 2024-03-28T05:36:18.636+0000 7f6dd72d6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:36:19.032 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 05:36:18 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-g[73521]: 2024-03-28T05:36:18.636+0000 7f6dd72d6700 -1 mds.g *** got signal Terminated *** 2024-03-28T05:36:22.577 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 05:36:22 smithi023 podman[97178]: 2024-03-28 05:36:22.318880409 +0000 UTC m=+3.786640089 container died 155fb204768b62ac186e523df983b29d0016fc985240d9276636143b71ca8e5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-g, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0) 2024-03-28T05:36:22.577 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 05:36:22 smithi023 podman[97178]: 2024-03-28 05:36:22.577794826 +0000 UTC m=+4.045554560 container cleanup 155fb204768b62ac186e523df983b29d0016fc985240d9276636143b71ca8e5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-g, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-28T05:36:22.845 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 05:36:22 smithi023 bash[97178]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-g 2024-03-28T05:36:22.990 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.g.service' 2024-03-28T05:36:23.024 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:36:23.025 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-28T05:36:23.025 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-28T05:36:23.025 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.j 2024-03-28T05:36:23.532 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 05:36:23 smithi023 systemd[1]: Stopping Ceph mds.j for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:36:23.532 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 05:36:23 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-j[72565]: 2024-03-28T05:36:23.314+0000 7f84df3fa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:36:23.532 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 05:36:23 smithi023 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-j[72565]: 2024-03-28T05:36:23.314+0000 7f84df3fa700 -1 mds.j *** got signal Terminated *** 2024-03-28T05:36:26.782 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 05:36:26 smithi023 podman[97300]: 2024-03-28 05:36:26.503383171 +0000 UTC m=+3.259590267 container died 13a59a33c52c1832caed2960329f25071c9aae086672c1f1192e58189f84cf5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-28T05:36:27.042 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 05:36:26 smithi023 podman[97300]: 2024-03-28 05:36:26.889799447 +0000 UTC m=+3.646006540 container cleanup 13a59a33c52c1832caed2960329f25071c9aae086672c1f1192e58189f84cf5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-j, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True) 2024-03-28T05:36:27.042 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 05:36:26 smithi023 bash[97300]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-j 2024-03-28T05:36:27.248 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.j.service' 2024-03-28T05:36:27.282 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:36:27.282 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-28T05:36:27.282 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-28T05:36:27.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.b 2024-03-28T05:36:27.566 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 28 05:36:27 smithi098 systemd[1]: Stopping Ceph mds.b for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:36:27.866 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 28 05:36:27 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-b[65800]: 2024-03-28T05:36:27.564+0000 7fbadd5e1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:36:27.867 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 28 05:36:27 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-b[65800]: 2024-03-28T05:36:27.564+0000 7fbadd5e1700 -1 mds.b *** got signal Terminated *** 2024-03-28T05:36:29.149 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 28 05:36:28 smithi098 podman[72629]: 2024-03-28 05:36:28.875856191 +0000 UTC m=+1.401071178 container died 61e83ec3302d427d3a1c78950c5577f797524c0da590b6c4b861b75f8e72fb58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-28T05:36:29.515 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 28 05:36:29 smithi098 podman[72629]: 2024-03-28 05:36:29.248274911 +0000 UTC m=+1.773489875 container cleanup 61e83ec3302d427d3a1c78950c5577f797524c0da590b6c4b861b75f8e72fb58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-b, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-28T05:36:29.516 INFO:journalctl@ceph.mds.b.smithi098.stdout:Mar 28 05:36:29 smithi098 bash[72629]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-b 2024-03-28T05:36:29.657 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.b.service' 2024-03-28T05:36:29.694 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:36:29.694 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-28T05:36:29.694 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-28T05:36:29.694 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.e 2024-03-28T05:36:30.117 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 28 05:36:29 smithi098 systemd[1]: Stopping Ceph mds.e for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:36:30.117 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 28 05:36:30 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-e[65184]: 2024-03-28T05:36:30.018+0000 7f43f52e0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:36:30.117 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 28 05:36:30 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-e[65184]: 2024-03-28T05:36:30.018+0000 7f43f52e0700 -1 mds.e *** got signal Terminated *** 2024-03-28T05:36:32.468 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 28 05:36:32 smithi098 podman[72750]: 2024-03-28 05:36:32.192389371 +0000 UTC m=+2.281243096 container died 5ec0c7c7e8b8b4a924685dc50245177a3805f0e63ee1380001505fd51e39cfea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-28T05:36:32.866 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 28 05:36:32 smithi098 podman[72750]: 2024-03-28 05:36:32.568731823 +0000 UTC m=+2.657585545 container cleanup 5ec0c7c7e8b8b4a924685dc50245177a3805f0e63ee1380001505fd51e39cfea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-e, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-28T05:36:32.867 INFO:journalctl@ceph.mds.e.smithi098.stdout:Mar 28 05:36:32 smithi098 bash[72750]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-e 2024-03-28T05:36:33.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.e.service' 2024-03-28T05:36:33.074 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:36:33.075 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-28T05:36:33.075 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-28T05:36:33.075 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.h 2024-03-28T05:36:33.406 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 28 05:36:33 smithi098 systemd[1]: Stopping Ceph mds.h for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:36:33.866 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 28 05:36:33 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-h[65492]: 2024-03-28T05:36:33.405+0000 7f3dca1f1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:36:33.867 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 28 05:36:33 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-h[65492]: 2024-03-28T05:36:33.405+0000 7f3dca1f1700 -1 mds.h *** got signal Terminated *** 2024-03-28T05:36:37.048 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 28 05:36:36 smithi098 podman[72870]: 2024-03-28 05:36:36.764588559 +0000 UTC m=+3.472653008 container died 2747c54fcb7d4167f60cab80b397a75ae5b53a6ddc109554ad3f16be1b5608e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-h, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T05:36:37.367 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 28 05:36:37 smithi098 podman[72870]: 2024-03-28 05:36:37.147713283 +0000 UTC m=+3.855777759 container cleanup 2747c54fcb7d4167f60cab80b397a75ae5b53a6ddc109554ad3f16be1b5608e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-h, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-28T05:36:37.367 INFO:journalctl@ceph.mds.h.smithi098.stdout:Mar 28 05:36:37 smithi098 bash[72870]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-h 2024-03-28T05:36:37.569 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.h.service' 2024-03-28T05:36:37.606 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:36:37.606 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-28T05:36:37.606 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-28T05:36:37.606 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.k 2024-03-28T05:36:38.117 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 28 05:36:37 smithi098 systemd[1]: Stopping Ceph mds.k for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:36:38.117 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 28 05:36:37 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-k[64878]: 2024-03-28T05:36:37.935+0000 7feb9899e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:36:38.117 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 28 05:36:37 smithi098 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-k[64878]: 2024-03-28T05:36:37.935+0000 7feb9899e700 -1 mds.k *** got signal Terminated *** 2024-03-28T05:36:40.052 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 28 05:36:39 smithi098 podman[73004]: 2024-03-28 05:36:39.770288964 +0000 UTC m=+1.947276306 container died 925761f3a98891341cbdd00d41be8fd849eb80b7e0b2dac70b5095f9e10b1464 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-k, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-28T05:36:40.366 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 28 05:36:40 smithi098 podman[73004]: 2024-03-28 05:36:40.135161033 +0000 UTC m=+2.312148381 container cleanup 925761f3a98891341cbdd00d41be8fd849eb80b7e0b2dac70b5095f9e10b1464 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-k, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-28T05:36:40.367 INFO:journalctl@ceph.mds.k.smithi098.stdout:Mar 28 05:36:40 smithi098 bash[73004]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-k 2024-03-28T05:36:40.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.k.service' 2024-03-28T05:36:40.591 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:36:40.591 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-28T05:36:40.591 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-28T05:36:40.591 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.c 2024-03-28T05:36:40.994 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 28 05:36:40 smithi113 systemd[1]: Stopping Ceph mds.c for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:36:41.253 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 28 05:36:40 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-c[67826]: 2024-03-28T05:36:40.997+0000 7fec16f5b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:36:41.253 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 28 05:36:40 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-c[67826]: 2024-03-28T05:36:40.997+0000 7fec16f5b700 -1 mds.c *** got signal Terminated *** 2024-03-28T05:36:45.280 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 28 05:36:45 smithi113 podman[74533]: 2024-03-28 05:36:45.005449031 +0000 UTC m=+4.216350707 container died 9a95cbf359a8f82b456b0310d42b9c53ba465d314e574159014b86d7a967a0b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True) 2024-03-28T05:36:45.655 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 28 05:36:45 smithi113 podman[74533]: 2024-03-28 05:36:45.362752524 +0000 UTC m=+4.573654200 container cleanup 9a95cbf359a8f82b456b0310d42b9c53ba465d314e574159014b86d7a967a0b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-c, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-28T05:36:45.655 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 28 05:36:45 smithi113 bash[74533]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-c 2024-03-28T05:36:45.786 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.c.service' 2024-03-28T05:36:45.824 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:36:45.824 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-28T05:36:45.824 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-28T05:36:45.824 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.f 2024-03-28T05:36:45.994 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 28 05:36:45 smithi113 systemd[1]: Stopping Ceph mds.f for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:36:46.494 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 28 05:36:46 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-f[67499]: 2024-03-28T05:36:46.165+0000 7fedcd805700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:36:46.494 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 28 05:36:46 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-f[67499]: 2024-03-28T05:36:46.165+0000 7fedcd805700 -1 mds.f *** got signal Terminated *** 2024-03-28T05:36:50.458 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 28 05:36:50 smithi113 podman[74669]: 2024-03-28 05:36:50.176025119 +0000 UTC m=+4.137746219 container died 3e4fd97c12f5fd7b2f52f22845983de77ad9220c2032b5bcd9a76eba6cbe8b68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-f, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T05:36:50.744 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 28 05:36:50 smithi113 podman[74669]: 2024-03-28 05:36:50.549213259 +0000 UTC m=+4.510934350 container cleanup 3e4fd97c12f5fd7b2f52f22845983de77ad9220c2032b5bcd9a76eba6cbe8b68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-f, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-28T05:36:50.744 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 28 05:36:50 smithi113 bash[74669]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-f 2024-03-28T05:36:50.916 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.f.service' 2024-03-28T05:36:50.951 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:36:50.951 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-28T05:36:50.951 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-28T05:36:50.951 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.i 2024-03-28T05:36:51.494 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 28 05:36:51 smithi113 systemd[1]: Stopping Ceph mds.i for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:36:51.494 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 28 05:36:51 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-i[68132]: 2024-03-28T05:36:51.269+0000 7f4966a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:36:51.494 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 28 05:36:51 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-i[68132]: 2024-03-28T05:36:51.269+0000 7f4966a00700 -1 mds.i *** got signal Terminated *** 2024-03-28T05:36:53.210 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 28 05:36:52 smithi113 podman[74790]: 2024-03-28 05:36:52.895661411 +0000 UTC m=+1.730186864 container died a3c39012d51c34548fbffe086fe5ecb0b21dd14ca120417ea03a80bb93042649 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-i, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-28T05:36:53.211 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 28 05:36:53 smithi113 podman[74790]: 2024-03-28 05:36:53.210962348 +0000 UTC m=+2.045487738 container cleanup a3c39012d51c34548fbffe086fe5ecb0b21dd14ca120417ea03a80bb93042649 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-i, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T05:36:53.494 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 28 05:36:53 smithi113 bash[74790]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-i 2024-03-28T05:36:53.494 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 28 05:36:53 smithi113 podman[74802]: 2024-03-28 05:36:53.412063326 +0000 UTC m=+0.514092808 container remove a3c39012d51c34548fbffe086fe5ecb0b21dd14ca120417ea03a80bb93042649 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-i, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-28T05:36:53.629 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.i.service' 2024-03-28T05:36:53.665 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:36:53.665 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-28T05:36:53.665 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-28T05:36:53.665 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.l 2024-03-28T05:36:53.981 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 28 05:36:53 smithi113 systemd[1]: Stopping Ceph mds.l for bf37ebc4-ecc1-11ee-95d0-87774f69a715... 2024-03-28T05:36:54.244 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 28 05:36:53 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-l[67187]: 2024-03-28T05:36:53.981+0000 7fbaa621c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T05:36:54.244 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 28 05:36:53 smithi113 ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-l[67187]: 2024-03-28T05:36:53.981+0000 7fbaa621c700 -1 mds.l *** got signal Terminated *** 2024-03-28T05:36:57.072 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 28 05:36:56 smithi113 podman[74912]: 2024-03-28 05:36:56.799427078 +0000 UTC m=+2.919261015 container died 20fe5eb045d592e8ecb1d48ccb3139b1dd2c877a00b1c032113737b20d567c8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-28T05:36:57.072 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 28 05:36:57 smithi113 podman[74912]: 2024-03-28 05:36:57.072806127 +0000 UTC m=+3.192640010 container cleanup 20fe5eb045d592e8ecb1d48ccb3139b1dd2c877a00b1c032113737b20d567c8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-l, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS) 2024-03-28T05:36:57.332 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 28 05:36:57 smithi113 bash[74912]: ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715-mds-l 2024-03-28T05:36:57.488 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf37ebc4-ecc1-11ee-95d0-87774f69a715@mds.l.service' 2024-03-28T05:36:57.522 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T05:36:57.523 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-28T05:36:57.523 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T05:36:57.720 INFO:teuthology.orchestra.run.smithi023.stdout:Deleting cluster with fsid: bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:37:00.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T05:37:00.371 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:37:02.796 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T05:37:02.986 INFO:teuthology.orchestra.run.smithi113.stdout:Deleting cluster with fsid: bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:37:05.500 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T05:37:05.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T05:37:05.567 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T05:37:05.595 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-28T05:37:05.597 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625303/remote/smithi023/crash 2024-03-28T05:37:05.598 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/crash -- . 2024-03-28T05:37:05.628 INFO:teuthology.orchestra.run.smithi023.stderr:tar: /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T05:37:05.629 INFO:teuthology.orchestra.run.smithi023.stderr:tar: Error is not recoverable: exiting now 2024-03-28T05:37:05.630 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625303/remote/smithi098/crash 2024-03-28T05:37:05.631 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/crash -- . 2024-03-28T05:37:05.669 INFO:teuthology.orchestra.run.smithi098.stderr:tar: /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T05:37:05.669 INFO:teuthology.orchestra.run.smithi098.stderr:tar: Error is not recoverable: exiting now 2024-03-28T05:37:05.671 DEBUG:teuthology.misc:Transferring archived files from smithi113:/var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625303/remote/smithi113/crash 2024-03-28T05:37:05.672 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar c -f - -C /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/crash -- . 2024-03-28T05:37:05.710 INFO:teuthology.orchestra.run.smithi113.stderr:tar: /var/lib/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T05:37:05.710 INFO:teuthology.orchestra.run.smithi113.stderr:tar: Error is not recoverable: exiting now 2024-03-28T05:37:05.711 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-28T05:37:05.711 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-28T05:37:05.746 INFO:teuthology.orchestra.run.smithi023.stdout:2024-03-28T05:15:06.547334+0000 mon.a (mon.0) 187 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-28T05:37:05.747 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-28T05:37:05.747 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[SEC\]' /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-28T05:37:05.822 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[ERR\]' /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-28T05:37:05.897 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[WRN\]' /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-28T05:37:05.971 INFO:teuthology.orchestra.run.smithi023.stdout:2024-03-28T05:15:06.547334+0000 mon.a (mon.0) 187 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-28T05:37:05.971 INFO:tasks.cephadm:Compressing logs... 2024-03-28T05:37:05.971 DEBUG:teuthology.orchestra.run.smithi023:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T05:37:06.013 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T05:37:06.015 DEBUG:teuthology.orchestra.run.smithi113:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T05:37:06.038 INFO:teuthology.orchestra.run.smithi023.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T05:37:06.038 INFO:teuthology.orchestra.run.smithi023.stderr:: No such file or directory 2024-03-28T05:37:06.038 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.6.log 2024-03-28T05:37:06.038 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5/var/log/ceph/cephadm.log: --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-28T05:37:06.039 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T05:37:06.039 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.6.log: /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.g.log 2024-03-28T05:37:06.039 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.j.log 2024-03-28T05:37:06.041 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-28T05:37:06.041 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-28T05:37:06.041 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-28T05:37:06.041 INFO:teuthology.orchestra.run.smithi098.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T05:37:06.042 INFO:teuthology.orchestra.run.smithi098.stderr:: No such file or directory 2024-03-28T05:37:06.042 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.10.log 2024-03-28T05:37:06.045 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mgr.x.log: 90.3% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T05:37:06.045 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T05:37:06.045 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.0.log: 94.5% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.g.log.gz 2024-03-28T05:37:06.045 INFO:teuthology.orchestra.run.smithi023.stderr: 94.6% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.j.log.gz 2024-03-28T05:37:06.045 INFO:teuthology.orchestra.run.smithi023.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T05:37:06.047 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T05:37:06.047 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.b.log 2024-03-28T05:37:06.047 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-28T05:37:06.048 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-28T05:37:06.048 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.b.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T05:37:06.048 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.k.log 2024-03-28T05:37:06.048 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.h.log 2024-03-28T05:37:06.048 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.1.log: /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.7.log 2024-03-28T05:37:06.048 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.e.log 2024-03-28T05:37:06.048 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.7.log: 90.4% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T05:37:06.048 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-28T05:37:06.049 INFO:teuthology.orchestra.run.smithi113.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-28T05:37:06.052 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.e.log: 94.4% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.k.log.gz 2024-03-28T05:37:06.052 INFO:teuthology.orchestra.run.smithi098.stderr: 94.5% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.h.log.gz 2024-03-28T05:37:06.052 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T05:37:06.052 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.log 2024-03-28T05:37:06.052 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.cephadm.log: 83.8% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T05:37:06.052 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T05:37:06.053 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.log: 94.5% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.e.log.gz 2024-03-28T05:37:06.053 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-28T05:37:06.054 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-volume.log: 86.7% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T05:37:06.060 INFO:teuthology.orchestra.run.smithi113.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T05:37:06.060 INFO:teuthology.orchestra.run.smithi113.stderr:gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.l.log 2024-03-28T05:37:06.060 INFO:teuthology.orchestra.run.smithi113.stderr:gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T05:37:06.060 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.i.log 2024-03-28T05:37:06.060 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.11.log 2024-03-28T05:37:06.061 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mgr.z.log 2024-03-28T05:37:06.061 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.8.log 2024-03-28T05:37:06.061 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-28T05:37:06.061 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.f.log 2024-03-28T05:37:06.061 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T05:37:06.061 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.log 2024-03-28T05:37:06.061 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-28T05:37:06.061 INFO:teuthology.orchestra.run.smithi113.stderr:gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.c.log 2024-03-28T05:37:06.061 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.log: /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T05:37:06.061 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-28T05:37:06.061 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mon.c.log: 94.7% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.l.log.gz 2024-03-28T05:37:06.063 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mon.b.log: 91.8% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-28T05:37:06.063 INFO:teuthology.orchestra.run.smithi113.stderr: 94.7% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.f.log.gz 2024-03-28T05:37:06.063 INFO:teuthology.orchestra.run.smithi113.stderr: 94.7% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.c.log.gz 2024-03-28T05:37:06.063 INFO:teuthology.orchestra.run.smithi113.stderr: 91.8% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mgr.z.log.gz 2024-03-28T05:37:06.065 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.log 2024-03-28T05:37:06.065 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T05:37:06.065 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.log: 84.1% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T05:37:06.065 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.9.log 2024-03-28T05:37:06.065 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.d.log 2024-03-28T05:37:06.066 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.9.log: 86.7% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T05:37:06.066 INFO:teuthology.orchestra.run.smithi098.stderr: 90.9% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T05:37:06.073 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.a.log 2024-03-28T05:37:06.074 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.d.log: 91.0% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T05:37:06.075 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-28T05:37:06.078 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.a.log: 94.6% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.d.log.gz 2024-03-28T05:37:06.079 INFO:teuthology.orchestra.run.smithi023.stderr: 94.6% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.a.log.gz 2024-03-28T05:37:06.080 INFO:teuthology.orchestra.run.smithi113.stderr: 83.7% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T05:37:06.081 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-28T05:37:06.123 INFO:teuthology.orchestra.run.smithi113.stderr: 86.7% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T05:37:06.204 INFO:teuthology.orchestra.run.smithi113.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T05:37:06.240 INFO:teuthology.orchestra.run.smithi113.stderr: 91.0% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T05:37:06.276 INFO:teuthology.orchestra.run.smithi113.stderr: 90.4% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T05:37:06.335 INFO:teuthology.orchestra.run.smithi098.stderr: 91.9% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-28T05:37:06.370 INFO:teuthology.orchestra.run.smithi023.stderr: 89.9% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-28T05:37:07.111 INFO:teuthology.orchestra.run.smithi113.stderr: 91.9% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-28T05:37:07.245 INFO:teuthology.orchestra.run.smithi023.stderr: 90.4% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-28T05:37:15.811 INFO:teuthology.orchestra.run.smithi098.stderr: 94.7% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.b.log.gz 2024-03-28T05:37:22.692 INFO:teuthology.orchestra.run.smithi098.stderr: 91.7% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-28T05:37:24.805 INFO:teuthology.orchestra.run.smithi098.stderr: 91.6% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.10.log.gz 2024-03-28T05:37:25.103 INFO:teuthology.orchestra.run.smithi023.stderr: 91.6% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-28T05:37:26.271 INFO:teuthology.orchestra.run.smithi098.stderr: 91.5% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.7.log.gz 2024-03-28T05:37:29.159 INFO:teuthology.orchestra.run.smithi023.stderr: 91.3% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.6.log.gz 2024-03-28T05:37:29.363 INFO:teuthology.orchestra.run.smithi023.stderr: 91.5% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-28T05:37:29.566 INFO:teuthology.orchestra.run.smithi023.stderr: 91.4% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.9.log.gz 2024-03-28T05:37:29.568 INFO:teuthology.orchestra.run.smithi023.stderr: 2024-03-28T05:37:29.568 INFO:teuthology.orchestra.run.smithi023.stderr:real 0m23.543s 2024-03-28T05:37:29.568 INFO:teuthology.orchestra.run.smithi023.stderr:user 1m27.912s 2024-03-28T05:37:29.568 INFO:teuthology.orchestra.run.smithi023.stderr:sys 0m2.557s 2024-03-28T05:37:32.030 INFO:teuthology.orchestra.run.smithi098.stderr: 91.7% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-28T05:37:32.032 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-28T05:37:32.032 INFO:teuthology.orchestra.run.smithi098.stderr:real 0m26.003s 2024-03-28T05:37:32.032 INFO:teuthology.orchestra.run.smithi098.stderr:user 1m28.625s 2024-03-28T05:37:32.032 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m2.792s 2024-03-28T05:37:49.715 INFO:teuthology.orchestra.run.smithi113.stderr: 91.6% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.8.log.gz 2024-03-28T05:37:53.552 INFO:teuthology.orchestra.run.smithi113.stderr: 91.6% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.11.log.gz 2024-03-28T05:37:54.785 INFO:teuthology.orchestra.run.smithi113.stderr: 91.7% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-28T05:38:06.683 INFO:teuthology.orchestra.run.smithi113.stderr: 91.2% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-28T05:40:54.385 INFO:teuthology.orchestra.run.smithi113.stderr: 94.2% -- replaced with /var/log/ceph/bf37ebc4-ecc1-11ee-95d0-87774f69a715/ceph-mds.i.log.gz 2024-03-28T05:40:54.386 INFO:teuthology.orchestra.run.smithi113.stderr: 2024-03-28T05:40:54.387 INFO:teuthology.orchestra.run.smithi113.stderr:real 3m48.357s 2024-03-28T05:40:54.387 INFO:teuthology.orchestra.run.smithi113.stderr:user 4m23.210s 2024-03-28T05:40:54.387 INFO:teuthology.orchestra.run.smithi113.stderr:sys 0m9.496s 2024-03-28T05:40:54.387 INFO:tasks.cephadm:Archiving logs... 2024-03-28T05:40:54.387 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625303/remote/smithi023/log 2024-03-28T05:40:54.389 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T05:41:08.568 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625303/remote/smithi098/log 2024-03-28T05:41:08.569 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T05:41:22.004 DEBUG:teuthology.misc:Transferring archived files from smithi113:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625303/remote/smithi113/log 2024-03-28T05:41:22.005 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T05:42:01.340 INFO:tasks.cephadm:Removing cluster... 2024-03-28T05:42:01.341 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 --force 2024-03-28T05:42:01.537 INFO:teuthology.orchestra.run.smithi023.stdout:Deleting cluster with fsid: bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:42:01.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 --force 2024-03-28T05:42:02.126 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:42:02.538 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bf37ebc4-ecc1-11ee-95d0-87774f69a715 --force 2024-03-28T05:42:02.733 INFO:teuthology.orchestra.run.smithi113.stdout:Deleting cluster with fsid: bf37ebc4-ecc1-11ee-95d0-87774f69a715 2024-03-28T05:42:03.235 INFO:tasks.cephadm:Removing cephadm ... 2024-03-28T05:42:03.235 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T05:42:03.251 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T05:42:03.269 DEBUG:teuthology.orchestra.run.smithi113:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T05:42:03.285 INFO:tasks.cephadm:Teardown complete 2024-03-28T05:42:03.286 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-28T05:42:03.298 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-28T05:42:03.298 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T05:42:03.301 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T05:42:03.313 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T05:42:03.429 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-28T05:42:03.441 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-28T05:42:03.441 DEBUG:teuthology.orchestra.run.smithi023:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T05:42:03.444 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T05:42:03.446 DEBUG:teuthology.orchestra.run.smithi113:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T05:42:03.455 INFO:teuthology.orchestra.run.smithi023.stderr:bash: line 1: ntpq: command not found 2024-03-28T05:42:03.459 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T05:42:03.459 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2024-03-28T05:42:03.459 INFO:teuthology.orchestra.run.smithi023.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 11 -726us[ -726us] +/- 77ms 2024-03-28T05:42:03.460 INFO:teuthology.orchestra.run.smithi023.stdout:^* hv02.front.sepia.ceph.com 3 6 377 14 -369us[ -330us] +/- 58ms 2024-03-28T05:42:03.460 INFO:teuthology.orchestra.run.smithi023.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 11 +1510us[+1510us] +/- 87ms 2024-03-28T05:42:03.460 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T05:42:03.460 INFO:teuthology.orchestra.run.smithi113.stderr:bash: line 1: ntpq: command not found 2024-03-28T05:42:03.460 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-03-28T05:42:03.464 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T05:42:03.464 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-03-28T05:42:03.465 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 8 -700us[ -700us] +/- 77ms 2024-03-28T05:42:03.465 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv02.front.sepia.ceph.com 3 6 377 13 -362us[ -363us] +/- 58ms 2024-03-28T05:42:03.465 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 +1523us[+1523us] +/- 87ms 2024-03-28T05:42:03.465 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T05:42:03.469 INFO:teuthology.orchestra.run.smithi113.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T05:42:03.469 INFO:teuthology.orchestra.run.smithi113.stdout:=============================================================================== 2024-03-28T05:42:03.469 INFO:teuthology.orchestra.run.smithi113.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 11 -737us[ -737us] +/- 77ms 2024-03-28T05:42:03.469 INFO:teuthology.orchestra.run.smithi113.stdout:^* hv02.front.sepia.ceph.com 3 6 377 12 -396us[ -380us] +/- 58ms 2024-03-28T05:42:03.469 INFO:teuthology.orchestra.run.smithi113.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 11 +1497us[+1497us] +/- 87ms 2024-03-28T05:42:03.469 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T05:42:03.489 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-28T05:42:03.498 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-28T05:42:03.499 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-28T05:42:03.507 DEBUG:teuthology.orchestra.run.smithi023:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T05:42:03.587 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T05:42:03.663 DEBUG:teuthology.orchestra.run.smithi113:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T05:42:03.757 DEBUG:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T05:42:03.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T05:42:03.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T05:42:03.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T05:42:03.820 DEBUG:teuthology.orchestra.run.smithi113:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T05:42:03.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T05:42:03.850 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-28T05:42:03.860 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-28T05:42:03.869 INFO:teuthology.task.internal:Duration was 2173.719434 seconds 2024-03-28T05:42:03.870 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-28T05:42:03.879 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-28T05:42:03.879 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T05:42:03.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T05:42:03.884 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T05:42:03.921 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T05:42:03.926 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T05:42:03.993 INFO:teuthology.orchestra.run.smithi113.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T05:42:04.399 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-28T05:42:04.399 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi023.front.sepia.ceph.com 2024-03-28T05:42:04.400 DEBUG:teuthology.orchestra.run.smithi023:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T05:42:04.419 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-03-28T05:42:04.419 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T05:42:04.442 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi113.front.sepia.ceph.com 2024-03-28T05:42:04.442 DEBUG:teuthology.orchestra.run.smithi113:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T05:42:04.463 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-28T05:42:04.463 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T05:42:04.466 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T05:42:04.485 DEBUG:teuthology.orchestra.run.smithi113:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T05:42:04.532 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-28T05:42:04.532 DEBUG:teuthology.orchestra.run.smithi023:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T05:42:04.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T05:42:04.555 DEBUG:teuthology.orchestra.run.smithi113:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T05:42:04.914 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-28T05:42:04.926 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-28T05:42:04.926 DEBUG:teuthology.orchestra.run.smithi023:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T05:42:04.953 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T05:42:04.980 DEBUG:teuthology.orchestra.run.smithi113:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T05:42:05.025 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-28T05:42:05.036 DEBUG:teuthology.orchestra.run.smithi023:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T05:42:05.039 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T05:42:05.042 DEBUG:teuthology.orchestra.run.smithi113:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T05:42:05.062 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = core 2024-03-28T05:42:05.064 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-03-28T05:42:05.091 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern = core 2024-03-28T05:42:05.131 DEBUG:teuthology.orchestra.run.smithi023:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T05:42:05.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T05:42:05.146 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T05:42:05.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T05:42:05.170 DEBUG:teuthology.orchestra.run.smithi113:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T05:42:05.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T05:42:05.186 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-28T05:42:05.196 INFO:teuthology.task.internal:Transferring archived files... 2024-03-28T05:42:05.197 DEBUG:teuthology.misc:Transferring archived files from smithi023:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625303/remote/smithi023 2024-03-28T05:42:05.198 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T05:42:05.245 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625303/remote/smithi098 2024-03-28T05:42:05.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T05:42:05.292 DEBUG:teuthology.misc:Transferring archived files from smithi113:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625303/remote/smithi113 2024-03-28T05:42:05.292 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T05:42:05.331 INFO:teuthology.task.internal:Removing archive directory... 2024-03-28T05:42:05.331 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T05:42:05.334 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T05:42:05.336 DEBUG:teuthology.orchestra.run.smithi113:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T05:42:05.379 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-28T05:42:05.392 INFO:teuthology.task.internal:Not uploading archives. 2024-03-28T05:42:05.392 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-28T05:42:05.402 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-28T05:42:05.402 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T05:42:05.404 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T05:42:05.407 DEBUG:teuthology.orchestra.run.smithi113:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T05:42:05.416 INFO:teuthology.orchestra.run.smithi023.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 05:42 /home/ubuntu/cephtest 2024-03-28T05:42:05.418 INFO:teuthology.orchestra.run.smithi098.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 05:42 /home/ubuntu/cephtest 2024-03-28T05:42:05.433 INFO:teuthology.orchestra.run.smithi113.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 05:42 /home/ubuntu/cephtest 2024-03-28T05:42:05.434 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-28T05:42:05.445 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-28T05:42:05.466 INFO:teuthology.nuke:Checking targets against current locks 2024-03-28T05:42:05.492 DEBUG:teuthology.nuke:shortname: smithi023 2024-03-28T05:42:05.493 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T05:42:05.516 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625303', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 04:55:40.385159', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGcXHqnEegKTQ9ebyHpfZMjiU4KptQZ9YMSDrpQMmlrzfi7WmArglGEiTJELPEulThAjk2p8G1sPG4rUfrCnpmE='} 2024-03-28T05:42:05.520 DEBUG:teuthology.nuke:shortname: smithi098 2024-03-28T05:42:05.520 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T05:42:05.545 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625303', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 04:55:40.382945', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIdTI17QUixmhmqfRSyea4E8FISJaOKfNIg5xLG7WF57n71QL7PhRe7qz04si31/T45JHr8AUTNalQX/xsSc6Yo='} 2024-03-28T05:42:05.546 DEBUG:teuthology.nuke:shortname: smithi113 2024-03-28T05:42:05.546 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T05:42:05.571 INFO:teuthology.orchestra.console.smithi023:Power off 2024-03-28T05:42:05.572 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T05:42:05.583 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi113.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625303', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 04:55:40.386874', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEzPeFrAgrCKNnDBti4oU+l/qiuSUt8/4o8ID2LRmpn+ISFgq7e38/AvoGNy+z4d6Y85wfrkpH4N67g8ebfQ8ZY='} 2024-03-28T05:42:05.596 DEBUG:teuthology.orchestra.console.smithi023:power off output: Chassis Power Control: Down/Off 2024-03-28T05:42:05.596 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T05:42:05.619 INFO:teuthology.orchestra.console.smithi098:Power off 2024-03-28T05:42:05.619 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T05:42:05.629 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2024-03-28T05:42:05.642 DEBUG:teuthology.orchestra.console.smithi098:power off output: Chassis Power Control: Down/Off 2024-03-28T05:42:05.642 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T05:42:05.655 INFO:teuthology.orchestra.console.smithi113:Power off 2024-03-28T05:42:05.655 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T05:42:05.664 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-28T05:42:05.678 DEBUG:teuthology.orchestra.console.smithi113:power off output: Chassis Power Control: Down/Off 2024-03-28T05:42:05.678 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T05:42:05.699 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is on 2024-03-28T05:42:09.630 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T05:42:09.665 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T05:42:09.700 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T05:42:09.749 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2024-03-28T05:42:09.776 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-28T05:42:09.810 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is on 2024-03-28T05:42:13.754 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T05:42:13.777 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T05:42:13.810 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T05:42:13.866 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is off 2024-03-28T05:42:13.886 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-28T05:42:13.919 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is on 2024-03-28T05:42:13.967 INFO:teuthology.orchestra.console.smithi023:Power off completed 2024-03-28T05:42:17.890 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T05:42:17.919 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T05:42:18.000 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off 2024-03-28T05:42:18.028 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is off 2024-03-28T05:42:18.101 INFO:teuthology.orchestra.console.smithi098:Power off completed 2024-03-28T05:42:18.129 INFO:teuthology.orchestra.console.smithi113:Power off completed 2024-03-28T05:42:18.233 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} duration: 2173.7194340229034 failure_reason: '"2024-03-28T05:15:06.547334+0000 mon.a (mon.0) 187 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log' flavor: default owner: mchangir success: false 2024-03-28T05:42:18.233 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T05:42:18.344 INFO:teuthology.run:FAIL